Oct 30 19:30:38.089405 ip-10-0-132-214 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:30:38.492948 ip-10-0-132-214 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:30:38.492948 ip-10-0-132-214 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:30:38.492948 ip-10-0-132-214 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:30:38.492948 ip-10-0-132-214 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:30:38.497871 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497790 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:38.497871 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497871 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497876 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497879 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497882 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497884 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497887 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497890 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497893 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497896 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497898 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497901 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497904 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497907 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497909 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497912 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497914 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497917 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497919 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497922 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:38.497937 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497925 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497929 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497939 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497943 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497947 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497950 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497953 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497955 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497958 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497961 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497963 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497966 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497968 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497971 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497973 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497976 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497979 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497981 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497984 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:38.498396 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497986 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497989 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497991 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497994 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497996 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.497998 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498001 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498003 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498006 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498008 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498010 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498014 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498017 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498020 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498023 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498026 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498028 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498031 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498033 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:38.498839 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498036 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498038 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498041 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498043 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498046 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498048 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498051 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498053 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498056 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498058 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498060 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498063 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498065 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498068 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498070 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498072 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498075 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498078 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498080 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498083 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:38.499297 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498085 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498087 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498090 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498093 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498095 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498100 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498103 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498105 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498108 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498111 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498113 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498116 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498118 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498121 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498124 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498126 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498129 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498131 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498134 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:38.499774 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498136 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498139 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498141 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498158 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498161 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498164 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498166 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498171 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498176 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498938 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498945 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498949 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498952 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498955 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498957 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498960 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498963 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498965 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498968 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498980 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:38.500235 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498984 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498987 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498990 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498993 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498995 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.498998 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499001 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499004 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499007 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499009 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499012 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499015 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499018 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499020 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499023 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499026 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499028 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499030 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499033 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:38.500747 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499036 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499038 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499041 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499044 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499046 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499049 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499051 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499054 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499056 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499059 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499062 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499066 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499069 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499078 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499080 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499083 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499085 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499088 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499090 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499093 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:38.501224 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499096 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499099 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499101 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499104 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499106 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499109 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499111 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499114 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499116 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499119 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499121 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499124 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499126 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499129 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499131 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499134 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499136 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499139 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499141 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499159 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:38.501710 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499162 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499164 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499167 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499170 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499172 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499175 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499183 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499186 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499189 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499191 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499194 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499196 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499201 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499204 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499207 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499209 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499212 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499214 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499217 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:38.502206 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499221 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499225 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499228 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499232 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499234 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499238 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499241 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499244 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499247 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499250 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499252 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499255 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499258 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499260 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499264 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499266 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499269 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499799 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499805 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:38.502665 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499814 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499817 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499820 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499823 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499826 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499828 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499832 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499835 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499838 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499840 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499843 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499845 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499848 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499850 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499854 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499857 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499859 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499862 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499865 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499868 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:38.503115 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499871 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499873 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499877 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499881 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499884 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499887 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499889 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499892 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499894 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499897 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499899 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499903 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499907 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499916 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499919 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499922 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499925 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499928 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499932 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:38.503604 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499935 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499938 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499940 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499943 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499945 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499948 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499950 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499953 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499956 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499958 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499961 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499963 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499966 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499968 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499971 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499973 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499976 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499978 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499981 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:38.504060 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499984 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499986 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499989 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499992 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499994 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499996 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.499999 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500001 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500011 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500014 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500016 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500019 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500023 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500025 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500029 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500031 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500034 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500037 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500039 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:38.504540 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500042 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500044 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500047 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500049 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500052 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500054 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500057 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500059 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500062 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500065 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500067 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500070 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500072 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500074 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500077 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500080 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500082 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500085 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500087 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500090 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:38.505031 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500092 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500095 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500103 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500106 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500108 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500113 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.500116 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501008 2579 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501018 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501031 2579 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501036 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501041 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501044 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501048 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501053 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501056 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501059 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501062 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501066 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501069 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501072 2579 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501075 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:30:38.505612 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501078 2579 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501081 2579 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501084 2579 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501091 2579 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501094 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501097 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501100 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501104 2579 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501108 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501111 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501114 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501117 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501121 2579 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501129 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501133 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501137 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501140 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501157 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501161 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501163 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501166 2579 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501169 2579 flags.go:64] FLAG: --enable-server="true" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501173 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501180 2579 flags.go:64] FLAG: --event-burst="100" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501184 2579 flags.go:64] FLAG: --event-qps="50" Oct 30 19:30:38.506132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501187 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501190 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501193 2579 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501197 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501200 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501203 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501206 2579 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501209 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501212 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501215 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501218 2579 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501220 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501224 2579 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501227 2579 flags.go:64] FLAG: --feature-gates="" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501231 2579 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501235 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501238 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501241 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501244 2579 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501248 2579 flags.go:64] FLAG: --help="false" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501251 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501266 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501269 2579 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:30:38.506795 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501272 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501276 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501280 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501283 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501285 2579 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501288 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501293 2579 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501296 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501300 2579 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501308 2579 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501312 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501315 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501318 2579 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501321 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501324 2579 flags.go:64] FLAG: --lock-file="" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501327 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501329 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501333 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501345 2579 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501348 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501351 2579 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501354 2579 flags.go:64] FLAG: --logging-format="text" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501357 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501360 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:30:38.507395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501363 2579 flags.go:64] FLAG: --manifest-url="" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501366 2579 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501371 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501374 2579 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501378 2579 flags.go:64] FLAG: --max-pods="110" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501381 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501384 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501392 2579 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501395 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501398 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501401 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501404 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501411 2579 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501414 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501419 2579 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501423 2579 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501426 2579 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501429 2579 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501432 2579 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501435 2579 flags.go:64] FLAG: --port="10250" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501441 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501444 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-082e3b01570ab4aae" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501448 2579 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501451 2579 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501454 2579 flags.go:64] FLAG: --register-node="true" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501457 2579 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:30:38.507967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501461 2579 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501464 2579 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501467 2579 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501469 2579 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501473 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501477 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501480 2579 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501483 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501485 2579 flags.go:64] FLAG: --runonce="false" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501489 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501492 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501495 2579 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501498 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501501 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501511 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501514 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501517 2579 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501520 2579 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501523 2579 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501526 2579 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501530 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501534 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501537 2579 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501540 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501545 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:30:38.508625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501548 2579 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501551 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501557 2579 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501560 2579 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501563 2579 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501566 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501569 2579 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501574 2579 flags.go:64] FLAG: --v="2" Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501579 2579 flags.go:64] FLAG: --version="false" Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501583 2579 flags.go:64] FLAG: --vmodule="" Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501587 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.501590 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501813 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501818 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501821 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501825 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501829 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501831 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501834 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501837 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501839 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501842 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:38.509217 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501850 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501853 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501856 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501859 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501861 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501865 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501868 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501871 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501873 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501876 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501879 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501881 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501884 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501887 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501889 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501892 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501894 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501898 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501901 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501903 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:38.509783 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501906 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501909 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501912 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501914 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501916 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501919 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501922 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501926 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501930 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501933 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501936 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501938 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501941 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501949 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501952 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501955 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501957 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501962 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501964 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:38.510507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501967 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501969 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501972 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501975 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501977 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501980 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501982 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501985 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501987 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501990 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501993 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501996 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.501998 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502001 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502004 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502006 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502009 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502012 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502014 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502017 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:38.511003 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502019 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502022 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502024 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502027 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502030 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502032 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502035 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502043 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502046 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502049 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502053 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502055 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502058 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502061 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502063 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502066 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502069 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502071 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502073 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502077 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:38.511507 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502079 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502082 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502086 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502090 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502094 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502097 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502100 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502102 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502105 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502108 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502110 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502113 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502115 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502118 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502121 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502123 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:38.512248 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502126 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.502164 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502213 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502218 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502227 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502230 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502234 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502236 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502239 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502242 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502244 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502247 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502250 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502252 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502255 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502257 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:38.512958 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502260 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502262 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502265 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502270 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502272 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502275 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502278 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502280 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502283 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502285 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502288 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502290 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502293 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502296 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502298 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502301 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502303 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502306 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502308 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502311 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:38.513587 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502314 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502323 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502327 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502330 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502333 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502335 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502338 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502340 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502344 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502348 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502351 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502355 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502357 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502360 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502363 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502366 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502369 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502371 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502374 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:38.514223 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502377 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502379 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502382 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502384 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502387 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502390 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502392 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502395 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502398 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502400 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502403 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502405 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502408 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502410 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502413 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502423 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502426 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502428 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502431 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502434 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:38.515020 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502436 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502439 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502442 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502444 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502447 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502449 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502452 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502454 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502464 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502466 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502469 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502471 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502474 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502477 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502479 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502482 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502485 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502487 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502490 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:38.515668 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502492 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502496 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502500 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502503 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502505 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502508 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502511 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502514 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502518 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502526 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502529 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502531 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502534 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.502536 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.509086 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.509101 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.509117 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.509122 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509192 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:38.516352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509198 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509202 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509205 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509209 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509213 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509216 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509219 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509222 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509225 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509228 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509230 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509233 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509236 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509239 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509242 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509245 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509248 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509251 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509254 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509257 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:38.516804 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509260 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509262 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509265 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509268 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509270 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509273 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509276 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509278 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509281 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509284 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509287 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509290 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509292 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509295 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509297 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509300 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509303 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509305 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509308 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509310 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:38.517295 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509313 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509316 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509318 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509321 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509323 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509326 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509329 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509332 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509334 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509337 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509340 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509342 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509345 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509348 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509350 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509353 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509355 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509358 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509360 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509363 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:38.517758 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509366 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509369 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509372 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509374 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509377 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509379 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509382 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509385 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509387 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509390 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509392 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509395 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509398 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509400 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509403 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509405 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509408 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509410 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509413 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:38.518246 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509415 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509418 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509420 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509423 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509426 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509429 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509431 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509434 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509437 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509439 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509443 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509447 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509450 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509454 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509457 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509460 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509462 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509465 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509467 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:38.518694 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509470 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:38.519272 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509473 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:38.519272 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509475 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:38.519272 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509478 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:38.519272 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509481 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:38.519272 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509483 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:38.519272 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.509486 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:38.519272 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.509514 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:30:38.519272 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512156 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:38.519272 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512178 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:38.519272 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512185 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:38.519272 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512190 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:38.519272 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512194 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:38.519272 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512200 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:38.519272 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512204 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:38.519272 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512209 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512213 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512218 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512222 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512226 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512237 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512241 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512245 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512249 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512254 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512257 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512261 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512266 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512270 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512274 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512279 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512283 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512288 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512292 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512296 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:38.519655 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512300 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512305 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512309 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512315 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512328 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512334 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512339 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512343 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512348 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512352 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512356 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512361 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512365 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512370 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512375 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512379 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512383 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512388 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512392 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:38.520225 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512396 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512400 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512405 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512409 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512414 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512418 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512422 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512427 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512431 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512436 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512440 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512445 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512449 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512454 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512458 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512463 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512467 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512471 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512476 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512482 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:38.520704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512487 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512491 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512496 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512500 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512504 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512509 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512513 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512518 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512522 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512526 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512531 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512535 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512539 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512543 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512548 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512552 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512557 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512563 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512567 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512572 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:38.521193 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512576 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512580 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512584 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512588 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512593 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512597 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512601 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512605 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512609 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512613 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512618 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512623 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512627 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512632 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512636 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512640 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512647 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512655 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512661 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:38.521940 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512665 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:38.522729 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.512712 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:30:38.522729 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512775 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:30:38.522729 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512783 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:30:38.522729 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512788 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:30:38.522729 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512793 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:30:38.522729 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512798 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:30:38.522729 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512803 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:30:38.522729 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512807 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:30:38.522729 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512811 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:30:38.522729 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512815 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:30:38.522729 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512820 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:30:38.522729 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512826 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:30:38.522729 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512831 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:30:38.522729 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512835 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:30:38.522729 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512840 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512844 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512849 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512853 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512857 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512862 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512866 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512871 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512875 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512880 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512885 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512890 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512894 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512899 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512903 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512908 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512912 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512916 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512920 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512925 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:30:38.523114 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512929 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512934 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512938 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512943 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512949 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512953 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512957 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512961 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512965 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512970 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512974 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512978 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512982 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512987 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512991 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512995 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.512999 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513003 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513008 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513012 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:30:38.523704 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513017 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513021 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513025 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513029 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513034 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513038 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513042 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513046 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513051 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513055 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513059 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513063 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513068 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513072 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513076 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513080 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513084 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513088 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513092 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:30:38.524317 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513099 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513105 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513109 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513115 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513122 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513128 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513134 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513140 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513160 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513165 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513169 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513173 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513178 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513182 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513186 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513190 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513195 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513199 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513203 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:30:38.525084 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513207 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513212 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513216 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513220 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513227 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513231 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513235 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513239 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513243 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513248 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513251 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513256 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513260 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513264 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:38.513268 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.513357 2579 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.515999 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.518875 2579 server.go:1014] "Starting client certificate rotation" Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.518973 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:30:38.525621 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.519001 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:30:38.526069 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.521999 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:30:38.526069 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.525248 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:30:38.540733 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.540713 2579 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:30:38.546395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.546378 2579 log.go:25] "Validated CRI v1 image API" Oct 30 19:30:38.547211 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.547190 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:30:38.552862 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.552842 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e3b0b617-352c-4958-8430-22033d746ba5:/dev/nvme0n1p3 e9817087-0aab-4af7-853b-f3ea4b800d4b:/dev/nvme0n1p4] Oct 30 19:30:38.552904 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.552862 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 19:30:38.560288 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.560170 2579 manager.go:217] Machine: {Timestamp:2026-10-30 19:30:38.557611879 +0000 UTC m=+0.348685511 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101199 MemoryCapacity:33164349440 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29767042ffddc5ac88c5bda08f429f SystemUUID:ec297670-42ff-ddc5-ac88-c5bda08f429f BootID:b10e7dd1-5056-4667-8d62-ff94d832cfaf Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582172672 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582176768 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ee:71:79:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ee:71:79:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:76:10:bb:a1:c3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164349440 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:30:38.561228 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.561216 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:30:38.561311 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.561300 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:30:38.563603 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.563588 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:30:38.567132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.567107 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:30:38.567299 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.567136 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-214.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:30:38.567381 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.567304 2579 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:30:38.567381 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.567317 2579 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:30:38.567381 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.567375 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:30:38.567487 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.567399 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:30:38.568304 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.568291 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:30:38.572181 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.572167 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:30:38.572322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.572311 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:30:38.572378 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.572351 2579 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:30:38.572378 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.572365 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:30:38.572378 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.572378 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:30:38.572492 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.572391 2579 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:30:38.572492 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.572402 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:30:38.572728 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.572715 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:30:38.575378 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.575364 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:30:38.581526 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.581511 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:30:38.582572 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.582560 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:30:38.582619 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.582589 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:30:38.582772 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.582759 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:30:38.582824 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.582779 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:30:38.582824 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.582787 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:30:38.582824 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.582794 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:30:38.582824 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.582801 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:30:38.582824 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.582810 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:30:38.583242 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.583232 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:30:38.583242 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.583242 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:30:38.583302 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.583257 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:30:38.583302 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.583264 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:30:38.583302 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.583277 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:30:38.583302 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.583285 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:30:38.583693 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.583684 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:30:38.583693 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.583693 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:30:38.586869 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.586851 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-214.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 19:30:38.587748 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.587727 2579 server.go:1263] "Started kubelet" Oct 30 19:30:38.587845 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.587817 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:30:38.588597 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.588535 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:30:38.588676 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.588618 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:30:38.588654 ip-10-0-132-214 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:30:38.590464 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.590445 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:30:38.592599 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.592581 2579 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:30:38.596597 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.596576 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:30:38.596688 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.596654 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 19:30:38.597691 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.597673 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 19:30:38.597753 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.597711 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-214.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 19:30:38.597797 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.597768 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:30:38.597892 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.597871 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:30:38.598533 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.598517 2579 factory.go:55] Registering systemd factory Oct 30 19:30:38.598533 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.598533 2579 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:30:38.598651 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.598549 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:30:38.598651 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.598549 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:30:38.598651 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.598573 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:30:38.598790 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.598661 2579 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:30:38.598790 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.598668 2579 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:30:38.598790 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.598757 2579 factory.go:153] Registering CRI-O factory Oct 30 19:30:38.598790 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.598767 2579 factory.go:223] Registration of the crio container factory successfully Oct 30 19:30:38.598975 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.598821 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:30:38.598975 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.598837 2579 factory.go:103] Registering Raw factory Oct 30 19:30:38.598975 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.598847 2579 manager.go:1199] Started watching for new ooms in manager Oct 30 19:30:38.599287 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.597617 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-214.us-east-2.compute.internal.18e365770b3c1ad7 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-214.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-214.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:30:38.587591383 +0000 UTC m=+0.378665013,LastTimestamp:2026-10-30 19:30:38.587591383 +0000 UTC m=+0.378665013,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-214.us-east-2.compute.internal,}" Oct 30 19:30:38.599818 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.599805 2579 manager.go:319] Starting recovery of all containers Oct 30 19:30:38.601727 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.601699 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-214.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 19:30:38.608989 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.608950 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 19:30:38.613809 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.613789 2579 manager.go:324] Recovery completed Oct 30 19:30:38.618857 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.618843 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:30:38.621731 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.621701 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:30:38.621796 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.621744 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:30:38.621796 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.621757 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:30:38.622332 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.622319 2579 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:30:38.622332 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.622324 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:30:38.622449 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.622332 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:30:38.622449 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.622391 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:30:38.623083 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.623068 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:30:38.623184 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.623099 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:30:38.623184 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.623113 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:30:38.624563 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.624486 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-214.us-east-2.compute.internal.18e365770d4502b1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-214.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-214.us-east-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-214.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:30:38.621729457 +0000 UTC m=+0.412803089,LastTimestamp:2026-10-30 19:30:38.621729457 +0000 UTC m=+0.412803089,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-214.us-east-2.compute.internal,}" Oct 30 19:30:38.624621 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.624575 2579 policy_none.go:50] "Start" Oct 30 19:30:38.624621 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.624588 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:30:38.624621 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.624601 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:30:38.625822 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.625805 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hkmb6" Oct 30 19:30:38.626607 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.626596 2579 policy_none.go:44] "Start" Oct 30 19:30:38.637950 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.637932 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hkmb6" Oct 30 19:30:38.644109 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.644031 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-214.us-east-2.compute.internal.18e365770d455766 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-214.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-214.us-east-2.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-214.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 19:30:38.621751142 +0000 UTC m=+0.412824773,LastTimestamp:2026-10-30 19:30:38.621751142 +0000 UTC m=+0.412824773,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-214.us-east-2.compute.internal,}" Oct 30 19:30:38.659716 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.659681 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:30:38.662274 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.662262 2579 manager.go:341] "Starting Device Plugin manager" Oct 30 19:30:38.662351 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.662287 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:30:38.662351 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.662295 2579 server.go:87] "Starting device plugin registration server" Oct 30 19:30:38.662505 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.662493 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:30:38.662551 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.662505 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:30:38.662642 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.662630 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:30:38.662699 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.662690 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:30:38.662699 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.662698 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:30:38.663222 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.663205 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:30:38.663308 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.663251 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:30:38.762614 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.762561 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:30:38.767274 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.767248 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:30:38.767357 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.767286 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:30:38.767357 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.767300 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:30:38.767357 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.767325 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:38.779050 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.779026 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:30:38.780662 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.780207 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:30:38.780743 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.780668 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:30:38.780743 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.780687 2579 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:30:38.780743 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.780722 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 19:30:38.786215 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.786201 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:38.786333 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.786220 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-214.us-east-2.compute.internal\": node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:30:38.805803 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.805788 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:30:38.840036 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.839910 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:30:38.881210 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.881188 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal"] Oct 30 19:30:38.881277 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.881238 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:30:38.882031 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.882019 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:30:38.882080 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.882043 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:30:38.882080 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.882052 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:30:38.884503 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.884491 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:30:38.884645 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.884631 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:38.884693 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.884660 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:30:38.885260 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.885244 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:30:38.885355 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.885274 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:30:38.885355 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.885288 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:30:38.885355 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.885244 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:30:38.885355 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.885349 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:30:38.885546 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.885360 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:30:38.887135 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.887121 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:38.887191 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.887158 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:30:38.888649 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.888636 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:30:38.888724 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.888659 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:30:38.888724 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.888669 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:30:38.900006 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.899989 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84cfec117dc1db7ebfa66a1a705b525e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"84cfec117dc1db7ebfa66a1a705b525e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:38.900095 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.900018 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f5f4b9c4d5c0325968d9812f1693d009-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"f5f4b9c4d5c0325968d9812f1693d009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:38.900095 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:38.900057 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5f4b9c4d5c0325968d9812f1693d009-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"f5f4b9c4d5c0325968d9812f1693d009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:38.913360 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.913344 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" node="ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:38.916108 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.916082 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" node="ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:38.940760 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:38.940744 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:30:39.000655 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.000636 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5f4b9c4d5c0325968d9812f1693d009-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"f5f4b9c4d5c0325968d9812f1693d009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:39.000730 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.000660 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84cfec117dc1db7ebfa66a1a705b525e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"84cfec117dc1db7ebfa66a1a705b525e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:39.000730 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.000677 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f5f4b9c4d5c0325968d9812f1693d009-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"f5f4b9c4d5c0325968d9812f1693d009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:39.000730 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.000709 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f5f4b9c4d5c0325968d9812f1693d009-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"f5f4b9c4d5c0325968d9812f1693d009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:39.000730 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.000718 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5f4b9c4d5c0325968d9812f1693d009-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"f5f4b9c4d5c0325968d9812f1693d009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:39.000882 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.000763 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84cfec117dc1db7ebfa66a1a705b525e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"84cfec117dc1db7ebfa66a1a705b525e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:39.041525 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:39.041483 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:30:39.142118 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:39.142080 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:30:39.217626 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.217606 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:39.220566 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.220553 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:39.242606 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:39.242588 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:30:39.343543 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:39.343514 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:30:39.444319 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:39.444281 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:30:39.516675 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.516634 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 19:30:39.517182 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.516845 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 19:30:39.545208 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:39.545176 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:30:39.643595 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.643505 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:25:39 +0000 UTC" deadline="2028-08-13 12:48:53.490140548 +0000 UTC" Oct 30 19:30:39.643595 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.643548 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15665h18m13.846595447s" Oct 30 19:30:39.645726 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:39.645703 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:30:39.746811 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:39.746782 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:30:39.757339 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.757318 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:30:39.758404 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.758389 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:30:39.758479 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.758419 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:30:39.758479 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.758436 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:30:39.847375 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:39.847346 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:30:39.915359 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.915288 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:30:39.948046 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:39.948018 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:30:39.979768 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:39.979731 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84cfec117dc1db7ebfa66a1a705b525e.slice/crio-17e9bb78e607eda1bf5e72de4044bbd3579c2fd9b4221af48e9c779d8de68156 WatchSource:0}: Error finding container 17e9bb78e607eda1bf5e72de4044bbd3579c2fd9b4221af48e9c779d8de68156: Status 404 returned error can't find the container with id 17e9bb78e607eda1bf5e72de4044bbd3579c2fd9b4221af48e9c779d8de68156 Oct 30 19:30:39.984682 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:39.984662 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:30:39.989402 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:39.989381 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5f4b9c4d5c0325968d9812f1693d009.slice/crio-07e65c955f517b432c546179e4db92833bc0e84cf063c74ee199eca9f315ef21 WatchSource:0}: Error finding container 07e65c955f517b432c546179e4db92833bc0e84cf063c74ee199eca9f315ef21: Status 404 returned error can't find the container with id 07e65c955f517b432c546179e4db92833bc0e84cf063c74ee199eca9f315ef21 Oct 30 19:30:40.042243 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.042222 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:30:40.099773 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.099746 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:40.119541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.119522 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:30:40.120422 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.120409 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:30:40.125305 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.125289 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:30:40.131304 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.131287 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:30:40.574738 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.574707 2579 apiserver.go:52] "Watching apiserver" Oct 30 19:30:40.582978 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.582955 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:30:40.583325 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.583298 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-mnf6c","openshift-dns/node-resolver-9g689","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal","openshift-multus/multus-additional-cni-plugins-kvwqx","openshift-ovn-kubernetes/ovnkube-node-fg2zw","kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal","openshift-multus/multus-zzxkl","openshift-network-diagnostics/network-check-target-p98lt","kube-system/global-pull-secret-syncer-mm6ld","openshift-image-registry/node-ca-m77bc","openshift-multus/network-metrics-daemon-t4fw7","openshift-network-operator/iptables-alerter-pcsxp","kube-system/konnectivity-agent-9hgsn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d"] Oct 30 19:30:40.585974 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.585953 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.588531 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.588512 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:30:40.591019 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.591001 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.593205 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.593186 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.595702 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.595680 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.597867 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.597845 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 19:30:40.601052 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.601034 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:40.601146 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:40.601111 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p98lt" podUID="6c633f50-fd32-4440-aefc-79e44203424e" Oct 30 19:30:40.601997 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.601977 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:40.602090 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:40.602068 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:30:40.602456 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.602436 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:30:40.602556 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.602439 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:30:40.603311 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.603263 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vcqml\"" Oct 30 19:30:40.603401 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.603361 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:30:40.603463 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.603397 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:30:40.604022 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.603991 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:30:40.604022 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.603994 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:30:40.606916 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.606892 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.609545 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.609525 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9g689" Oct 30 19:30:40.610547 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.610522 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-cni-multus\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.610645 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.610566 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.610645 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.610596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bfb73f4-9290-4440-938a-feb9fafa3287-kubelet-config\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:40.610760 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.610651 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vglqr\" (UniqueName: \"kubernetes.io/projected/d95b4719-782e-47f3-9a7d-ea04268f82da-kube-api-access-vglqr\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.610760 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.610689 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-ovnkube-config\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.610760 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.610709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vkzm\" (UniqueName: \"kubernetes.io/projected/2f2ee082-2070-4390-baf3-55fa09089049-kube-api-access-6vkzm\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.610920 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.610766 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-socket-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.610920 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.610806 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-cni-bin\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.610920 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.610845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-cni-bin\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.610920 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.610878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-etc-kubernetes\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.610920 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.610894 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f0f5eb3-92c0-4822-867c-cf802c9e8490-agent-certs\") pod \"konnectivity-agent-9hgsn\" (UID: \"6f0f5eb3-92c0-4822-867c-cf802c9e8490\") " pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:30:40.611304 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.610940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-netns\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.611304 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.610973 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-conf-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.611304 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611019 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-device-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.611304 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611062 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.611304 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-ovn\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.611304 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611199 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-log-socket\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.611304 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611225 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35066f63-29d2-42da-83b7-edf8894d8e5c-cni-binary-copy\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.611304 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611249 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5c84\" (UniqueName: \"kubernetes.io/projected/89bcca43-c158-47fb-9cb9-1442abc2cd45-kube-api-access-g5c84\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.611304 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611306 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-system-cni-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.611838 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611352 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-multus-certs\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.611838 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611418 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-run-netns\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.611838 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611471 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-etc-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.611838 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611509 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-cnibin\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.611838 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611534 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-system-cni-dir\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.611838 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-cnibin\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.611838 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bfb73f4-9290-4440-938a-feb9fafa3287-dbus\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:40.611838 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611663 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-etc-selinux\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.611838 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611712 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-systemd-units\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.611838 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-node-log\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.611838 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d95b4719-782e-47f3-9a7d-ea04268f82da-ovn-node-metrics-cert\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.611838 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611782 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.611838 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611798 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:40.611838 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611813 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-cni-netd\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.611838 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611838 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-ovnkube-script-lib\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-registration-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611887 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-cni-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-socket-dir-parent\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611934 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f0f5eb3-92c0-4822-867c-cf802c9e8490-konnectivity-ca\") pod \"konnectivity-agent-9hgsn\" (UID: \"6f0f5eb3-92c0-4822-867c-cf802c9e8490\") " pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.611957 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612022 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-slash\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612049 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-env-overrides\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-k8s-cni-cncf-io\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612105 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-kubelet\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-daemon-config\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24nrk\" (UniqueName: \"kubernetes.io/projected/35066f63-29d2-42da-83b7-edf8894d8e5c-kube-api-access-24nrk\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612211 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-run-ovn-kubernetes\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612256 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612308 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-systemd\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612347 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-sys-fs\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612395 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9vjq\" (UniqueName: \"kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq\") pod \"network-check-target-p98lt\" (UID: \"6c633f50-fd32-4440-aefc-79e44203424e\") " pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:40.612541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612399 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:30:40.613395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-hostroot\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.613395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612450 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.613395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612474 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-kubelet\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.613395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612519 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-var-lib-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.613395 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.612536 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.615026 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.614944 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:40.615026 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:40.615013 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:30:40.615271 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.615247 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:30:40.615271 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.615272 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:30:40.615495 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.615308 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:30:40.615495 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.615318 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:30:40.615495 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.615328 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:30:40.615719 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.615563 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:30:40.617029 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.617011 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:30:40.617140 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.617078 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:30:40.617505 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.617283 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:30:40.617505 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.617361 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:30:40.617505 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.617286 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g4c9f\"" Oct 30 19:30:40.617505 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.617374 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-x9qfs\"" Oct 30 19:30:40.617505 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.617450 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4mthb\"" Oct 30 19:30:40.617505 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.617454 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mr7r\"" Oct 30 19:30:40.617505 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.617450 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:30:40.617505 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.617480 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:30:40.636131 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.636100 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:30:40.663813 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.663725 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 19:30:40.663813 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.663793 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-x5w5g" Oct 30 19:30:40.683543 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.683514 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-x5w5g" Oct 30 19:30:40.699527 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.699451 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:30:40.708663 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.708640 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:30:40.710258 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.708784 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:30:40.710258 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.709076 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:30:40.710258 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.709636 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:30:40.710258 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.709769 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7c6bv\"" Oct 30 19:30:40.710258 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.709830 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:30:40.710258 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.709896 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5r9wt\"" Oct 30 19:30:40.710258 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.709935 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:30:40.710258 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.709993 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:30:40.710258 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.710054 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ztltl\"" Oct 30 19:30:40.710258 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.710085 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:30:40.710258 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.710120 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:30:40.710258 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.710146 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:30:40.710258 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.710198 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-79gww\"" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.712871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vjq\" (UniqueName: \"kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq\") pod \"network-check-target-p98lt\" (UID: \"6c633f50-fd32-4440-aefc-79e44203424e\") " pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.712910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysctl-d\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.712971 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-sys\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-hostroot\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713049 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713076 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-kubelet\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713118 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t8r2\" (UniqueName: \"kubernetes.io/projected/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-kube-api-access-5t8r2\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713152 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwf6q\" (UniqueName: \"kubernetes.io/projected/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-kube-api-access-hwf6q\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713200 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-host\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-var-lib-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713264 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-hostroot\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713346 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-var-lib-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-cni-multus\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713437 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713461 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.715750 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713540 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bfb73f4-9290-4440-938a-feb9fafa3287-kubelet-config\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713567 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vglqr\" (UniqueName: \"kubernetes.io/projected/d95b4719-782e-47f3-9a7d-ea04268f82da-kube-api-access-vglqr\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713594 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-tmp-dir\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713620 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb5d13ef-8197-4960-b142-d8a1bdd025e2-serviceca\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fcwx\" (UniqueName: \"kubernetes.io/projected/cc37d377-9cb8-4f67-aa5d-1368a886533c-kube-api-access-5fcwx\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713710 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bfb73f4-9290-4440-938a-feb9fafa3287-kubelet-config\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-ovnkube-config\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713771 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vkzm\" (UniqueName: \"kubernetes.io/projected/2f2ee082-2070-4390-baf3-55fa09089049-kube-api-access-6vkzm\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-socket-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713808 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-hosts-file\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-cni-bin\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713973 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-cni-bin\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.713990 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-etc-kubernetes\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714050 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f0f5eb3-92c0-4822-867c-cf802c9e8490-agent-certs\") pod \"konnectivity-agent-9hgsn\" (UID: \"6f0f5eb3-92c0-4822-867c-cf802c9e8490\") " pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714078 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb5d13ef-8197-4960-b142-d8a1bdd025e2-host\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714122 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysctl-conf\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.716763 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714139 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52n6r\" (UniqueName: \"kubernetes.io/projected/8d7792ad-14ae-482b-9def-eda8e029dcda-kube-api-access-52n6r\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714202 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-netns\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714239 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-socket-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714253 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-conf-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714278 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-cni-bin\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714311 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-etc-kubernetes\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714359 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-device-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714380 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714404 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-ovn\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-log-socket\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714411 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714532 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d7792ad-14ae-482b-9def-eda8e029dcda-tmp\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714554 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35066f63-29d2-42da-83b7-edf8894d8e5c-cni-binary-copy\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5c84\" (UniqueName: \"kubernetes.io/projected/89bcca43-c158-47fb-9cb9-1442abc2cd45-kube-api-access-g5c84\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714585 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714616 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-kubernetes\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714629 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-device-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.717321 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714657 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-var-lib-kubelet\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714684 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-log-socket\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714692 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-system-cni-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714770 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-netns\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714814 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-cni-bin\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-multus-certs\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714851 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-run-netns\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-etc-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc37d377-9cb8-4f67-aa5d-1368a886533c-host-slash\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714920 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-cnibin\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714954 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-system-cni-dir\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714980 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-cni-multus\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714991 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-conf-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714983 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-cnibin\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715016 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-run-netns\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714958 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-ovn\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715041 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-etc-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715054 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bfb73f4-9290-4440-938a-feb9fafa3287-dbus\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:40.717821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715101 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-system-cni-dir\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysconfig\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715146 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-run\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715157 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-cnibin\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715201 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-cnibin\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715226 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-system-cni-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.714115 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-kubelet\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715263 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-multus-certs\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-etc-selinux\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715358 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-systemd-units\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715361 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715385 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bfb73f4-9290-4440-938a-feb9fafa3287-dbus\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715445 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-etc-selinux\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715456 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-node-log\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715490 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35066f63-29d2-42da-83b7-edf8894d8e5c-cni-binary-copy\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715515 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-node-log\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715517 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-systemd-units\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715556 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d95b4719-782e-47f3-9a7d-ea04268f82da-ovn-node-metrics-cert\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.718431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715593 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-ovnkube-config\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715600 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-tuned\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715702 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:40.715894 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.715937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-cni-netd\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-ovnkube-script-lib\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716013 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-cni-netd\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:40.716022 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret podName:5bfb73f4-9290-4440-938a-feb9fafa3287 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:41.215980195 +0000 UTC m=+3.007053831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret") pod "global-pull-secret-syncer-mm6ld" (UID: "5bfb73f4-9290-4440-938a-feb9fafa3287") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716087 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-lib-modules\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-registration-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716212 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-cni-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716238 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-socket-dir-parent\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716263 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f0f5eb3-92c0-4822-867c-cf802c9e8490-konnectivity-ca\") pod \"konnectivity-agent-9hgsn\" (UID: \"6f0f5eb3-92c0-4822-867c-cf802c9e8490\") " pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716310 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-slash\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.719322 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716320 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-cni-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc37d377-9cb8-4f67-aa5d-1368a886533c-iptables-alerter-script\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716372 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-env-overrides\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716381 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-registration-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-k8s-cni-cncf-io\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716427 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-kubelet\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716451 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-daemon-config\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.716475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24nrk\" (UniqueName: \"kubernetes.io/projected/35066f63-29d2-42da-83b7-edf8894d8e5c-kube-api-access-24nrk\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.718330 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-ovnkube-script-lib\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.718398 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-run-ovn-kubernetes\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.718456 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fnzv\" (UniqueName: \"kubernetes.io/projected/fb5d13ef-8197-4960-b142-d8a1bdd025e2-kube-api-access-7fnzv\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.718500 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.718500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-modprobe-d\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.718584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.718649 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-systemd\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.718687 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-systemd\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.718758 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-systemd\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.720198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.718802 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-sys-fs\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.720987 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.718987 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-sys-fs\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.720987 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.719053 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f0f5eb3-92c0-4822-867c-cf802c9e8490-agent-certs\") pod \"konnectivity-agent-9hgsn\" (UID: \"6f0f5eb3-92c0-4822-867c-cf802c9e8490\") " pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:30:40.720987 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.719108 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.720987 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.719259 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-socket-dir-parent\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.720987 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.719594 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-env-overrides\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.720987 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.719673 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-k8s-cni-cncf-io\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.720987 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.719729 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.720987 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.719777 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-kubelet\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.720987 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.720047 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d95b4719-782e-47f3-9a7d-ea04268f82da-ovn-node-metrics-cert\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.720987 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.720119 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-run-ovn-kubernetes\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.720987 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.720309 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-daemon-config\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.720987 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.720347 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f0f5eb3-92c0-4822-867c-cf802c9e8490-konnectivity-ca\") pod \"konnectivity-agent-9hgsn\" (UID: \"6f0f5eb3-92c0-4822-867c-cf802c9e8490\") " pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:30:40.720987 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.720423 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-slash\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.731631 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:40.731609 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:30:40.731731 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:40.731636 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:30:40.731731 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:40.731647 2579 projected.go:196] Error preparing data for projected volume kube-api-access-j9vjq for pod openshift-network-diagnostics/network-check-target-p98lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:30:40.731731 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:40.731705 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq podName:6c633f50-fd32-4440-aefc-79e44203424e nodeName:}" failed. No retries permitted until 2026-10-30 19:30:41.231688385 +0000 UTC m=+3.022762004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-j9vjq" (UniqueName: "kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq") pod "network-check-target-p98lt" (UID: "6c633f50-fd32-4440-aefc-79e44203424e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:30:40.733501 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.733479 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vkzm\" (UniqueName: \"kubernetes.io/projected/2f2ee082-2070-4390-baf3-55fa09089049-kube-api-access-6vkzm\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.734413 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.734385 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5c84\" (UniqueName: \"kubernetes.io/projected/89bcca43-c158-47fb-9cb9-1442abc2cd45-kube-api-access-g5c84\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.736478 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.736453 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vglqr\" (UniqueName: \"kubernetes.io/projected/d95b4719-782e-47f3-9a7d-ea04268f82da-kube-api-access-vglqr\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.749426 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.749400 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24nrk\" (UniqueName: \"kubernetes.io/projected/35066f63-29d2-42da-83b7-edf8894d8e5c-kube-api-access-24nrk\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.787842 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.787721 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" event={"ID":"f5f4b9c4d5c0325968d9812f1693d009","Type":"ContainerStarted","Data":"07e65c955f517b432c546179e4db92833bc0e84cf063c74ee199eca9f315ef21"} Oct 30 19:30:40.790754 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.790685 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" event={"ID":"84cfec117dc1db7ebfa66a1a705b525e","Type":"ContainerStarted","Data":"17e9bb78e607eda1bf5e72de4044bbd3579c2fd9b4221af48e9c779d8de68156"} Oct 30 19:30:40.819572 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-tuned\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.819731 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:40.819731 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819625 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-lib-modules\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.819731 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc37d377-9cb8-4f67-aa5d-1368a886533c-iptables-alerter-script\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:30:40.819731 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fnzv\" (UniqueName: \"kubernetes.io/projected/fb5d13ef-8197-4960-b142-d8a1bdd025e2-kube-api-access-7fnzv\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:30:40.819731 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819713 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-modprobe-d\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.820009 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819739 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-systemd\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.820009 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819777 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysctl-d\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.820009 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819801 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-sys\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.820009 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819828 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8r2\" (UniqueName: \"kubernetes.io/projected/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-kube-api-access-5t8r2\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:30:40.820009 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwf6q\" (UniqueName: \"kubernetes.io/projected/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-kube-api-access-hwf6q\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:40.820009 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819875 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-host\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.820009 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819912 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-tmp-dir\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:30:40.820009 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb5d13ef-8197-4960-b142-d8a1bdd025e2-serviceca\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:30:40.820009 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819959 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fcwx\" (UniqueName: \"kubernetes.io/projected/cc37d377-9cb8-4f67-aa5d-1368a886533c-kube-api-access-5fcwx\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:30:40.820009 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.819989 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-hosts-file\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:30:40.820562 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.820018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb5d13ef-8197-4960-b142-d8a1bdd025e2-host\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:30:40.820562 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.820041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysctl-conf\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.820562 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.820066 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52n6r\" (UniqueName: \"kubernetes.io/projected/8d7792ad-14ae-482b-9def-eda8e029dcda-kube-api-access-52n6r\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.820562 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.820095 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d7792ad-14ae-482b-9def-eda8e029dcda-tmp\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.820562 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.820121 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-kubernetes\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.820562 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.820145 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-var-lib-kubelet\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.820562 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.820176 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc37d377-9cb8-4f67-aa5d-1368a886533c-host-slash\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:30:40.820562 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.820230 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysconfig\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.820562 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.820255 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-run\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.820562 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.820367 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-run\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.820961 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.820930 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-systemd\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.821058 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:40.821041 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:30:40.821102 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.821088 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-tmp-dir\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:30:40.821139 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:40.821107 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs podName:af2922a1-2b0f-4ee1-8ef3-4911d029e93d nodeName:}" failed. No retries permitted until 2026-10-30 19:30:41.321087619 +0000 UTC m=+3.112161243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs") pod "network-metrics-daemon-t4fw7" (UID: "af2922a1-2b0f-4ee1-8ef3-4911d029e93d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:30:40.821305 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.821276 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysctl-d\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.821362 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.821342 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-sys\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.821430 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.821400 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-lib-modules\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.821855 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.821800 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-host\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.821944 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.821927 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-modprobe-d\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.822111 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.822089 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc37d377-9cb8-4f67-aa5d-1368a886533c-iptables-alerter-script\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:30:40.822201 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.822188 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-var-lib-kubelet\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.822608 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.822580 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb5d13ef-8197-4960-b142-d8a1bdd025e2-serviceca\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:30:40.822842 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.822819 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb5d13ef-8197-4960-b142-d8a1bdd025e2-host\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:30:40.822923 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.822890 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-hosts-file\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:30:40.823057 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.823016 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysctl-conf\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.823128 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.823078 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-kubernetes\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.823128 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.823119 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc37d377-9cb8-4f67-aa5d-1368a886533c-host-slash\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:30:40.823278 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.823169 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysconfig\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.825523 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.825443 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d7792ad-14ae-482b-9def-eda8e029dcda-tmp\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.825697 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.825676 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-tuned\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.849884 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.849848 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8r2\" (UniqueName: \"kubernetes.io/projected/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-kube-api-access-5t8r2\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:30:40.850038 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.850018 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fcwx\" (UniqueName: \"kubernetes.io/projected/cc37d377-9cb8-4f67-aa5d-1368a886533c-kube-api-access-5fcwx\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:30:40.850517 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.850493 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwf6q\" (UniqueName: \"kubernetes.io/projected/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-kube-api-access-hwf6q\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:40.852127 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.852106 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fnzv\" (UniqueName: \"kubernetes.io/projected/fb5d13ef-8197-4960-b142-d8a1bdd025e2-kube-api-access-7fnzv\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:30:40.853260 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.853240 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52n6r\" (UniqueName: \"kubernetes.io/projected/8d7792ad-14ae-482b-9def-eda8e029dcda-kube-api-access-52n6r\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.901012 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.900988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zzxkl" Oct 30 19:30:40.912068 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:40.912020 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35066f63_29d2_42da_83b7_edf8894d8e5c.slice/crio-f9b6276abba369dffcce5917f8594ebe32418e7876b9d4048757aa2aaee0f04e WatchSource:0}: Error finding container f9b6276abba369dffcce5917f8594ebe32418e7876b9d4048757aa2aaee0f04e: Status 404 returned error can't find the container with id f9b6276abba369dffcce5917f8594ebe32418e7876b9d4048757aa2aaee0f04e Oct 30 19:30:40.912913 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.912890 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:30:40.922687 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:40.922654 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0f5eb3_92c0_4822_867c_cf802c9e8490.slice/crio-4f065122df4f28ace6aa0054779e9e09fdcb31b0093ad0f24d81cfb30ab25f9b WatchSource:0}: Error finding container 4f065122df4f28ace6aa0054779e9e09fdcb31b0093ad0f24d81cfb30ab25f9b: Status 404 returned error can't find the container with id 4f065122df4f28ace6aa0054779e9e09fdcb31b0093ad0f24d81cfb30ab25f9b Oct 30 19:30:40.928158 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.928001 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:30:40.938407 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:40.938385 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f2ee082_2070_4390_baf3_55fa09089049.slice/crio-a2f0e1459db066802ca9b3b5072041807b613908bdf9d00ee78221c5d7fc3730 WatchSource:0}: Error finding container a2f0e1459db066802ca9b3b5072041807b613908bdf9d00ee78221c5d7fc3730: Status 404 returned error can't find the container with id a2f0e1459db066802ca9b3b5072041807b613908bdf9d00ee78221c5d7fc3730 Oct 30 19:30:40.949127 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.948196 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:30:40.951057 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.951029 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:30:40.962612 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:40.962586 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89bcca43_c158_47fb_9cb9_1442abc2cd45.slice/crio-c951fe4859bfa6a7da5e6428c2209e6ae00baa8cae45386104660f22be56c53f WatchSource:0}: Error finding container c951fe4859bfa6a7da5e6428c2209e6ae00baa8cae45386104660f22be56c53f: Status 404 returned error can't find the container with id c951fe4859bfa6a7da5e6428c2209e6ae00baa8cae45386104660f22be56c53f Oct 30 19:30:40.963824 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:40.963804 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d7792ad_14ae_482b_9def_eda8e029dcda.slice/crio-fc050878812afa1d2a5bf3df01c3f68a7e70f3c46cde6500781c325dd702b44f WatchSource:0}: Error finding container fc050878812afa1d2a5bf3df01c3f68a7e70f3c46cde6500781c325dd702b44f: Status 404 returned error can't find the container with id fc050878812afa1d2a5bf3df01c3f68a7e70f3c46cde6500781c325dd702b44f Oct 30 19:30:40.966018 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.965836 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-9g689" Oct 30 19:30:40.968666 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.968641 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:30:40.975516 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.975326 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:30:40.977391 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:40.977370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:30:40.979391 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:40.979362 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa163f8_129c_4f64_ad4b_ad3cc08bed13.slice/crio-d1365cbce7525d7206cba488262330f62c465d65a50e638864581050077f6ce1 WatchSource:0}: Error finding container d1365cbce7525d7206cba488262330f62c465d65a50e638864581050077f6ce1: Status 404 returned error can't find the container with id d1365cbce7525d7206cba488262330f62c465d65a50e638864581050077f6ce1 Oct 30 19:30:40.983447 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:30:40.983418 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95b4719_782e_47f3_9a7d_ea04268f82da.slice/crio-0eb71ea148ca645b71236656f8c56e4aa6e92120dfee48fcb650f0d005b3b047 WatchSource:0}: Error finding container 0eb71ea148ca645b71236656f8c56e4aa6e92120dfee48fcb650f0d005b3b047: Status 404 returned error can't find the container with id 0eb71ea148ca645b71236656f8c56e4aa6e92120dfee48fcb650f0d005b3b047 Oct 30 19:30:41.224090 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.224006 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:41.224326 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:41.224189 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:30:41.224326 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:41.224261 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret podName:5bfb73f4-9290-4440-938a-feb9fafa3287 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:42.224240192 +0000 UTC m=+4.015313825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret") pod "global-pull-secret-syncer-mm6ld" (UID: "5bfb73f4-9290-4440-938a-feb9fafa3287") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:30:41.324873 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.324832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:41.325059 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.324891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vjq\" (UniqueName: \"kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq\") pod \"network-check-target-p98lt\" (UID: \"6c633f50-fd32-4440-aefc-79e44203424e\") " pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:41.325059 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:41.325023 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:30:41.325059 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:41.325037 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:30:41.325059 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:41.325048 2579 projected.go:196] Error preparing data for projected volume kube-api-access-j9vjq for pod openshift-network-diagnostics/network-check-target-p98lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:30:41.325267 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:41.325094 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq podName:6c633f50-fd32-4440-aefc-79e44203424e nodeName:}" failed. No retries permitted until 2026-10-30 19:30:42.325081204 +0000 UTC m=+4.116154838 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-j9vjq" (UniqueName: "kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq") pod "network-check-target-p98lt" (UID: "6c633f50-fd32-4440-aefc-79e44203424e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:30:41.325267 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:41.325020 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:30:41.325267 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:41.325196 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs podName:af2922a1-2b0f-4ee1-8ef3-4911d029e93d nodeName:}" failed. No retries permitted until 2026-10-30 19:30:42.325174503 +0000 UTC m=+4.116248126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs") pod "network-metrics-daemon-t4fw7" (UID: "af2922a1-2b0f-4ee1-8ef3-4911d029e93d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:30:41.687364 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.687176 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:25:41 +0000 UTC" deadline="2028-04-20 07:39:30.670463781 +0000 UTC" Oct 30 19:30:41.687364 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.687232 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12900h8m48.983235139s" Oct 30 19:30:41.781722 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.781696 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:41.781875 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:41.781850 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:30:41.781950 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.781903 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:41.782072 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:41.782037 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:30:41.831394 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.831361 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9g689" event={"ID":"7fa163f8-129c-4f64-ad4b-ad3cc08bed13","Type":"ContainerStarted","Data":"d1365cbce7525d7206cba488262330f62c465d65a50e638864581050077f6ce1"} Oct 30 19:30:41.853406 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.852709 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9hgsn" event={"ID":"6f0f5eb3-92c0-4822-867c-cf802c9e8490","Type":"ContainerStarted","Data":"4f065122df4f28ace6aa0054779e9e09fdcb31b0093ad0f24d81cfb30ab25f9b"} Oct 30 19:30:41.864707 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.864628 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pcsxp" event={"ID":"cc37d377-9cb8-4f67-aa5d-1368a886533c","Type":"ContainerStarted","Data":"2759d713bb56bcdeba60dd019aafb131dfe684d5b2c0bf8a711247cfa6333336"} Oct 30 19:30:41.866889 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.866805 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m77bc" event={"ID":"fb5d13ef-8197-4960-b142-d8a1bdd025e2","Type":"ContainerStarted","Data":"6b36afa016de24b8aa10c69c1f397522ee3d58e603a05adf04708b9df644e170"} Oct 30 19:30:41.885849 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.885817 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" event={"ID":"d95b4719-782e-47f3-9a7d-ea04268f82da","Type":"ContainerStarted","Data":"0eb71ea148ca645b71236656f8c56e4aa6e92120dfee48fcb650f0d005b3b047"} Oct 30 19:30:41.895037 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.895006 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" event={"ID":"8d7792ad-14ae-482b-9def-eda8e029dcda","Type":"ContainerStarted","Data":"fc050878812afa1d2a5bf3df01c3f68a7e70f3c46cde6500781c325dd702b44f"} Oct 30 19:30:41.903577 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.903543 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvwqx" event={"ID":"89bcca43-c158-47fb-9cb9-1442abc2cd45","Type":"ContainerStarted","Data":"c951fe4859bfa6a7da5e6428c2209e6ae00baa8cae45386104660f22be56c53f"} Oct 30 19:30:41.915970 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.915743 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" event={"ID":"2f2ee082-2070-4390-baf3-55fa09089049","Type":"ContainerStarted","Data":"a2f0e1459db066802ca9b3b5072041807b613908bdf9d00ee78221c5d7fc3730"} Oct 30 19:30:41.932210 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:41.932068 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zzxkl" event={"ID":"35066f63-29d2-42da-83b7-edf8894d8e5c","Type":"ContainerStarted","Data":"f9b6276abba369dffcce5917f8594ebe32418e7876b9d4048757aa2aaee0f04e"} Oct 30 19:30:42.233369 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:42.233330 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:42.233570 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:42.233473 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:30:42.233570 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:42.233534 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret podName:5bfb73f4-9290-4440-938a-feb9fafa3287 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:44.23351581 +0000 UTC m=+6.024589446 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret") pod "global-pull-secret-syncer-mm6ld" (UID: "5bfb73f4-9290-4440-938a-feb9fafa3287") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:30:42.334248 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:42.334195 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:42.334248 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:42.334246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vjq\" (UniqueName: \"kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq\") pod \"network-check-target-p98lt\" (UID: \"6c633f50-fd32-4440-aefc-79e44203424e\") " pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:42.334511 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:42.334355 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:30:42.334511 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:42.334371 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:30:42.334511 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:42.334380 2579 projected.go:196] Error preparing data for projected volume kube-api-access-j9vjq for pod openshift-network-diagnostics/network-check-target-p98lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:30:42.334511 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:42.334420 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq podName:6c633f50-fd32-4440-aefc-79e44203424e nodeName:}" failed. No retries permitted until 2026-10-30 19:30:44.334408168 +0000 UTC m=+6.125481787 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-j9vjq" (UniqueName: "kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq") pod "network-check-target-p98lt" (UID: "6c633f50-fd32-4440-aefc-79e44203424e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:30:42.334760 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:42.334714 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:30:42.334760 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:42.334745 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs podName:af2922a1-2b0f-4ee1-8ef3-4911d029e93d nodeName:}" failed. No retries permitted until 2026-10-30 19:30:44.334736382 +0000 UTC m=+6.125810015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs") pod "network-metrics-daemon-t4fw7" (UID: "af2922a1-2b0f-4ee1-8ef3-4911d029e93d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:30:42.688209 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:42.687709 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:25:41 +0000 UTC" deadline="2028-05-28 08:12:00.900167041 +0000 UTC" Oct 30 19:30:42.688209 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:42.687747 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13812h41m18.212425462s" Oct 30 19:30:42.785769 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:42.781457 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:42.785769 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:42.781591 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p98lt" podUID="6c633f50-fd32-4440-aefc-79e44203424e" Oct 30 19:30:42.952253 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:42.950955 2579 generic.go:356] "Generic (PLEG): container finished" podID="f5f4b9c4d5c0325968d9812f1693d009" containerID="4e36fa05873ca0e43e3644e018d6334ff5bf46c0c538d13038328de655aabf8a" exitCode=0 Oct 30 19:30:42.952253 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:42.951309 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" event={"ID":"f5f4b9c4d5c0325968d9812f1693d009","Type":"ContainerDied","Data":"4e36fa05873ca0e43e3644e018d6334ff5bf46c0c538d13038328de655aabf8a"} Oct 30 19:30:42.960864 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:42.960335 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" event={"ID":"84cfec117dc1db7ebfa66a1a705b525e","Type":"ContainerStarted","Data":"c19a97b6bededb7d4a9b35efed5664bfff1864e300287ad0237fc45de3c65370"} Oct 30 19:30:43.781014 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:43.780931 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:43.781511 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:43.781094 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:30:43.781587 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:43.781521 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:43.781649 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:43.781629 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:30:43.971738 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:43.971703 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" event={"ID":"f5f4b9c4d5c0325968d9812f1693d009","Type":"ContainerStarted","Data":"34def9a07570e749a94faad670dccb763d05de3c26a245752d0515623fcf800b"} Oct 30 19:30:43.991646 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:43.991598 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" podStartSLOduration=3.991583199 podStartE2EDuration="3.991583199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:42.990337436 +0000 UTC m=+4.781411080" watchObservedRunningTime="2026-10-30 19:30:43.991583199 +0000 UTC m=+5.782656841" Oct 30 19:30:43.992211 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:43.992172 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" podStartSLOduration=3.992157734 podStartE2EDuration="3.992157734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:43.991347269 +0000 UTC m=+5.782420913" watchObservedRunningTime="2026-10-30 19:30:43.992157734 +0000 UTC m=+5.783231380" Oct 30 19:30:44.250963 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:44.250919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:44.251117 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:44.251067 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:30:44.251197 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:44.251130 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret podName:5bfb73f4-9290-4440-938a-feb9fafa3287 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:48.251111455 +0000 UTC m=+10.042185090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret") pod "global-pull-secret-syncer-mm6ld" (UID: "5bfb73f4-9290-4440-938a-feb9fafa3287") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:30:44.352327 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:44.352231 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vjq\" (UniqueName: \"kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq\") pod \"network-check-target-p98lt\" (UID: \"6c633f50-fd32-4440-aefc-79e44203424e\") " pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:44.352327 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:44.352313 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:44.352542 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:44.352441 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:30:44.352542 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:44.352501 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs podName:af2922a1-2b0f-4ee1-8ef3-4911d029e93d nodeName:}" failed. No retries permitted until 2026-10-30 19:30:48.352482061 +0000 UTC m=+10.143555694 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs") pod "network-metrics-daemon-t4fw7" (UID: "af2922a1-2b0f-4ee1-8ef3-4911d029e93d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:30:44.352940 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:44.352923 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:30:44.353055 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:44.352944 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:30:44.353055 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:44.352957 2579 projected.go:196] Error preparing data for projected volume kube-api-access-j9vjq for pod openshift-network-diagnostics/network-check-target-p98lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:30:44.353055 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:44.353003 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq podName:6c633f50-fd32-4440-aefc-79e44203424e nodeName:}" failed. No retries permitted until 2026-10-30 19:30:48.35298748 +0000 UTC m=+10.144061114 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-j9vjq" (UniqueName: "kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq") pod "network-check-target-p98lt" (UID: "6c633f50-fd32-4440-aefc-79e44203424e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:30:44.784905 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:44.784257 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:44.784905 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:44.784421 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p98lt" podUID="6c633f50-fd32-4440-aefc-79e44203424e" Oct 30 19:30:44.980257 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:44.980201 2579 generic.go:356] "Generic (PLEG): container finished" podID="89bcca43-c158-47fb-9cb9-1442abc2cd45" containerID="4f796438ba42579b9fe3e1124e9ad3be8a18c82865803b7959d12f37547e0a3b" exitCode=0 Oct 30 19:30:44.980507 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:44.980277 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvwqx" event={"ID":"89bcca43-c158-47fb-9cb9-1442abc2cd45","Type":"ContainerDied","Data":"4f796438ba42579b9fe3e1124e9ad3be8a18c82865803b7959d12f37547e0a3b"} Oct 30 19:30:45.781355 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:45.781318 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:45.781561 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:45.781455 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:30:45.781920 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:45.781903 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:45.782093 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:45.782071 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:30:45.992070 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:45.992031 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9hgsn" event={"ID":"6f0f5eb3-92c0-4822-867c-cf802c9e8490","Type":"ContainerStarted","Data":"925fe4db6ad133da80f182f831dc7b9771f4bf24314649675cea41577d41c37b"} Oct 30 19:30:46.027248 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:46.027158 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9hgsn" podStartSLOduration=2.597983415 podStartE2EDuration="7.02713779s" totalImagesPullingTime="4.429154375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:46.026878268 +0000 UTC m=+7.817951912" watchObservedRunningTime="2026-10-30 19:30:46.02713779 +0000 UTC m=+7.818211433" Oct 30 19:30:46.784852 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:46.783315 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:46.784852 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:46.784207 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p98lt" podUID="6c633f50-fd32-4440-aefc-79e44203424e" Oct 30 19:30:46.997479 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:46.996717 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-m77bc" event={"ID":"fb5d13ef-8197-4960-b142-d8a1bdd025e2","Type":"ContainerStarted","Data":"a60b8d08a1f8a22cd3ca668b3c04b5723b826edbcb15f8be2a83e83308a76061"} Oct 30 19:30:47.023378 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:47.023315 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-m77bc" podStartSLOduration=2.651589611 podStartE2EDuration="8.023297819s" totalImagesPullingTime="5.371708208s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:47.023284104 +0000 UTC m=+8.814357746" watchObservedRunningTime="2026-10-30 19:30:47.023297819 +0000 UTC m=+8.814371462" Oct 30 19:30:47.781035 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:47.781001 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:47.781262 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:47.781160 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:30:47.781673 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:47.781650 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:47.781782 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:47.781752 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:30:48.005782 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:48.005710 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" event={"ID":"2f2ee082-2070-4390-baf3-55fa09089049","Type":"ContainerStarted","Data":"a9658d7d157affedabedf574dca0ff0a20ea5e9cbd7589b8e854a59080109d75"} Oct 30 19:30:48.282996 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:48.282962 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:48.283123 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:48.283092 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:30:48.283197 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:48.283155 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret podName:5bfb73f4-9290-4440-938a-feb9fafa3287 nodeName:}" failed. No retries permitted until 2026-10-30 19:30:56.283129869 +0000 UTC m=+18.074203490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret") pod "global-pull-secret-syncer-mm6ld" (UID: "5bfb73f4-9290-4440-938a-feb9fafa3287") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:30:48.383642 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:48.383606 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:48.383792 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:48.383670 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vjq\" (UniqueName: \"kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq\") pod \"network-check-target-p98lt\" (UID: \"6c633f50-fd32-4440-aefc-79e44203424e\") " pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:48.383876 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:48.383836 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:30:48.383876 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:48.383857 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:30:48.383876 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:48.383870 2579 projected.go:196] Error preparing data for projected volume kube-api-access-j9vjq for pod openshift-network-diagnostics/network-check-target-p98lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:30:48.383990 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:48.383928 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq podName:6c633f50-fd32-4440-aefc-79e44203424e nodeName:}" failed. No retries permitted until 2026-10-30 19:30:56.383909652 +0000 UTC m=+18.174983284 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-j9vjq" (UniqueName: "kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq") pod "network-check-target-p98lt" (UID: "6c633f50-fd32-4440-aefc-79e44203424e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:30:48.384363 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:48.384341 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:30:48.384445 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:48.384403 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs podName:af2922a1-2b0f-4ee1-8ef3-4911d029e93d nodeName:}" failed. No retries permitted until 2026-10-30 19:30:56.384386975 +0000 UTC m=+18.175460594 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs") pod "network-metrics-daemon-t4fw7" (UID: "af2922a1-2b0f-4ee1-8ef3-4911d029e93d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:30:48.661776 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:48.661737 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:30:48.668269 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:48.667783 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:30:48.661757277Z","UUID":"988eefcb-7699-46ca-8759-1d7f361e143c","Handler":null,"Name":"","Endpoint":""} Oct 30 19:30:48.669531 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:48.669122 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:30:48.669531 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:48.669163 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:30:48.783217 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:48.782533 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:48.783217 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:48.782643 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p98lt" podUID="6c633f50-fd32-4440-aefc-79e44203424e" Oct 30 19:30:49.010484 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:49.010447 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" event={"ID":"2f2ee082-2070-4390-baf3-55fa09089049","Type":"ContainerStarted","Data":"15bdc04689d2a90f9e52073d03aed98d182911e1ab831608516e5c4baafd0e5e"} Oct 30 19:30:49.406524 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:49.405261 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:30:49.406524 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:49.405928 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:30:49.780944 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:49.780916 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:49.781067 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:49.780955 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:49.781126 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:49.781062 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:30:49.781245 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:49.781224 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:30:50.016755 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:50.016703 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvwqx" event={"ID":"89bcca43-c158-47fb-9cb9-1442abc2cd45","Type":"ContainerStarted","Data":"811ce339461a265908c5efa81df75e9c7129d84caecb6d3c0cb0a641b5d90499"} Oct 30 19:30:50.021956 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:50.021905 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zzxkl" event={"ID":"35066f63-29d2-42da-83b7-edf8894d8e5c","Type":"ContainerStarted","Data":"43228115b079fe45dba4b61930866c8fa9b858cb0d7299231ee1c4938ac4629e"} Oct 30 19:30:50.025346 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:50.025316 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-9g689" event={"ID":"7fa163f8-129c-4f64-ad4b-ad3cc08bed13","Type":"ContainerStarted","Data":"673ccde96222f22d48a7cf261c40e1a9340e012790ed515d1e2ab5256820f65f"} Oct 30 19:30:50.025635 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:50.025591 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:30:50.026311 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:50.026271 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:30:50.089644 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:50.088784 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-9g689" podStartSLOduration=2.44292518 podStartE2EDuration="11.088769919s" totalImagesPullingTime="8.645844739s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:50.088529973 +0000 UTC m=+11.879603612" watchObservedRunningTime="2026-10-30 19:30:50.088769919 +0000 UTC m=+11.879843558" Oct 30 19:30:50.089644 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:50.089208 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-zzxkl" podStartSLOduration=2.880951698 podStartE2EDuration="11.089197414s" totalImagesPullingTime="8.208245716s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:50.067005499 +0000 UTC m=+11.858079140" watchObservedRunningTime="2026-10-30 19:30:50.089197414 +0000 UTC m=+11.880271054" Oct 30 19:30:50.781190 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:50.781137 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:50.781352 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:50.781301 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p98lt" podUID="6c633f50-fd32-4440-aefc-79e44203424e" Oct 30 19:30:51.031040 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:51.030545 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" event={"ID":"2f2ee082-2070-4390-baf3-55fa09089049","Type":"ContainerStarted","Data":"df8ded9563c4e767d7cf5abe2ee887f8443bf7a2031e7d3ce4b7b54f0baf0336"} Oct 30 19:30:51.033219 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:51.033154 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pcsxp" event={"ID":"cc37d377-9cb8-4f67-aa5d-1368a886533c","Type":"ContainerStarted","Data":"a53860cdee86405d87399355d36119b6a4216a06ea86c990b7ae74cd8cd1acf3"} Oct 30 19:30:51.035388 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:51.035288 2579 generic.go:356] "Generic (PLEG): container finished" podID="89bcca43-c158-47fb-9cb9-1442abc2cd45" containerID="811ce339461a265908c5efa81df75e9c7129d84caecb6d3c0cb0a641b5d90499" exitCode=0 Oct 30 19:30:51.035482 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:51.035386 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvwqx" event={"ID":"89bcca43-c158-47fb-9cb9-1442abc2cd45","Type":"ContainerDied","Data":"811ce339461a265908c5efa81df75e9c7129d84caecb6d3c0cb0a641b5d90499"} Oct 30 19:30:51.051522 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:51.051469 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" podStartSLOduration=3.357670821 podStartE2EDuration="12.051452834s" totalImagesPullingTime="8.693782013s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:39 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:51.051441508 +0000 UTC m=+12.842515148" watchObservedRunningTime="2026-10-30 19:30:51.051452834 +0000 UTC m=+12.842526474" Oct 30 19:30:51.093376 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:51.093320 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pcsxp" podStartSLOduration=3.582463527 podStartE2EDuration="12.093305581s" totalImagesPullingTime="8.510842054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:51.09288259 +0000 UTC m=+12.883956230" watchObservedRunningTime="2026-10-30 19:30:51.093305581 +0000 UTC m=+12.884379221" Oct 30 19:30:51.781022 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:51.780998 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:51.781140 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:51.781061 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:51.781648 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:51.781203 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:30:51.781648 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:51.781604 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:30:52.040123 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:52.040086 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" event={"ID":"8d7792ad-14ae-482b-9def-eda8e029dcda","Type":"ContainerStarted","Data":"0e00c3d398445d53f285a7ef594416aab6c21cb9db8f24cd117463338bbdbad3"} Oct 30 19:30:52.062393 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:52.062346 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" podStartSLOduration=2.290776389 podStartE2EDuration="13.062333659s" totalImagesPullingTime="10.77155727s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:52.062311199 +0000 UTC m=+13.853384840" watchObservedRunningTime="2026-10-30 19:30:52.062333659 +0000 UTC m=+13.853407298" Oct 30 19:30:52.781627 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:52.781594 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:52.781816 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:52.781725 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p98lt" podUID="6c633f50-fd32-4440-aefc-79e44203424e" Oct 30 19:30:53.044473 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:53.044392 2579 generic.go:356] "Generic (PLEG): container finished" podID="89bcca43-c158-47fb-9cb9-1442abc2cd45" containerID="c2dddcc3a8880575c112705ba19f68d6ed3363c7bc7eac35757ee21400b86a1c" exitCode=0 Oct 30 19:30:53.044998 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:53.044480 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvwqx" event={"ID":"89bcca43-c158-47fb-9cb9-1442abc2cd45","Type":"ContainerDied","Data":"c2dddcc3a8880575c112705ba19f68d6ed3363c7bc7eac35757ee21400b86a1c"} Oct 30 19:30:53.781847 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:53.781793 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:53.781847 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:53.781845 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:53.782056 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:53.781932 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:30:53.782056 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:53.782009 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:30:54.784437 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:54.784222 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:54.784900 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:54.784522 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p98lt" podUID="6c633f50-fd32-4440-aefc-79e44203424e" Oct 30 19:30:55.051906 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:55.051825 2579 generic.go:356] "Generic (PLEG): container finished" podID="89bcca43-c158-47fb-9cb9-1442abc2cd45" containerID="5cf01851dbd4be000a8b097af076fe8224184a348b076ccad6286b5adffabbc6" exitCode=0 Oct 30 19:30:55.051906 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:55.051887 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvwqx" event={"ID":"89bcca43-c158-47fb-9cb9-1442abc2cd45","Type":"ContainerDied","Data":"5cf01851dbd4be000a8b097af076fe8224184a348b076ccad6286b5adffabbc6"} Oct 30 19:30:55.781587 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:55.781555 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:55.781821 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:55.781555 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:55.781821 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:55.781681 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:30:55.781821 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:55.781735 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:30:56.353528 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:56.353341 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:56.354021 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:56.353490 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 19:30:56.354021 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:56.353616 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret podName:5bfb73f4-9290-4440-938a-feb9fafa3287 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.353601532 +0000 UTC m=+34.144675148 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret") pod "global-pull-secret-syncer-mm6ld" (UID: "5bfb73f4-9290-4440-938a-feb9fafa3287") : object "kube-system"/"original-pull-secret" not registered Oct 30 19:30:56.454793 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:56.454745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:56.454937 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:56.454808 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vjq\" (UniqueName: \"kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq\") pod \"network-check-target-p98lt\" (UID: \"6c633f50-fd32-4440-aefc-79e44203424e\") " pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:56.455005 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:56.454963 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 19:30:56.455005 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:56.454984 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 19:30:56.455005 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:56.454994 2579 projected.go:196] Error preparing data for projected volume kube-api-access-j9vjq for pod openshift-network-diagnostics/network-check-target-p98lt: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:30:56.455005 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:56.454994 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:30:56.455186 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:56.455047 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq podName:6c633f50-fd32-4440-aefc-79e44203424e nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.455030372 +0000 UTC m=+34.246104011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-j9vjq" (UniqueName: "kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq") pod "network-check-target-p98lt" (UID: "6c633f50-fd32-4440-aefc-79e44203424e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 19:30:56.455186 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:56.455063 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs podName:af2922a1-2b0f-4ee1-8ef3-4911d029e93d nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.455055233 +0000 UTC m=+34.246128850 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs") pod "network-metrics-daemon-t4fw7" (UID: "af2922a1-2b0f-4ee1-8ef3-4911d029e93d") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 19:30:56.784271 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:56.784243 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:56.784433 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:56.784361 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p98lt" podUID="6c633f50-fd32-4440-aefc-79e44203424e" Oct 30 19:30:57.062486 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:57.062454 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" event={"ID":"d95b4719-782e-47f3-9a7d-ea04268f82da","Type":"ContainerStarted","Data":"a71331dc741a0f1af6f961cd8cb14ccae54700f326a69a2fbfce7b79b8464e6c"} Oct 30 19:30:57.062645 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:57.062568 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" event={"ID":"d95b4719-782e-47f3-9a7d-ea04268f82da","Type":"ContainerStarted","Data":"ef7f2de7677b64bf1f87574ac671478e118ddabb65f228e590eb78eed22b67ca"} Oct 30 19:30:57.062645 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:57.062621 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" event={"ID":"d95b4719-782e-47f3-9a7d-ea04268f82da","Type":"ContainerStarted","Data":"8267ef5c5302a5f3179211f831e32750e5c36f013733f16a7daa106790c5598a"} Oct 30 19:30:57.062717 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:57.062667 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" event={"ID":"d95b4719-782e-47f3-9a7d-ea04268f82da","Type":"ContainerStarted","Data":"a28c0ea8c26e850ba7ae149f47dcf25d6d3f8c19a6c2f94146061e7b99d7baa3"} Oct 30 19:30:57.062717 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:57.062704 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" event={"ID":"d95b4719-782e-47f3-9a7d-ea04268f82da","Type":"ContainerStarted","Data":"4aacf54580bbe4b27d38e156fa9a9607cc914050dcfb2ef058c2e9d53a27346f"} Oct 30 19:30:57.062789 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:57.062746 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" event={"ID":"d95b4719-782e-47f3-9a7d-ea04268f82da","Type":"ContainerStarted","Data":"90c34f928706cb1a838b0d980b2897aef19f6b243e1daa385d19b15da9d8f3c1"} Oct 30 19:30:57.065033 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:57.065008 2579 generic.go:356] "Generic (PLEG): container finished" podID="89bcca43-c158-47fb-9cb9-1442abc2cd45" containerID="060c949aa60750f9348b948945a565e1a28e12014c2aa067ddfbee714d255af4" exitCode=0 Oct 30 19:30:57.065121 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:57.065048 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvwqx" event={"ID":"89bcca43-c158-47fb-9cb9-1442abc2cd45","Type":"ContainerDied","Data":"060c949aa60750f9348b948945a565e1a28e12014c2aa067ddfbee714d255af4"} Oct 30 19:30:57.781708 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:57.781672 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:57.781708 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:57.781714 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:57.782201 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:57.781796 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:30:57.782201 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:57.781941 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:30:58.069978 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:58.069887 2579 generic.go:356] "Generic (PLEG): container finished" podID="89bcca43-c158-47fb-9cb9-1442abc2cd45" containerID="e03d7e4cb0518deec2e709c10e25ee41f13b22fc2ff47e735e9f1e87448b2e9e" exitCode=0 Oct 30 19:30:58.069978 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:58.069935 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvwqx" event={"ID":"89bcca43-c158-47fb-9cb9-1442abc2cd45","Type":"ContainerDied","Data":"e03d7e4cb0518deec2e709c10e25ee41f13b22fc2ff47e735e9f1e87448b2e9e"} Oct 30 19:30:58.781704 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:58.781675 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:30:58.781863 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:58.781776 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p98lt" podUID="6c633f50-fd32-4440-aefc-79e44203424e" Oct 30 19:30:59.075625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:59.075543 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kvwqx" event={"ID":"89bcca43-c158-47fb-9cb9-1442abc2cd45","Type":"ContainerStarted","Data":"a0902122cda421aa684a4de6422b4b99f18261a3bcc962859794bc1c9c72a284"} Oct 30 19:30:59.078385 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:59.078362 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" event={"ID":"d95b4719-782e-47f3-9a7d-ea04268f82da","Type":"ContainerStarted","Data":"be28aa16f0c2f93e32ffba7a1a932c1979340713efb951aef20b90ca5ae43730"} Oct 30 19:30:59.105801 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:59.105756 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kvwqx" podStartSLOduration=8.583551914 podStartE2EDuration="20.105742354s" totalImagesPullingTime="11.52219044s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:39 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:30:59.104229205 +0000 UTC m=+20.895302844" watchObservedRunningTime="2026-10-30 19:30:59.105742354 +0000 UTC m=+20.896815996" Oct 30 19:30:59.781879 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:59.781838 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:30:59.781879 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:30:59.781887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:30:59.782307 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:59.781948 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:30:59.782307 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:30:59.782062 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:31:00.781923 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:00.781664 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:31:00.782456 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:00.781945 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p98lt" podUID="6c633f50-fd32-4440-aefc-79e44203424e" Oct 30 19:31:01.087769 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:01.087691 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" event={"ID":"d95b4719-782e-47f3-9a7d-ea04268f82da","Type":"ContainerStarted","Data":"9eea9de6137bbb4b1ead461089d6f4c3761188292c387a4ff8d9f71f7bbd554b"} Oct 30 19:31:01.087943 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:01.087925 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:01.088014 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:01.088002 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:01.103553 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:01.103521 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:01.103676 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:01.103633 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:01.122958 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:01.122913 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" podStartSLOduration=7.117599493 podStartE2EDuration="22.122901452s" totalImagesPullingTime="15.005301959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:01.122655057 +0000 UTC m=+22.913728695" watchObservedRunningTime="2026-10-30 19:31:01.122901452 +0000 UTC m=+22.913975091" Oct 30 19:31:01.781732 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:01.781698 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:31:01.781732 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:01.781732 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:01.781941 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:01.781845 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:31:01.782315 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:01.781939 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:31:02.090636 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:02.090566 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:31:02.395047 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:02.394974 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p98lt"] Oct 30 19:31:02.395304 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:02.395092 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:31:02.395304 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:02.395198 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p98lt" podUID="6c633f50-fd32-4440-aefc-79e44203424e" Oct 30 19:31:02.400886 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:02.400856 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t4fw7"] Oct 30 19:31:02.401011 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:02.400966 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:31:02.401109 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:02.401081 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:31:02.401423 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:02.401407 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mm6ld"] Oct 30 19:31:02.401511 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:02.401484 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:02.401595 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:02.401578 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:31:02.918474 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:02.918445 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:03.781564 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:03.781528 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:31:03.781729 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:03.781541 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:31:03.781729 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:03.781663 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:31:03.781729 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:03.781706 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:03.781896 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:03.781775 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p98lt" podUID="6c633f50-fd32-4440-aefc-79e44203424e" Oct 30 19:31:03.781896 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:03.781834 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:31:05.781520 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:05.781490 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:05.781992 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:05.781563 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:31:05.781992 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:05.781589 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:31:05.781992 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:05.781679 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p98lt" podUID="6c633f50-fd32-4440-aefc-79e44203424e" Oct 30 19:31:05.781992 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:05.781714 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:31:05.781992 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:05.781818 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:31:07.781052 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:07.781009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:31:07.781599 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:07.781009 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:31:07.781599 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:07.781138 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-p98lt" podUID="6c633f50-fd32-4440-aefc-79e44203424e" Oct 30 19:31:07.781599 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:07.781030 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:07.781599 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:07.781210 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-t4fw7" podUID="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" Oct 30 19:31:07.781599 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:07.781297 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mm6ld" podUID="5bfb73f4-9290-4440-938a-feb9fafa3287" Oct 30 19:31:08.991654 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:08.991442 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeReady" Oct 30 19:31:08.992000 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:08.991707 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 19:31:09.034881 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.034853 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pggqx"] Oct 30 19:31:09.038757 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.038739 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pggqx" Oct 30 19:31:09.041003 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.040959 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rtf4t"] Oct 30 19:31:09.045253 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.045170 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fkjv2"] Oct 30 19:31:09.045454 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.045428 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rtf4t" Oct 30 19:31:09.048716 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.048693 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 19:31:09.048820 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.048767 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 19:31:09.048820 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.048813 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb"] Oct 30 19:31:09.048980 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.048961 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fkjv2" Oct 30 19:31:09.052317 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.052295 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-787bddbb47-hfcvw"] Oct 30 19:31:09.052427 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.052400 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:09.055846 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.055829 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg"] Oct 30 19:31:09.055975 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.055960 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.059278 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.059262 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl"] Oct 30 19:31:09.059379 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.059363 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:09.062560 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.062539 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7jhj7"] Oct 30 19:31:09.062723 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.062703 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:09.067131 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.066264 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f6d8b54bc-kmdpv"] Oct 30 19:31:09.067131 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.066404 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.072973 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.072950 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp"] Oct 30 19:31:09.073097 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.073081 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.076383 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.076359 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vblv9"] Oct 30 19:31:09.076502 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.076429 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:09.077756 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.077735 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 19:31:09.077868 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.077783 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 19:31:09.077868 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.077831 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 19:31:09.077969 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.077903 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5j45v\"" Oct 30 19:31:09.077969 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.077906 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 19:31:09.077969 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.077938 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 19:31:09.078162 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.077736 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jflbh\"" Oct 30 19:31:09.078384 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.078366 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 19:31:09.079826 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.079806 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2q7gs\"" Oct 30 19:31:09.081010 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.080988 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-q99f8\"" Oct 30 19:31:09.081123 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.081044 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-4kpqr"] Oct 30 19:31:09.081972 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.081529 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vblv9" Oct 30 19:31:09.085716 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.085695 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-twdf7"] Oct 30 19:31:09.086464 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.086402 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:09.089820 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.089794 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-549665644b-9xg4z"] Oct 30 19:31:09.090825 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.090804 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:09.092953 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.092933 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-kzbbk"] Oct 30 19:31:09.093101 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.093084 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.100344 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.099937 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pggqx"] Oct 30 19:31:09.100344 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.100002 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rtf4t"] Oct 30 19:31:09.100344 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.100046 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fkjv2"] Oct 30 19:31:09.100344 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.100072 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-77ms2"] Oct 30 19:31:09.100957 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.100932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kzbbk" Oct 30 19:31:09.103554 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.103532 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-j2h9z"] Oct 30 19:31:09.103644 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.103634 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-77ms2" Oct 30 19:31:09.106676 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.106656 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg"] Oct 30 19:31:09.106778 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.106694 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl"] Oct 30 19:31:09.106778 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.106719 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7jhj7"] Oct 30 19:31:09.106778 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.106739 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-787bddbb47-hfcvw"] Oct 30 19:31:09.106778 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.106760 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp"] Oct 30 19:31:09.106970 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.106789 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-4kpqr"] Oct 30 19:31:09.106970 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.106808 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb"] Oct 30 19:31:09.106970 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.106831 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f6d8b54bc-kmdpv"] Oct 30 19:31:09.106970 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.106848 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j2h9z"] Oct 30 19:31:09.106970 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.106870 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-twdf7"] Oct 30 19:31:09.106970 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.106887 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-77ms2"] Oct 30 19:31:09.106970 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.106904 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vblv9"] Oct 30 19:31:09.106970 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.106921 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-549665644b-9xg4z"] Oct 30 19:31:09.106970 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.106938 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-kzbbk"] Oct 30 19:31:09.107506 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.107015 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:09.145599 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.145575 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/24ac5496-c680-4b2a-8362-f12b662df997-snapshots\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.145724 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.145606 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ac5496-c680-4b2a-8362-f12b662df997-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.145724 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.145625 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/02832de1-9584-4e12-9b94-8b6b52a71941-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:09.145724 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.145644 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c260520-55b0-44ca-8ccb-e15cab54fa90-config\") pod \"service-ca-operator-544d749c8c-dqnkb\" (UID: \"6c260520-55b0-44ca-8ccb-e15cab54fa90\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:09.145724 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.145699 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmz6\" (UniqueName: \"kubernetes.io/projected/6c260520-55b0-44ca-8ccb-e15cab54fa90-kube-api-access-dtmz6\") pod \"service-ca-operator-544d749c8c-dqnkb\" (UID: \"6c260520-55b0-44ca-8ccb-e15cab54fa90\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:09.145918 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.145760 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-default-certificate\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.145918 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.145809 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c260520-55b0-44ca-8ccb-e15cab54fa90-serving-cert\") pod \"service-ca-operator-544d749c8c-dqnkb\" (UID: \"6c260520-55b0-44ca-8ccb-e15cab54fa90\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:09.145918 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.145842 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvx5x\" (UniqueName: \"kubernetes.io/projected/24ac5496-c680-4b2a-8362-f12b662df997-kube-api-access-qvx5x\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.145918 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.145898 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-stats-auth\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.146098 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.145919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:09.146098 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.145942 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24ac5496-c680-4b2a-8362-f12b662df997-tmp\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.146098 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.145961 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41c1116a-2830-4ea1-aae5-3a052f073293-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg\" (UID: \"41c1116a-2830-4ea1-aae5-3a052f073293\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:09.146098 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.146067 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ac5496-c680-4b2a-8362-f12b662df997-service-ca-bundle\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.146247 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.146103 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfqd8\" (UniqueName: \"kubernetes.io/projected/ad103ddc-523c-4881-8526-00458255bb6c-kube-api-access-qfqd8\") pod \"kas-connection-checker-685895c7c9-fkjv2\" (UID: \"ad103ddc-523c-4881-8526-00458255bb6c\") " pod="kube-system/kas-connection-checker-685895c7c9-fkjv2" Oct 30 19:31:09.146247 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.146198 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.146247 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.146228 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrblb\" (UniqueName: \"kubernetes.io/projected/a8e91049-4a10-4472-922d-7b7875e7de3a-kube-api-access-rrblb\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.146340 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.146305 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-847qk\" (UniqueName: \"kubernetes.io/projected/06bbc899-e828-4a45-bd9a-3e703c4416e7-kube-api-access-847qk\") pod \"kas-connection-checker-685895c7c9-rtf4t\" (UID: \"06bbc899-e828-4a45-bd9a-3e703c4416e7\") " pod="kube-system/kas-connection-checker-685895c7c9-rtf4t" Oct 30 19:31:09.146340 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.146326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.146414 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.146349 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnpd5\" (UniqueName: \"kubernetes.io/projected/9458eb02-2921-4f34-9578-60056e3505a4-kube-api-access-cnpd5\") pod \"kas-connection-checker-685895c7c9-pggqx\" (UID: \"9458eb02-2921-4f34-9578-60056e3505a4\") " pod="kube-system/kas-connection-checker-685895c7c9-pggqx" Oct 30 19:31:09.146414 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.146374 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41c1116a-2830-4ea1-aae5-3a052f073293-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg\" (UID: \"41c1116a-2830-4ea1-aae5-3a052f073293\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:09.146414 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.146403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh66m\" (UniqueName: \"kubernetes.io/projected/41c1116a-2830-4ea1-aae5-3a052f073293-kube-api-access-rh66m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg\" (UID: \"41c1116a-2830-4ea1-aae5-3a052f073293\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:09.146557 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.146427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24ac5496-c680-4b2a-8362-f12b662df997-serving-cert\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.146557 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.146469 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dqtb\" (UniqueName: \"kubernetes.io/projected/02832de1-9584-4e12-9b94-8b6b52a71941-kube-api-access-2dqtb\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:09.148578 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.148559 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:31:09.148702 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.148605 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:31:09.148702 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.148603 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:31:09.148702 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.148690 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 19:31:09.148887 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.148774 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:31:09.148887 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.148782 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 19:31:09.148887 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.148812 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 19:31:09.149187 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.149169 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:31:09.149262 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.149210 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:31:09.149469 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.149450 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 19:31:09.149547 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.149492 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 19:31:09.149601 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.149590 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:31:09.149673 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.149661 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 19:31:09.149726 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.149717 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 19:31:09.149767 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.149758 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:31:09.149820 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.149811 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:31:09.149879 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.149870 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 19:31:09.149935 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.149925 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:31:09.149970 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.149955 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:31:09.150248 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.150229 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:31:09.150317 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.150245 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:31:09.150488 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.150251 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:31:09.150488 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.150270 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:31:09.150488 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.150291 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 19:31:09.150625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.150334 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 19:31:09.150625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.150340 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 19:31:09.150625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.150393 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 19:31:09.150625 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.150445 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:31:09.155434 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.155383 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 19:31:09.158174 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.158138 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 19:31:09.158297 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.158280 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 19:31:09.158335 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.158301 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 19:31:09.170886 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.170861 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 19:31:09.171166 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.171135 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 19:31:09.171259 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.171248 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 19:31:09.171431 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.171419 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:31:09.171502 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.171479 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:31:09.171577 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.171566 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-52tnm\"" Oct 30 19:31:09.171623 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.171612 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-n522r\"" Oct 30 19:31:09.171670 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.171645 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-64274\"" Oct 30 19:31:09.171705 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.171699 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9h8tn\"" Oct 30 19:31:09.171779 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.171769 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h46gl\"" Oct 30 19:31:09.171830 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.171807 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 19:31:09.171888 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.171878 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dvfnr\"" Oct 30 19:31:09.171920 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.171915 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 19:31:09.171982 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.171974 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-scxl6\"" Oct 30 19:31:09.172027 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.172018 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 19:31:09.172096 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.172086 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 19:31:09.172171 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.172160 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dp4dn\"" Oct 30 19:31:09.172315 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.172301 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-t79gm\"" Oct 30 19:31:09.247797 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.247769 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-trusted-ca\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.247797 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.247801 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6e13582a-7d64-443a-a62f-c14ecc655f18-nginx-conf\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:09.248034 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.247832 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847qk\" (UniqueName: \"kubernetes.io/projected/06bbc899-e828-4a45-bd9a-3e703c4416e7-kube-api-access-847qk\") pod \"kas-connection-checker-685895c7c9-rtf4t\" (UID: \"06bbc899-e828-4a45-bd9a-3e703c4416e7\") " pod="kube-system/kas-connection-checker-685895c7c9-rtf4t" Oct 30 19:31:09.248034 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.247850 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.248034 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.247917 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/731f78f9-9938-4170-a8ff-10ba7a863759-ca-trust-extracted\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.248034 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.247969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhzl5\" (UniqueName: \"kubernetes.io/projected/6693a492-433f-4614-a8e3-71ee75c177c2-kube-api-access-jhzl5\") pod \"volume-data-source-validator-58c9f5f46d-vblv9\" (UID: \"6693a492-433f-4614-a8e3-71ee75c177c2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vblv9" Oct 30 19:31:09.248196 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248036 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.248196 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.248160 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:31:09.248253 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.248224 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs podName:a8e91049-4a10-4472-922d-7b7875e7de3a nodeName:}" failed. No retries permitted until 2026-10-30 19:31:09.748209228 +0000 UTC m=+31.539282859 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs") pod "router-default-787bddbb47-hfcvw" (UID: "a8e91049-4a10-4472-922d-7b7875e7de3a") : secret "router-metrics-certs-default" not found Oct 30 19:31:09.248253 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248239 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-bound-sa-token\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.248329 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248265 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhvp8\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-kube-api-access-rhvp8\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.248365 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/24ac5496-c680-4b2a-8362-f12b662df997-snapshots\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.248399 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248370 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ac5496-c680-4b2a-8362-f12b662df997-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.248475 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248458 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/067a4436-690d-4b2c-aaf2-1d84830b7287-serving-cert\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:09.248535 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248500 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:09.248592 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248528 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-installation-pull-secrets\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.248592 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248584 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c260520-55b0-44ca-8ccb-e15cab54fa90-config\") pod \"service-ca-operator-544d749c8c-dqnkb\" (UID: \"6c260520-55b0-44ca-8ccb-e15cab54fa90\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:09.248694 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmz6\" (UniqueName: \"kubernetes.io/projected/6c260520-55b0-44ca-8ccb-e15cab54fa90-kube-api-access-dtmz6\") pod \"service-ca-operator-544d749c8c-dqnkb\" (UID: \"6c260520-55b0-44ca-8ccb-e15cab54fa90\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:09.248694 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248641 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8161b0bd-5184-44fc-864e-869dc7737a55-ca-trust-extracted\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.248694 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-default-certificate\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.248832 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248693 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzlcx\" (UniqueName: \"kubernetes.io/projected/b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad-kube-api-access-bzlcx\") pod \"network-check-source-6f78d59f99-kzbbk\" (UID: \"b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kzbbk" Oct 30 19:31:09.248832 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248743 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-image-registry-private-configuration\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.248930 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-trusted-ca\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.248930 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvx5x\" (UniqueName: \"kubernetes.io/projected/24ac5496-c680-4b2a-8362-f12b662df997-kube-api-access-qvx5x\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.249019 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248925 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/24ac5496-c680-4b2a-8362-f12b662df997-snapshots\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.249019 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.248968 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/067a4436-690d-4b2c-aaf2-1d84830b7287-trusted-ca\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:09.249112 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:09.249291 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.249271 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:31:09.249417 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24ac5496-c680-4b2a-8362-f12b662df997-tmp\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.249417 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249386 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c260520-55b0-44ca-8ccb-e15cab54fa90-config\") pod \"service-ca-operator-544d749c8c-dqnkb\" (UID: \"6c260520-55b0-44ca-8ccb-e15cab54fa90\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:09.249538 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.249438 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls podName:02832de1-9584-4e12-9b94-8b6b52a71941 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:09.749420435 +0000 UTC m=+31.540494072 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fcjbl" (UID: "02832de1-9584-4e12-9b94-8b6b52a71941") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:31:09.249538 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/067a4436-690d-4b2c-aaf2-1d84830b7287-config\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:09.249679 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249535 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01188e4b-c110-477a-ba88-d9f48741c0a2-config-volume\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:09.249679 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249598 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41c1116a-2830-4ea1-aae5-3a052f073293-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg\" (UID: \"41c1116a-2830-4ea1-aae5-3a052f073293\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:09.249679 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249603 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24ac5496-c680-4b2a-8362-f12b662df997-tmp\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.249679 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-installation-pull-secrets\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.249848 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:09.249848 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249692 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ac5496-c680-4b2a-8362-f12b662df997-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.249848 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.249848 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249763 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrblb\" (UniqueName: \"kubernetes.io/projected/a8e91049-4a10-4472-922d-7b7875e7de3a-kube-api-access-rrblb\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.249848 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249790 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql7gq\" (UniqueName: \"kubernetes.io/projected/534aacd9-7102-4a50-8696-c50362ce87e4-kube-api-access-ql7gq\") pod \"ingress-canary-77ms2\" (UID: \"534aacd9-7102-4a50-8696-c50362ce87e4\") " pod="openshift-ingress-canary/ingress-canary-77ms2" Oct 30 19:31:09.249848 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249815 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01188e4b-c110-477a-ba88-d9f48741c0a2-tmp-dir\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:09.250085 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249852 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-bound-sa-token\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.250085 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249881 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cvnm\" (UniqueName: \"kubernetes.io/projected/067a4436-690d-4b2c-aaf2-1d84830b7287-kube-api-access-5cvnm\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:09.250085 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249903 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/534aacd9-7102-4a50-8696-c50362ce87e4-cert\") pod \"ingress-canary-77ms2\" (UID: \"534aacd9-7102-4a50-8696-c50362ce87e4\") " pod="openshift-ingress-canary/ingress-canary-77ms2" Oct 30 19:31:09.250085 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.249998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41c1116a-2830-4ea1-aae5-3a052f073293-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg\" (UID: \"41c1116a-2830-4ea1-aae5-3a052f073293\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:09.250283 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.250158 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnpd5\" (UniqueName: \"kubernetes.io/projected/9458eb02-2921-4f34-9578-60056e3505a4-kube-api-access-cnpd5\") pod \"kas-connection-checker-685895c7c9-pggqx\" (UID: \"9458eb02-2921-4f34-9578-60056e3505a4\") " pod="kube-system/kas-connection-checker-685895c7c9-pggqx" Oct 30 19:31:09.250283 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.250191 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh66m\" (UniqueName: \"kubernetes.io/projected/41c1116a-2830-4ea1-aae5-3a052f073293-kube-api-access-rh66m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg\" (UID: \"41c1116a-2830-4ea1-aae5-3a052f073293\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:09.250283 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.250270 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24ac5496-c680-4b2a-8362-f12b662df997-serving-cert\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.250407 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.250308 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dqtb\" (UniqueName: \"kubernetes.io/projected/02832de1-9584-4e12-9b94-8b6b52a71941-kube-api-access-2dqtb\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:09.250407 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.250340 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-image-registry-private-configuration\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.250480 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.250427 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.250480 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.250462 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/02832de1-9584-4e12-9b94-8b6b52a71941-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:09.250552 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.250491 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-registry-certificates\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.250586 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.250566 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle podName:a8e91049-4a10-4472-922d-7b7875e7de3a nodeName:}" failed. No retries permitted until 2026-10-30 19:31:09.750549757 +0000 UTC m=+31.541623402 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle") pod "router-default-787bddbb47-hfcvw" (UID: "a8e91049-4a10-4472-922d-7b7875e7de3a") : configmap references non-existent config key: service-ca.crt Oct 30 19:31:09.250586 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.250577 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41c1116a-2830-4ea1-aae5-3a052f073293-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg\" (UID: \"41c1116a-2830-4ea1-aae5-3a052f073293\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:09.250843 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.250822 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c260520-55b0-44ca-8ccb-e15cab54fa90-serving-cert\") pod \"service-ca-operator-544d749c8c-dqnkb\" (UID: \"6c260520-55b0-44ca-8ccb-e15cab54fa90\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:09.250904 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.250889 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-registry-certificates\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.251069 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.251042 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:09.251141 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.251108 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-stats-auth\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.251208 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.251138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbtrc\" (UniqueName: \"kubernetes.io/projected/7d903311-c71c-4197-8ac3-486702bd46e9-kube-api-access-qbtrc\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:09.251245 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.251200 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ac5496-c680-4b2a-8362-f12b662df997-service-ca-bundle\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.251295 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.251241 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvxqn\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-kube-api-access-jvxqn\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.251295 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.251267 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:09.251404 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.251300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfqd8\" (UniqueName: \"kubernetes.io/projected/ad103ddc-523c-4881-8526-00458255bb6c-kube-api-access-qfqd8\") pod \"kas-connection-checker-685895c7c9-fkjv2\" (UID: \"ad103ddc-523c-4881-8526-00458255bb6c\") " pod="kube-system/kas-connection-checker-685895c7c9-fkjv2" Oct 30 19:31:09.251404 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.251333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ghq4\" (UniqueName: \"kubernetes.io/projected/01188e4b-c110-477a-ba88-d9f48741c0a2-kube-api-access-5ghq4\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:09.251507 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.251399 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/02832de1-9584-4e12-9b94-8b6b52a71941-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:09.251756 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.251730 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ac5496-c680-4b2a-8362-f12b662df997-service-ca-bundle\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.253761 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.253737 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24ac5496-c680-4b2a-8362-f12b662df997-serving-cert\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.253876 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.253736 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c260520-55b0-44ca-8ccb-e15cab54fa90-serving-cert\") pod \"service-ca-operator-544d749c8c-dqnkb\" (UID: \"6c260520-55b0-44ca-8ccb-e15cab54fa90\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:09.253876 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.253772 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41c1116a-2830-4ea1-aae5-3a052f073293-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg\" (UID: \"41c1116a-2830-4ea1-aae5-3a052f073293\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:09.253992 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.253975 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-stats-auth\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.254034 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.254009 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-default-certificate\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.261262 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.261234 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-847qk\" (UniqueName: \"kubernetes.io/projected/06bbc899-e828-4a45-bd9a-3e703c4416e7-kube-api-access-847qk\") pod \"kas-connection-checker-685895c7c9-rtf4t\" (UID: \"06bbc899-e828-4a45-bd9a-3e703c4416e7\") " pod="kube-system/kas-connection-checker-685895c7c9-rtf4t" Oct 30 19:31:09.263834 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.263806 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh66m\" (UniqueName: \"kubernetes.io/projected/41c1116a-2830-4ea1-aae5-3a052f073293-kube-api-access-rh66m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg\" (UID: \"41c1116a-2830-4ea1-aae5-3a052f073293\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:09.264013 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.263995 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnpd5\" (UniqueName: \"kubernetes.io/projected/9458eb02-2921-4f34-9578-60056e3505a4-kube-api-access-cnpd5\") pod \"kas-connection-checker-685895c7c9-pggqx\" (UID: \"9458eb02-2921-4f34-9578-60056e3505a4\") " pod="kube-system/kas-connection-checker-685895c7c9-pggqx" Oct 30 19:31:09.264200 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.264178 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmz6\" (UniqueName: \"kubernetes.io/projected/6c260520-55b0-44ca-8ccb-e15cab54fa90-kube-api-access-dtmz6\") pod \"service-ca-operator-544d749c8c-dqnkb\" (UID: \"6c260520-55b0-44ca-8ccb-e15cab54fa90\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:09.264352 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.264335 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrblb\" (UniqueName: \"kubernetes.io/projected/a8e91049-4a10-4472-922d-7b7875e7de3a-kube-api-access-rrblb\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.264653 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.264634 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvx5x\" (UniqueName: \"kubernetes.io/projected/24ac5496-c680-4b2a-8362-f12b662df997-kube-api-access-qvx5x\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.264725 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.264710 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dqtb\" (UniqueName: \"kubernetes.io/projected/02832de1-9584-4e12-9b94-8b6b52a71941-kube-api-access-2dqtb\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:09.264932 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.264912 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfqd8\" (UniqueName: \"kubernetes.io/projected/ad103ddc-523c-4881-8526-00458255bb6c-kube-api-access-qfqd8\") pod \"kas-connection-checker-685895c7c9-fkjv2\" (UID: \"ad103ddc-523c-4881-8526-00458255bb6c\") " pod="kube-system/kas-connection-checker-685895c7c9-fkjv2" Oct 30 19:31:09.352323 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352290 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ghq4\" (UniqueName: \"kubernetes.io/projected/01188e4b-c110-477a-ba88-d9f48741c0a2-kube-api-access-5ghq4\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:09.352323 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352327 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-trusted-ca\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.352578 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352343 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6e13582a-7d64-443a-a62f-c14ecc655f18-nginx-conf\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:09.352578 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352379 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.352578 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/731f78f9-9938-4170-a8ff-10ba7a863759-ca-trust-extracted\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.352578 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352412 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhzl5\" (UniqueName: \"kubernetes.io/projected/6693a492-433f-4614-a8e3-71ee75c177c2-kube-api-access-jhzl5\") pod \"volume-data-source-validator-58c9f5f46d-vblv9\" (UID: \"6693a492-433f-4614-a8e3-71ee75c177c2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vblv9" Oct 30 19:31:09.352578 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352445 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-bound-sa-token\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.352578 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352469 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvp8\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-kube-api-access-rhvp8\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.352578 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/067a4436-690d-4b2c-aaf2-1d84830b7287-serving-cert\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:09.352578 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:09.352578 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352568 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-installation-pull-secrets\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.353099 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352600 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8161b0bd-5184-44fc-864e-869dc7737a55-ca-trust-extracted\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.353099 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352627 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlcx\" (UniqueName: \"kubernetes.io/projected/b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad-kube-api-access-bzlcx\") pod \"network-check-source-6f78d59f99-kzbbk\" (UID: \"b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kzbbk" Oct 30 19:31:09.353099 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352652 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-image-registry-private-configuration\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.353099 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352678 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-trusted-ca\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.353099 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352703 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/067a4436-690d-4b2c-aaf2-1d84830b7287-trusted-ca\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:09.353099 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/067a4436-690d-4b2c-aaf2-1d84830b7287-config\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:09.353099 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352767 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01188e4b-c110-477a-ba88-d9f48741c0a2-config-volume\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:09.353099 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352797 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-installation-pull-secrets\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.353099 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:09.353099 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352883 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql7gq\" (UniqueName: \"kubernetes.io/projected/534aacd9-7102-4a50-8696-c50362ce87e4-kube-api-access-ql7gq\") pod \"ingress-canary-77ms2\" (UID: \"534aacd9-7102-4a50-8696-c50362ce87e4\") " pod="openshift-ingress-canary/ingress-canary-77ms2" Oct 30 19:31:09.353099 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01188e4b-c110-477a-ba88-d9f48741c0a2-tmp-dir\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:09.353099 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352944 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-bound-sa-token\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.353099 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352975 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvnm\" (UniqueName: \"kubernetes.io/projected/067a4436-690d-4b2c-aaf2-1d84830b7287-kube-api-access-5cvnm\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:09.353099 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352989 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/731f78f9-9938-4170-a8ff-10ba7a863759-ca-trust-extracted\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.353099 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.352998 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/534aacd9-7102-4a50-8696-c50362ce87e4-cert\") pod \"ingress-canary-77ms2\" (UID: \"534aacd9-7102-4a50-8696-c50362ce87e4\") " pod="openshift-ingress-canary/ingress-canary-77ms2" Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.353078 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-image-registry-private-configuration\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.353188 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.353265 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6e13582a-7d64-443a-a62f-c14ecc655f18-nginx-conf\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.353277 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-registry-certificates\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.353281 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-trusted-ca\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.353323 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-registry-certificates\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.353362 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.353368 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8161b0bd-5184-44fc-864e-869dc7737a55-ca-trust-extracted\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.353388 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.353395 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbtrc\" (UniqueName: \"kubernetes.io/projected/7d903311-c71c-4197-8ac3-486702bd46e9-kube-api-access-qbtrc\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.353404 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-549665644b-9xg4z: secret "image-registry-tls" not found Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.353435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvxqn\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-kube-api-access-jvxqn\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.353452 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.353572 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls podName:8161b0bd-5184-44fc-864e-869dc7737a55 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:09.853447419 +0000 UTC m=+31.644521055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls") pod "image-registry-549665644b-9xg4z" (UID: "8161b0bd-5184-44fc-864e-869dc7737a55") : secret "image-registry-tls" not found Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.353393 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:31:09.353823 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.353590 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f6d8b54bc-kmdpv: secret "image-registry-tls" not found Oct 30 19:31:09.354570 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.353840 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-registry-certificates\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.354570 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.353841 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/067a4436-690d-4b2c-aaf2-1d84830b7287-trusted-ca\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:09.354570 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.353932 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls podName:01188e4b-c110-477a-ba88-d9f48741c0a2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:09.853915601 +0000 UTC m=+31.644989230 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls") pod "dns-default-j2h9z" (UID: "01188e4b-c110-477a-ba88-d9f48741c0a2") : secret "dns-default-metrics-tls" not found Oct 30 19:31:09.354570 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.353958 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls podName:731f78f9-9938-4170-a8ff-10ba7a863759 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:09.853948413 +0000 UTC m=+31.645022032 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls") pod "image-registry-7f6d8b54bc-kmdpv" (UID: "731f78f9-9938-4170-a8ff-10ba7a863759") : secret "image-registry-tls" not found Oct 30 19:31:09.354570 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.353982 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:09.354570 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.354093 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/067a4436-690d-4b2c-aaf2-1d84830b7287-config\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:09.354570 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.354120 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-registry-certificates\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.354570 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.354125 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01188e4b-c110-477a-ba88-d9f48741c0a2-tmp-dir\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:09.354570 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.354228 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:31:09.354570 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.354269 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls podName:7d903311-c71c-4197-8ac3-486702bd46e9 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:09.854256471 +0000 UTC m=+31.645330102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-b8bsp" (UID: "7d903311-c71c-4197-8ac3-486702bd46e9") : secret "samples-operator-tls" not found Oct 30 19:31:09.354570 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.354551 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:31:09.355107 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.354606 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert podName:6e13582a-7d64-443a-a62f-c14ecc655f18 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:09.854592766 +0000 UTC m=+31.645666400 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-twdf7" (UID: "6e13582a-7d64-443a-a62f-c14ecc655f18") : secret "networking-console-plugin-cert" not found Oct 30 19:31:09.355107 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.354673 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-pggqx" Oct 30 19:31:09.355107 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.354983 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01188e4b-c110-477a-ba88-d9f48741c0a2-config-volume\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:09.355298 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.355250 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:09.355570 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.355548 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-trusted-ca\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.356246 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.356208 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-image-registry-private-configuration\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.356869 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.356848 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-image-registry-private-configuration\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.356971 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.356925 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-installation-pull-secrets\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.357136 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.357119 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/534aacd9-7102-4a50-8696-c50362ce87e4-cert\") pod \"ingress-canary-77ms2\" (UID: \"534aacd9-7102-4a50-8696-c50362ce87e4\") " pod="openshift-ingress-canary/ingress-canary-77ms2" Oct 30 19:31:09.357385 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.357363 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/067a4436-690d-4b2c-aaf2-1d84830b7287-serving-cert\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:09.358541 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.358522 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-installation-pull-secrets\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.360691 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.360678 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-rtf4t" Oct 30 19:31:09.369679 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.369645 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-fkjv2" Oct 30 19:31:09.377623 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.377597 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:09.378388 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.378363 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvxqn\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-kube-api-access-jvxqn\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.379922 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.379897 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-bound-sa-token\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.380454 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.380423 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-bound-sa-token\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.380636 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.380607 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbtrc\" (UniqueName: \"kubernetes.io/projected/7d903311-c71c-4197-8ac3-486702bd46e9-kube-api-access-qbtrc\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:09.380716 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.380635 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ghq4\" (UniqueName: \"kubernetes.io/projected/01188e4b-c110-477a-ba88-d9f48741c0a2-kube-api-access-5ghq4\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:09.380716 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.380685 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhzl5\" (UniqueName: \"kubernetes.io/projected/6693a492-433f-4614-a8e3-71ee75c177c2-kube-api-access-jhzl5\") pod \"volume-data-source-validator-58c9f5f46d-vblv9\" (UID: \"6693a492-433f-4614-a8e3-71ee75c177c2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vblv9" Oct 30 19:31:09.380716 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.380683 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvp8\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-kube-api-access-rhvp8\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.381136 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.381105 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvnm\" (UniqueName: \"kubernetes.io/projected/067a4436-690d-4b2c-aaf2-1d84830b7287-kube-api-access-5cvnm\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:09.381241 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.381219 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql7gq\" (UniqueName: \"kubernetes.io/projected/534aacd9-7102-4a50-8696-c50362ce87e4-kube-api-access-ql7gq\") pod \"ingress-canary-77ms2\" (UID: \"534aacd9-7102-4a50-8696-c50362ce87e4\") " pod="openshift-ingress-canary/ingress-canary-77ms2" Oct 30 19:31:09.381316 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.381294 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlcx\" (UniqueName: \"kubernetes.io/projected/b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad-kube-api-access-bzlcx\") pod \"network-check-source-6f78d59f99-kzbbk\" (UID: \"b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kzbbk" Oct 30 19:31:09.399474 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.399434 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:09.417264 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.417238 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:09.434655 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.434630 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vblv9" Oct 30 19:31:09.447723 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.444092 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:09.465313 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.463377 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kzbbk" Oct 30 19:31:09.468572 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.468377 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-77ms2" Oct 30 19:31:09.666733 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.666644 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-rtf4t"] Oct 30 19:31:09.667722 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.667702 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-fkjv2"] Oct 30 19:31:09.669685 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.669660 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb"] Oct 30 19:31:09.670313 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:31:09.670241 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad103ddc_523c_4881_8526_00458255bb6c.slice/crio-db6295ed5bfdd69e1ede730aeec1442244a828472e1c56df5afc8724f5307958 WatchSource:0}: Error finding container db6295ed5bfdd69e1ede730aeec1442244a828472e1c56df5afc8724f5307958: Status 404 returned error can't find the container with id db6295ed5bfdd69e1ede730aeec1442244a828472e1c56df5afc8724f5307958 Oct 30 19:31:09.670740 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:31:09.670702 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bbc899_e828_4a45_bd9a_3e703c4416e7.slice/crio-5deed405f40cce56eb0d6e42ae157a656bea9637c3f425661d727b051239159f WatchSource:0}: Error finding container 5deed405f40cce56eb0d6e42ae157a656bea9637c3f425661d727b051239159f: Status 404 returned error can't find the container with id 5deed405f40cce56eb0d6e42ae157a656bea9637c3f425661d727b051239159f Oct 30 19:31:09.672217 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:31:09.672137 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c260520_55b0_44ca_8ccb_e15cab54fa90.slice/crio-c50974d77e3047f90b03bdb6069b22f656f65aff6ae918799c0669b692cf5e79 WatchSource:0}: Error finding container c50974d77e3047f90b03bdb6069b22f656f65aff6ae918799c0669b692cf5e79: Status 404 returned error can't find the container with id c50974d77e3047f90b03bdb6069b22f656f65aff6ae918799c0669b692cf5e79 Oct 30 19:31:09.673032 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.672971 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-pggqx"] Oct 30 19:31:09.675563 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:31:09.675546 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9458eb02_2921_4f34_9578_60056e3505a4.slice/crio-c722584061fea70d48ce3aaddd579e3b67277faa4f0b3800de3bb2e73c882fb6 WatchSource:0}: Error finding container c722584061fea70d48ce3aaddd579e3b67277faa4f0b3800de3bb2e73c882fb6: Status 404 returned error can't find the container with id c722584061fea70d48ce3aaddd579e3b67277faa4f0b3800de3bb2e73c882fb6 Oct 30 19:31:09.716051 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.715825 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg"] Oct 30 19:31:09.719293 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:31:09.719250 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41c1116a_2830_4ea1_aae5_3a052f073293.slice/crio-b348f37bb5e67bf1200b8567986992f023d9f6f3e021d4a1004dae81ef5fc873 WatchSource:0}: Error finding container b348f37bb5e67bf1200b8567986992f023d9f6f3e021d4a1004dae81ef5fc873: Status 404 returned error can't find the container with id b348f37bb5e67bf1200b8567986992f023d9f6f3e021d4a1004dae81ef5fc873 Oct 30 19:31:09.764960 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.764868 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-7jhj7"] Oct 30 19:31:09.767216 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.765350 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.767216 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.765438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:09.767216 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.765494 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:09.767216 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.765759 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle podName:a8e91049-4a10-4472-922d-7b7875e7de3a nodeName:}" failed. No retries permitted until 2026-10-30 19:31:10.765740365 +0000 UTC m=+32.556814000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle") pod "router-default-787bddbb47-hfcvw" (UID: "a8e91049-4a10-4472-922d-7b7875e7de3a") : configmap references non-existent config key: service-ca.crt Oct 30 19:31:09.767216 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.766190 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:31:09.767216 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.766242 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs podName:a8e91049-4a10-4472-922d-7b7875e7de3a nodeName:}" failed. No retries permitted until 2026-10-30 19:31:10.766225551 +0000 UTC m=+32.557299186 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs") pod "router-default-787bddbb47-hfcvw" (UID: "a8e91049-4a10-4472-922d-7b7875e7de3a") : secret "router-metrics-certs-default" not found Oct 30 19:31:09.767216 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.766303 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:31:09.767216 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.766339 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls podName:02832de1-9584-4e12-9b94-8b6b52a71941 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:10.766328528 +0000 UTC m=+32.557402160 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fcjbl" (UID: "02832de1-9584-4e12-9b94-8b6b52a71941") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:31:09.769707 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.769691 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vblv9"] Oct 30 19:31:09.771269 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.771248 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-4kpqr"] Oct 30 19:31:09.778352 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:31:09.778325 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod067a4436_690d_4b2c_aaf2_1d84830b7287.slice/crio-515e6a514b5bac59facab7b59d56084b49d1112f3e585cb8f258afd2c416e556 WatchSource:0}: Error finding container 515e6a514b5bac59facab7b59d56084b49d1112f3e585cb8f258afd2c416e556: Status 404 returned error can't find the container with id 515e6a514b5bac59facab7b59d56084b49d1112f3e585cb8f258afd2c416e556 Oct 30 19:31:09.781833 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.781812 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:09.781926 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.781900 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:31:09.782090 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.782074 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:31:09.787381 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.787365 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rpkw7\"" Oct 30 19:31:09.787447 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.787417 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wr97x\"" Oct 30 19:31:09.787447 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.787428 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jznq8\"" Oct 30 19:31:09.787520 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.787447 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:31:09.787554 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.787523 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:31:09.823953 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.823821 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-kzbbk"] Oct 30 19:31:09.826546 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.826519 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-77ms2"] Oct 30 19:31:09.850226 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:31:09.850182 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod534aacd9_7102_4a50_8696_c50362ce87e4.slice/crio-a6161bc1168b86cd57c41a6ff433a7fd1aaf76078435b1e6134db395ecfad2d5 WatchSource:0}: Error finding container a6161bc1168b86cd57c41a6ff433a7fd1aaf76078435b1e6134db395ecfad2d5: Status 404 returned error can't find the container with id a6161bc1168b86cd57c41a6ff433a7fd1aaf76078435b1e6134db395ecfad2d5 Oct 30 19:31:09.850429 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:31:09.850408 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5ecde06_7797_4df8_8cf8_1b30a5f0a8ad.slice/crio-97a89a6ee7741dfeefcac263c266168d14613ae7ff6fcd0bbfd2d3badc97cdb4 WatchSource:0}: Error finding container 97a89a6ee7741dfeefcac263c266168d14613ae7ff6fcd0bbfd2d3badc97cdb4: Status 404 returned error can't find the container with id 97a89a6ee7741dfeefcac263c266168d14613ae7ff6fcd0bbfd2d3badc97cdb4 Oct 30 19:31:09.866613 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.866587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:09.866780 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.866640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:09.866780 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.866737 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:31:09.866780 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.866760 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:31:09.866924 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.866789 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls podName:01188e4b-c110-477a-ba88-d9f48741c0a2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:10.866772407 +0000 UTC m=+32.657846041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls") pod "dns-default-j2h9z" (UID: "01188e4b-c110-477a-ba88-d9f48741c0a2") : secret "dns-default-metrics-tls" not found Oct 30 19:31:09.866924 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.866850 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:09.866924 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.866906 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert podName:6e13582a-7d64-443a-a62f-c14ecc655f18 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:10.866883631 +0000 UTC m=+32.657957258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-twdf7" (UID: "6e13582a-7d64-443a-a62f-c14ecc655f18") : secret "networking-console-plugin-cert" not found Oct 30 19:31:09.867077 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.866938 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:09.867077 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:09.867018 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:09.867077 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.866943 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:31:09.867245 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.867079 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-549665644b-9xg4z: secret "image-registry-tls" not found Oct 30 19:31:09.867245 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.867116 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls podName:8161b0bd-5184-44fc-864e-869dc7737a55 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:10.867105906 +0000 UTC m=+32.658179547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls") pod "image-registry-549665644b-9xg4z" (UID: "8161b0bd-5184-44fc-864e-869dc7737a55") : secret "image-registry-tls" not found Oct 30 19:31:09.867245 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.866990 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:31:09.867245 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.867141 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:31:09.867245 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.867177 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f6d8b54bc-kmdpv: secret "image-registry-tls" not found Oct 30 19:31:09.867245 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.867190 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls podName:7d903311-c71c-4197-8ac3-486702bd46e9 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:10.867174563 +0000 UTC m=+32.658248183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-b8bsp" (UID: "7d903311-c71c-4197-8ac3-486702bd46e9") : secret "samples-operator-tls" not found Oct 30 19:31:09.867245 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:09.867214 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls podName:731f78f9-9938-4170-a8ff-10ba7a863759 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:10.867202602 +0000 UTC m=+32.658276232 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls") pod "image-registry-7f6d8b54bc-kmdpv" (UID: "731f78f9-9938-4170-a8ff-10ba7a863759") : secret "image-registry-tls" not found Oct 30 19:31:10.109567 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.109486 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-77ms2" event={"ID":"534aacd9-7102-4a50-8696-c50362ce87e4","Type":"ContainerStarted","Data":"a6161bc1168b86cd57c41a6ff433a7fd1aaf76078435b1e6134db395ecfad2d5"} Oct 30 19:31:10.110651 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.110627 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vblv9" event={"ID":"6693a492-433f-4614-a8e3-71ee75c177c2","Type":"ContainerStarted","Data":"ee2cb1522ccc3e004a8be9b1aaa2a11a50e3cba75920aee3076e57dd2dc664f6"} Oct 30 19:31:10.111691 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.111669 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" event={"ID":"067a4436-690d-4b2c-aaf2-1d84830b7287","Type":"ContainerStarted","Data":"515e6a514b5bac59facab7b59d56084b49d1112f3e585cb8f258afd2c416e556"} Oct 30 19:31:10.112628 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.112607 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7jhj7" event={"ID":"24ac5496-c680-4b2a-8362-f12b662df997","Type":"ContainerStarted","Data":"013f0924a531f605911690a8934cbfa44dc6804bf0631ef95546eda783d9e48f"} Oct 30 19:31:10.113657 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.113640 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" event={"ID":"41c1116a-2830-4ea1-aae5-3a052f073293","Type":"ContainerStarted","Data":"b348f37bb5e67bf1200b8567986992f023d9f6f3e021d4a1004dae81ef5fc873"} Oct 30 19:31:10.115112 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.115090 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rtf4t" event={"ID":"06bbc899-e828-4a45-bd9a-3e703c4416e7","Type":"ContainerStarted","Data":"bec115eff6227a5627de284e867b411a60b9db7ef7bb4c4f631da338a54ea756"} Oct 30 19:31:10.115234 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.115129 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-rtf4t" event={"ID":"06bbc899-e828-4a45-bd9a-3e703c4416e7","Type":"ContainerStarted","Data":"5deed405f40cce56eb0d6e42ae157a656bea9637c3f425661d727b051239159f"} Oct 30 19:31:10.116241 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.116223 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kzbbk" event={"ID":"b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad","Type":"ContainerStarted","Data":"97a89a6ee7741dfeefcac263c266168d14613ae7ff6fcd0bbfd2d3badc97cdb4"} Oct 30 19:31:10.117247 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.117229 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" event={"ID":"6c260520-55b0-44ca-8ccb-e15cab54fa90","Type":"ContainerStarted","Data":"c50974d77e3047f90b03bdb6069b22f656f65aff6ae918799c0669b692cf5e79"} Oct 30 19:31:10.118485 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.118465 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fkjv2" event={"ID":"ad103ddc-523c-4881-8526-00458255bb6c","Type":"ContainerStarted","Data":"20416d7d19a7c8986fa076e343b962f31e772b661bab439019155d979650740c"} Oct 30 19:31:10.118559 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.118500 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-fkjv2" event={"ID":"ad103ddc-523c-4881-8526-00458255bb6c","Type":"ContainerStarted","Data":"db6295ed5bfdd69e1ede730aeec1442244a828472e1c56df5afc8724f5307958"} Oct 30 19:31:10.119789 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.119770 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pggqx" event={"ID":"9458eb02-2921-4f34-9578-60056e3505a4","Type":"ContainerStarted","Data":"da1e21af64f24460b9480047c68d5cca821a90736d2dd5ae96ca509cf0078eaf"} Oct 30 19:31:10.119857 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.119800 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-pggqx" event={"ID":"9458eb02-2921-4f34-9578-60056e3505a4","Type":"ContainerStarted","Data":"c722584061fea70d48ce3aaddd579e3b67277faa4f0b3800de3bb2e73c882fb6"} Oct 30 19:31:10.133677 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.133632 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-rtf4t" podStartSLOduration=404.133619569 podStartE2EDuration="6m44.133619569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:10.132933112 +0000 UTC m=+31.924006756" watchObservedRunningTime="2026-10-30 19:31:10.133619569 +0000 UTC m=+31.924693202" Oct 30 19:31:10.154439 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.154382 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-pggqx" podStartSLOduration=404.154367866 podStartE2EDuration="6m44.154367866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:10.153390407 +0000 UTC m=+31.944464046" watchObservedRunningTime="2026-10-30 19:31:10.154367866 +0000 UTC m=+31.945441509" Oct 30 19:31:10.171709 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.171651 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-fkjv2" podStartSLOduration=404.171640212 podStartE2EDuration="6m44.171640212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:10.170569998 +0000 UTC m=+31.961643639" watchObservedRunningTime="2026-10-30 19:31:10.171640212 +0000 UTC m=+31.962713852" Oct 30 19:31:10.776828 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.776790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:10.776978 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.776842 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:10.776978 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.776922 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:10.777056 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.777019 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:31:10.777111 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.777071 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs podName:a8e91049-4a10-4472-922d-7b7875e7de3a nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.777054872 +0000 UTC m=+34.568128489 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs") pod "router-default-787bddbb47-hfcvw" (UID: "a8e91049-4a10-4472-922d-7b7875e7de3a") : secret "router-metrics-certs-default" not found Oct 30 19:31:10.777268 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.777254 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle podName:a8e91049-4a10-4472-922d-7b7875e7de3a nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.777240653 +0000 UTC m=+34.568314280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle") pod "router-default-787bddbb47-hfcvw" (UID: "a8e91049-4a10-4472-922d-7b7875e7de3a") : configmap references non-existent config key: service-ca.crt Oct 30 19:31:10.777341 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.777325 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:31:10.777388 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.777360 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls podName:02832de1-9584-4e12-9b94-8b6b52a71941 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.777349484 +0000 UTC m=+34.568423103 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fcjbl" (UID: "02832de1-9584-4e12-9b94-8b6b52a71941") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:31:10.878428 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.878387 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:10.878588 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.878483 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:10.878588 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.878526 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:10.878681 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.878587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:10.878681 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:10.878645 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:10.879574 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.878790 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:31:10.879574 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.878856 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert podName:6e13582a-7d64-443a-a62f-c14ecc655f18 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.878838416 +0000 UTC m=+34.669912035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-twdf7" (UID: "6e13582a-7d64-443a-a62f-c14ecc655f18") : secret "networking-console-plugin-cert" not found Oct 30 19:31:10.879574 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.878922 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:31:10.879574 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.878955 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls podName:01188e4b-c110-477a-ba88-d9f48741c0a2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.878944445 +0000 UTC m=+34.670018075 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls") pod "dns-default-j2h9z" (UID: "01188e4b-c110-477a-ba88-d9f48741c0a2") : secret "dns-default-metrics-tls" not found Oct 30 19:31:10.879574 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.879015 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:31:10.879574 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.879025 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-549665644b-9xg4z: secret "image-registry-tls" not found Oct 30 19:31:10.879574 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.879055 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls podName:8161b0bd-5184-44fc-864e-869dc7737a55 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.879044986 +0000 UTC m=+34.670118617 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls") pod "image-registry-549665644b-9xg4z" (UID: "8161b0bd-5184-44fc-864e-869dc7737a55") : secret "image-registry-tls" not found Oct 30 19:31:10.879574 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.879107 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:31:10.879574 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.879139 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls podName:7d903311-c71c-4197-8ac3-486702bd46e9 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.8791299 +0000 UTC m=+34.670203533 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-b8bsp" (UID: "7d903311-c71c-4197-8ac3-486702bd46e9") : secret "samples-operator-tls" not found Oct 30 19:31:10.880285 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.879235 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:31:10.880285 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.880256 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f6d8b54bc-kmdpv: secret "image-registry-tls" not found Oct 30 19:31:10.880403 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:10.880316 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls podName:731f78f9-9938-4170-a8ff-10ba7a863759 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:12.880299028 +0000 UTC m=+34.671372669 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls") pod "image-registry-7f6d8b54bc-kmdpv" (UID: "731f78f9-9938-4170-a8ff-10ba7a863759") : secret "image-registry-tls" not found Oct 30 19:31:12.395915 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:12.395206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:12.400430 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:12.400371 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:12.497017 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:12.496239 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:31:12.497017 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:12.496410 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vjq\" (UniqueName: \"kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq\") pod \"network-check-target-p98lt\" (UID: \"6c633f50-fd32-4440-aefc-79e44203424e\") " pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:31:12.497017 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.497004 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Oct 30 19:31:12.497422 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.497078 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs podName:af2922a1-2b0f-4ee1-8ef3-4911d029e93d nodeName:}" failed. No retries permitted until 2026-10-30 19:31:44.497060694 +0000 UTC m=+66.288134328 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs") pod "network-metrics-daemon-t4fw7" (UID: "af2922a1-2b0f-4ee1-8ef3-4911d029e93d") : secret "metrics-daemon-secret" not found Oct 30 19:31:12.502890 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:12.502861 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9vjq\" (UniqueName: \"kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq\") pod \"network-check-target-p98lt\" (UID: \"6c633f50-fd32-4440-aefc-79e44203424e\") " pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:31:12.667054 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:12.666809 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:12.678936 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:12.678909 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:31:12.805842 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:12.805811 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:12.805933 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:12.805902 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:12.805997 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:12.805970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:12.806252 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.806219 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle podName:a8e91049-4a10-4472-922d-7b7875e7de3a nodeName:}" failed. No retries permitted until 2026-10-30 19:31:16.80619828 +0000 UTC m=+38.597271904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle") pod "router-default-787bddbb47-hfcvw" (UID: "a8e91049-4a10-4472-922d-7b7875e7de3a") : configmap references non-existent config key: service-ca.crt Oct 30 19:31:12.806332 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.806275 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:31:12.806332 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.806327 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs podName:a8e91049-4a10-4472-922d-7b7875e7de3a nodeName:}" failed. No retries permitted until 2026-10-30 19:31:16.806314829 +0000 UTC m=+38.597388463 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs") pod "router-default-787bddbb47-hfcvw" (UID: "a8e91049-4a10-4472-922d-7b7875e7de3a") : secret "router-metrics-certs-default" not found Oct 30 19:31:12.806441 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.806402 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:31:12.806487 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.806445 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls podName:02832de1-9584-4e12-9b94-8b6b52a71941 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:16.806427827 +0000 UTC m=+38.597501459 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fcjbl" (UID: "02832de1-9584-4e12-9b94-8b6b52a71941") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:31:12.907548 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:12.907511 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:12.907680 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:12.907572 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:12.907680 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:12.907626 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:12.907740 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:12.907676 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:12.907773 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:12.907740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:12.907903 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.907890 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:31:12.907961 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.907952 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls podName:01188e4b-c110-477a-ba88-d9f48741c0a2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:16.907934527 +0000 UTC m=+38.699008160 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls") pod "dns-default-j2h9z" (UID: "01188e4b-c110-477a-ba88-d9f48741c0a2") : secret "dns-default-metrics-tls" not found Oct 30 19:31:12.908377 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.908360 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:31:12.908414 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.908381 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-549665644b-9xg4z: secret "image-registry-tls" not found Oct 30 19:31:12.908442 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.908422 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls podName:8161b0bd-5184-44fc-864e-869dc7737a55 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:16.908408646 +0000 UTC m=+38.699482280 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls") pod "image-registry-549665644b-9xg4z" (UID: "8161b0bd-5184-44fc-864e-869dc7737a55") : secret "image-registry-tls" not found Oct 30 19:31:12.908494 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.908482 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:31:12.908527 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.908522 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls podName:7d903311-c71c-4197-8ac3-486702bd46e9 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:16.908510923 +0000 UTC m=+38.699584539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-b8bsp" (UID: "7d903311-c71c-4197-8ac3-486702bd46e9") : secret "samples-operator-tls" not found Oct 30 19:31:12.908579 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.908566 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:31:12.908615 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.908590 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:31:12.908615 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.908603 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f6d8b54bc-kmdpv: secret "image-registry-tls" not found Oct 30 19:31:12.908615 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.908611 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert podName:6e13582a-7d64-443a-a62f-c14ecc655f18 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:16.908598067 +0000 UTC m=+38.699671702 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-twdf7" (UID: "6e13582a-7d64-443a-a62f-c14ecc655f18") : secret "networking-console-plugin-cert" not found Oct 30 19:31:12.908695 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:12.908630 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls podName:731f78f9-9938-4170-a8ff-10ba7a863759 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:16.908621243 +0000 UTC m=+38.699694874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls") pod "image-registry-7f6d8b54bc-kmdpv" (UID: "731f78f9-9938-4170-a8ff-10ba7a863759") : secret "image-registry-tls" not found Oct 30 19:31:13.012304 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:13.012272 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mm6ld"] Oct 30 19:31:13.021944 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:13.021916 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-p98lt"] Oct 30 19:31:13.023166 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:31:13.023118 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c633f50_fd32_4440_aefc_79e44203424e.slice/crio-0f92a5095fb5066e5da408999533fdb735868525098e3e3f9e008bf9b3f3003e WatchSource:0}: Error finding container 0f92a5095fb5066e5da408999533fdb735868525098e3e3f9e008bf9b3f3003e: Status 404 returned error can't find the container with id 0f92a5095fb5066e5da408999533fdb735868525098e3e3f9e008bf9b3f3003e Oct 30 19:31:13.026895 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:31:13.026770 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bfb73f4_9290_4440_938a_feb9fafa3287.slice/crio-96394dbd894cfd2ad3a566b2b843e24a858c2f2f669a455f5b7f2d27397c973b WatchSource:0}: Error finding container 96394dbd894cfd2ad3a566b2b843e24a858c2f2f669a455f5b7f2d27397c973b: Status 404 returned error can't find the container with id 96394dbd894cfd2ad3a566b2b843e24a858c2f2f669a455f5b7f2d27397c973b Oct 30 19:31:13.143121 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:13.143044 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vblv9" event={"ID":"6693a492-433f-4614-a8e3-71ee75c177c2","Type":"ContainerStarted","Data":"d42e3447747f1a908e5cfbe3b834c451cc4ba8b5509968754031f8e443c40827"} Oct 30 19:31:13.144661 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:13.144630 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mm6ld" event={"ID":"5bfb73f4-9290-4440-938a-feb9fafa3287","Type":"ContainerStarted","Data":"96394dbd894cfd2ad3a566b2b843e24a858c2f2f669a455f5b7f2d27397c973b"} Oct 30 19:31:13.148031 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:13.147999 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p98lt" event={"ID":"6c633f50-fd32-4440-aefc-79e44203424e","Type":"ContainerStarted","Data":"0f92a5095fb5066e5da408999533fdb735868525098e3e3f9e008bf9b3f3003e"} Oct 30 19:31:13.162679 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:13.162524 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vblv9" podStartSLOduration=366.937413654 podStartE2EDuration="6m9.162510186s" totalImagesPullingTime="2.225096532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:13.160964747 +0000 UTC m=+34.952038386" watchObservedRunningTime="2026-10-30 19:31:13.162510186 +0000 UTC m=+34.953583822" Oct 30 19:31:13.354587 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:13.354492 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:13.566450 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:13.566368 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:15.241614 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.241583 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:15.772094 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.770628 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-fndt2"] Oct 30 19:31:15.789718 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.789427 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rgfw9-smvbs"] Oct 30 19:31:15.789718 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.789647 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:15.794725 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.794447 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:31:15.795041 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.795020 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f2mgm\"" Oct 30 19:31:15.795166 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.795096 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:31:15.804201 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.804160 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fndt2"] Oct 30 19:31:15.804304 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.804267 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:15.804504 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.804486 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rgfw9-smvbs"] Oct 30 19:31:15.844011 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.843083 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:15.844011 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.843183 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-trusted-ca-bundle\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:15.844011 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.843211 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:15.844011 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.843247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phptb\" (UniqueName: \"kubernetes.io/projected/9e4295e8-368d-4799-b1c4-70355610378c-kube-api-access-phptb\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:15.844011 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.843284 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6e429ff-3fb2-48da-a77b-7806b90139e5-crio-socket\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:15.844011 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.843333 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzv8w\" (UniqueName: \"kubernetes.io/projected/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-api-access-wzv8w\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:15.844011 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.843367 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9e4295e8-368d-4799-b1c4-70355610378c-archives-path\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:15.844011 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.843382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6e429ff-3fb2-48da-a77b-7806b90139e5-data-volume\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:15.844011 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.843439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-service-ca-bundle\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:15.844859 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.844837 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:15.944435 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.944336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-service-ca-bundle\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:15.944435 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.944384 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:15.944655 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.944458 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-trusted-ca-bundle\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:15.944655 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.944499 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:15.944655 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.944544 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phptb\" (UniqueName: \"kubernetes.io/projected/9e4295e8-368d-4799-b1c4-70355610378c-kube-api-access-phptb\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:15.944655 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.944596 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6e429ff-3fb2-48da-a77b-7806b90139e5-crio-socket\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:15.944810 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.944657 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzv8w\" (UniqueName: \"kubernetes.io/projected/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-api-access-wzv8w\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:15.944810 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.944702 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9e4295e8-368d-4799-b1c4-70355610378c-archives-path\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:15.944810 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.944725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6e429ff-3fb2-48da-a77b-7806b90139e5-data-volume\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:15.946421 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:15.944996 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:31:15.946421 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:15.945022 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:31:15.946421 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:15.945077 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls podName:b6e429ff-3fb2-48da-a77b-7806b90139e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:16.44505766 +0000 UTC m=+38.236131285 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fndt2" (UID: "b6e429ff-3fb2-48da-a77b-7806b90139e5") : secret "insights-runtime-extractor-tls" not found Oct 30 19:31:15.946421 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:15.945095 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm podName:b6e429ff-3fb2-48da-a77b-7806b90139e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:16.445085867 +0000 UTC m=+38.236159484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fndt2" (UID: "b6e429ff-3fb2-48da-a77b-7806b90139e5") : configmap "kube-rbac-proxy" not found Oct 30 19:31:15.946421 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.945095 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6e429ff-3fb2-48da-a77b-7806b90139e5-data-volume\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:15.946421 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.945350 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6e429ff-3fb2-48da-a77b-7806b90139e5-crio-socket\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:15.946421 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.946017 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-service-ca-bundle\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:15.946421 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.946119 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-trusted-ca-bundle\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:15.946421 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.946382 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9e4295e8-368d-4799-b1c4-70355610378c-archives-path\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:15.960498 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.960475 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phptb\" (UniqueName: \"kubernetes.io/projected/9e4295e8-368d-4799-b1c4-70355610378c-kube-api-access-phptb\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:15.960921 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.960901 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzv8w\" (UniqueName: \"kubernetes.io/projected/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-api-access-wzv8w\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:15.992128 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:15.992088 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:16.161933 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.161842 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-77ms2" event={"ID":"534aacd9-7102-4a50-8696-c50362ce87e4","Type":"ContainerStarted","Data":"4fcff38e3e1b1d8d71425d31907f6c7bcc80682d89cdc29798a74f96b175de7b"} Oct 30 19:31:16.163483 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.163458 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/0.log" Oct 30 19:31:16.163628 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.163503 2579 generic.go:356] "Generic (PLEG): container finished" podID="067a4436-690d-4b2c-aaf2-1d84830b7287" containerID="a7206a7b6c19c6d914eb95635f798d9ccddae82b7f137a74f3e1c85973167db0" exitCode=255 Oct 30 19:31:16.163628 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.163572 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" event={"ID":"067a4436-690d-4b2c-aaf2-1d84830b7287","Type":"ContainerDied","Data":"a7206a7b6c19c6d914eb95635f798d9ccddae82b7f137a74f3e1c85973167db0"} Oct 30 19:31:16.163816 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.163800 2579 scope.go:122] "RemoveContainer" containerID="a7206a7b6c19c6d914eb95635f798d9ccddae82b7f137a74f3e1c85973167db0" Oct 30 19:31:16.165656 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.165635 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7jhj7" event={"ID":"24ac5496-c680-4b2a-8362-f12b662df997","Type":"ContainerStarted","Data":"0ba6c02fc5a5d01e405ee62c66025ff3dbd10d48025eb6c710e785a903710bea"} Oct 30 19:31:16.167652 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.167629 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:16.167874 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.167728 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" event={"ID":"41c1116a-2830-4ea1-aae5-3a052f073293","Type":"ContainerStarted","Data":"6ad589fb6f687aed8c5d1c29d9454efa30fd4151d787cce480432962d2dfab3c"} Oct 30 19:31:16.169966 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.169943 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" event={"ID":"6c260520-55b0-44ca-8ccb-e15cab54fa90","Type":"ContainerStarted","Data":"691006c8d5acdb575b9c091b3ec5d9271e8a6d912e6611228c073107f34ae2eb"} Oct 30 19:31:16.179074 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.178997 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:16.187095 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.187046 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-77ms2" podStartSLOduration=1.538112454 podStartE2EDuration="7.187031332s" totalImagesPullingTime="5.648918878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:16.18546384 +0000 UTC m=+37.976537515" watchObservedRunningTime="2026-10-30 19:31:16.187031332 +0000 UTC m=+37.978104971" Oct 30 19:31:16.208637 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.208606 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-h6ssv"] Oct 30 19:31:16.226015 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.225985 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-h6ssv"] Oct 30 19:31:16.226266 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.226030 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vqzdh"] Oct 30 19:31:16.226266 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.226106 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h6ssv" Oct 30 19:31:16.233351 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.233309 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 19:31:16.233497 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.233348 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 19:31:16.233497 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.233418 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w9bpk\"" Oct 30 19:31:16.245132 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.245101 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vqzdh"] Oct 30 19:31:16.245581 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.245245 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vqzdh" Oct 30 19:31:16.246125 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.246072 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" podStartSLOduration=366.855144427 podStartE2EDuration="6m12.24605485s" totalImagesPullingTime="5.390910423s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:16.244553095 +0000 UTC m=+38.035626732" watchObservedRunningTime="2026-10-30 19:31:16.24605485 +0000 UTC m=+38.037128490" Oct 30 19:31:16.247198 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.247072 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glnjk\" (UniqueName: \"kubernetes.io/projected/f031b635-0cf3-4670-a9c6-278f14922b37-kube-api-access-glnjk\") pod \"migrator-66df5d496b-h6ssv\" (UID: \"f031b635-0cf3-4670-a9c6-278f14922b37\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h6ssv" Oct 30 19:31:16.265570 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.265523 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-7jhj7" podStartSLOduration=366.978083611 podStartE2EDuration="6m12.265508309s" totalImagesPullingTime="5.287424698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:16.264202721 +0000 UTC m=+38.055276375" watchObservedRunningTime="2026-10-30 19:31:16.265508309 +0000 UTC m=+38.056581948" Oct 30 19:31:16.292127 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.292076 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" podStartSLOduration=366.942359395 podStartE2EDuration="6m12.292063119s" totalImagesPullingTime="5.349703724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:16.291926471 +0000 UTC m=+38.083000123" watchObservedRunningTime="2026-10-30 19:31:16.292063119 +0000 UTC m=+38.083136754" Oct 30 19:31:16.353392 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.353301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9v4h\" (UniqueName: \"kubernetes.io/projected/a2cace4b-3a50-4fe5-8ead-c9765b9540d9-kube-api-access-t9v4h\") pod \"migrator-66df5d496b-vqzdh\" (UID: \"a2cace4b-3a50-4fe5-8ead-c9765b9540d9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vqzdh" Oct 30 19:31:16.353500 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.353443 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnjk\" (UniqueName: \"kubernetes.io/projected/f031b635-0cf3-4670-a9c6-278f14922b37-kube-api-access-glnjk\") pod \"migrator-66df5d496b-h6ssv\" (UID: \"f031b635-0cf3-4670-a9c6-278f14922b37\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h6ssv" Oct 30 19:31:16.358369 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.358102 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:16.375097 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.375069 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glnjk\" (UniqueName: \"kubernetes.io/projected/f031b635-0cf3-4670-a9c6-278f14922b37-kube-api-access-glnjk\") pod \"migrator-66df5d496b-h6ssv\" (UID: \"f031b635-0cf3-4670-a9c6-278f14922b37\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h6ssv" Oct 30 19:31:16.445958 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.445908 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-nfhdn"] Oct 30 19:31:16.456043 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.456017 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:16.456193 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.456180 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9v4h\" (UniqueName: \"kubernetes.io/projected/a2cace4b-3a50-4fe5-8ead-c9765b9540d9-kube-api-access-t9v4h\") pod \"migrator-66df5d496b-vqzdh\" (UID: \"a2cace4b-3a50-4fe5-8ead-c9765b9540d9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vqzdh" Oct 30 19:31:16.456255 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.456198 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:31:16.457491 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.457472 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls podName:b6e429ff-3fb2-48da-a77b-7806b90139e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:17.457448462 +0000 UTC m=+39.248522086 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fndt2" (UID: "b6e429ff-3fb2-48da-a77b-7806b90139e5") : secret "insights-runtime-extractor-tls" not found Oct 30 19:31:16.457731 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.457705 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:16.459425 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.459404 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:31:16.459592 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.459580 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm podName:b6e429ff-3fb2-48da-a77b-7806b90139e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:17.459561474 +0000 UTC m=+39.250635107 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fndt2" (UID: "b6e429ff-3fb2-48da-a77b-7806b90139e5") : configmap "kube-rbac-proxy" not found Oct 30 19:31:16.481175 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.478728 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-nfhdn"] Oct 30 19:31:16.481175 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.478902 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:16.481175 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.479388 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9v4h\" (UniqueName: \"kubernetes.io/projected/a2cace4b-3a50-4fe5-8ead-c9765b9540d9-kube-api-access-t9v4h\") pod \"migrator-66df5d496b-vqzdh\" (UID: \"a2cace4b-3a50-4fe5-8ead-c9765b9540d9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vqzdh" Oct 30 19:31:16.486380 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.486356 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7c7hk\"" Oct 30 19:31:16.486497 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.486463 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 19:31:16.486609 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.486595 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 19:31:16.486723 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.486694 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 19:31:16.490941 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.490573 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 19:31:16.505568 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.505532 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:16.558501 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.558463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/22599c63-1821-4c98-9096-9d4c43549ab7-signing-cabundle\") pod \"service-ca-5fffc97bdf-nfhdn\" (UID: \"22599c63-1821-4c98-9096-9d4c43549ab7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:16.558658 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.558564 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kk5qr\" (UniqueName: \"kubernetes.io/projected/22599c63-1821-4c98-9096-9d4c43549ab7-kube-api-access-kk5qr\") pod \"service-ca-5fffc97bdf-nfhdn\" (UID: \"22599c63-1821-4c98-9096-9d4c43549ab7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:16.558710 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.558654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/22599c63-1821-4c98-9096-9d4c43549ab7-signing-key\") pod \"service-ca-5fffc97bdf-nfhdn\" (UID: \"22599c63-1821-4c98-9096-9d4c43549ab7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:16.562253 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.562229 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h6ssv" Oct 30 19:31:16.584798 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.584774 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vqzdh" Oct 30 19:31:16.610577 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.610504 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rgfw9-smvbs"] Oct 30 19:31:16.663967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.662745 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/22599c63-1821-4c98-9096-9d4c43549ab7-signing-cabundle\") pod \"service-ca-5fffc97bdf-nfhdn\" (UID: \"22599c63-1821-4c98-9096-9d4c43549ab7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:16.663967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.662815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk5qr\" (UniqueName: \"kubernetes.io/projected/22599c63-1821-4c98-9096-9d4c43549ab7-kube-api-access-kk5qr\") pod \"service-ca-5fffc97bdf-nfhdn\" (UID: \"22599c63-1821-4c98-9096-9d4c43549ab7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:16.663967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.662864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/22599c63-1821-4c98-9096-9d4c43549ab7-signing-key\") pod \"service-ca-5fffc97bdf-nfhdn\" (UID: \"22599c63-1821-4c98-9096-9d4c43549ab7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:16.663967 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.663895 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/22599c63-1821-4c98-9096-9d4c43549ab7-signing-cabundle\") pod \"service-ca-5fffc97bdf-nfhdn\" (UID: \"22599c63-1821-4c98-9096-9d4c43549ab7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:16.665621 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.665595 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/22599c63-1821-4c98-9096-9d4c43549ab7-signing-key\") pod \"service-ca-5fffc97bdf-nfhdn\" (UID: \"22599c63-1821-4c98-9096-9d4c43549ab7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:16.683184 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.683120 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk5qr\" (UniqueName: \"kubernetes.io/projected/22599c63-1821-4c98-9096-9d4c43549ab7-kube-api-access-kk5qr\") pod \"service-ca-5fffc97bdf-nfhdn\" (UID: \"22599c63-1821-4c98-9096-9d4c43549ab7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:16.825707 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.825683 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:16.848173 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.848119 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:16.866273 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.866138 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-h6ssv"] Oct 30 19:31:16.867141 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.866501 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:16.867141 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.866573 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:16.867141 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.866612 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Oct 30 19:31:16.867141 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.866679 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls podName:02832de1-9584-4e12-9b94-8b6b52a71941 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:24.866658758 +0000 UTC m=+46.657732381 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-fcjbl" (UID: "02832de1-9584-4e12-9b94-8b6b52a71941") : secret "cluster-monitoring-operator-tls" not found Oct 30 19:31:16.867141 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.867093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:16.867455 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.867344 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:31:16.867455 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.867403 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs podName:a8e91049-4a10-4472-922d-7b7875e7de3a nodeName:}" failed. No retries permitted until 2026-10-30 19:31:24.867374961 +0000 UTC m=+46.658448578 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs") pod "router-default-787bddbb47-hfcvw" (UID: "a8e91049-4a10-4472-922d-7b7875e7de3a") : secret "router-metrics-certs-default" not found Oct 30 19:31:16.867678 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.867653 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle podName:a8e91049-4a10-4472-922d-7b7875e7de3a nodeName:}" failed. No retries permitted until 2026-10-30 19:31:24.867637817 +0000 UTC m=+46.658711436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle") pod "router-default-787bddbb47-hfcvw" (UID: "a8e91049-4a10-4472-922d-7b7875e7de3a") : configmap references non-existent config key: service-ca.crt Oct 30 19:31:16.888113 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:31:16.877997 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf031b635_0cf3_4670_a9c6_278f14922b37.slice/crio-00501ced5606129d0345f27bee1112c7caab5f0d7a6b940d1eb3316f85a839c8 WatchSource:0}: Error finding container 00501ced5606129d0345f27bee1112c7caab5f0d7a6b940d1eb3316f85a839c8: Status 404 returned error can't find the container with id 00501ced5606129d0345f27bee1112c7caab5f0d7a6b940d1eb3316f85a839c8 Oct 30 19:31:16.900693 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.900270 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-vqzdh"] Oct 30 19:31:16.910323 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:31:16.910297 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2cace4b_3a50_4fe5_8ead_c9765b9540d9.slice/crio-ad622f0d7340410ea5ffcf586d39ff66d514b44ba6c7502e4564e69c8b7bca67 WatchSource:0}: Error finding container ad622f0d7340410ea5ffcf586d39ff66d514b44ba6c7502e4564e69c8b7bca67: Status 404 returned error can't find the container with id ad622f0d7340410ea5ffcf586d39ff66d514b44ba6c7502e4564e69c8b7bca67 Oct 30 19:31:16.968013 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.967978 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:16.968204 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.968051 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:16.968204 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.968117 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:16.968204 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.968173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:16.968336 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:16.968227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:16.968336 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.968237 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:31:16.968336 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.968310 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert podName:6e13582a-7d64-443a-a62f-c14ecc655f18 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:24.968294289 +0000 UTC m=+46.759367923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-twdf7" (UID: "6e13582a-7d64-443a-a62f-c14ecc655f18") : secret "networking-console-plugin-cert" not found Oct 30 19:31:16.968435 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.968363 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:31:16.968435 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.968379 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f6d8b54bc-kmdpv: secret "image-registry-tls" not found Oct 30 19:31:16.968435 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.968425 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls podName:731f78f9-9938-4170-a8ff-10ba7a863759 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:24.968410371 +0000 UTC m=+46.759483990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls") pod "image-registry-7f6d8b54bc-kmdpv" (UID: "731f78f9-9938-4170-a8ff-10ba7a863759") : secret "image-registry-tls" not found Oct 30 19:31:16.968545 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.968492 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:31:16.968545 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.968526 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls podName:01188e4b-c110-477a-ba88-d9f48741c0a2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:24.968513838 +0000 UTC m=+46.759587457 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls") pod "dns-default-j2h9z" (UID: "01188e4b-c110-477a-ba88-d9f48741c0a2") : secret "dns-default-metrics-tls" not found Oct 30 19:31:16.968607 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.968596 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:31:16.968636 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.968607 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-549665644b-9xg4z: secret "image-registry-tls" not found Oct 30 19:31:16.968663 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.968638 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls podName:8161b0bd-5184-44fc-864e-869dc7737a55 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:24.968626604 +0000 UTC m=+46.759700222 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls") pod "image-registry-549665644b-9xg4z" (UID: "8161b0bd-5184-44fc-864e-869dc7737a55") : secret "image-registry-tls" not found Oct 30 19:31:16.968697 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.968660 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Oct 30 19:31:16.968725 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:16.968708 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls podName:7d903311-c71c-4197-8ac3-486702bd46e9 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:24.96869489 +0000 UTC m=+46.759768525 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-b8bsp" (UID: "7d903311-c71c-4197-8ac3-486702bd46e9") : secret "samples-operator-tls" not found Oct 30 19:31:17.075356 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.075322 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:17.136138 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.136105 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-nfhdn"] Oct 30 19:31:17.141199 ip-10-0-132-214 kubenswrapper[2579]: W1030 19:31:17.141172 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22599c63_1821_4c98_9096_9d4c43549ab7.slice/crio-892e9f6edfc7c956d0e78c4669fa1043e08537af5400263662721c7373b31963 WatchSource:0}: Error finding container 892e9f6edfc7c956d0e78c4669fa1043e08537af5400263662721c7373b31963: Status 404 returned error can't find the container with id 892e9f6edfc7c956d0e78c4669fa1043e08537af5400263662721c7373b31963 Oct 30 19:31:17.175979 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.175917 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:31:17.176466 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.176449 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/0.log" Oct 30 19:31:17.176581 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.176491 2579 generic.go:356] "Generic (PLEG): container finished" podID="067a4436-690d-4b2c-aaf2-1d84830b7287" containerID="d2b9813548f63ce8ff5ee1d96c7d773bedbaa3663d02970290b3b3f268e977b6" exitCode=255 Oct 30 19:31:17.176632 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.176577 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" event={"ID":"067a4436-690d-4b2c-aaf2-1d84830b7287","Type":"ContainerDied","Data":"d2b9813548f63ce8ff5ee1d96c7d773bedbaa3663d02970290b3b3f268e977b6"} Oct 30 19:31:17.176670 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.176646 2579 scope.go:122] "RemoveContainer" containerID="a7206a7b6c19c6d914eb95635f798d9ccddae82b7f137a74f3e1c85973167db0" Oct 30 19:31:17.176876 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.176856 2579 scope.go:122] "RemoveContainer" containerID="d2b9813548f63ce8ff5ee1d96c7d773bedbaa3663d02970290b3b3f268e977b6" Oct 30 19:31:17.177104 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:17.177082 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4kpqr_openshift-console-operator(067a4436-690d-4b2c-aaf2-1d84830b7287)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" podUID="067a4436-690d-4b2c-aaf2-1d84830b7287" Oct 30 19:31:17.178265 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.178133 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-p98lt" event={"ID":"6c633f50-fd32-4440-aefc-79e44203424e","Type":"ContainerStarted","Data":"b8c2edbe534f65ef3e6aa646c021e421dc7df5e42d1c1beba64d5e0288425edd"} Oct 30 19:31:17.178265 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.178222 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:31:17.179570 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.179551 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" event={"ID":"22599c63-1821-4c98-9096-9d4c43549ab7","Type":"ContainerStarted","Data":"892e9f6edfc7c956d0e78c4669fa1043e08537af5400263662721c7373b31963"} Oct 30 19:31:17.181304 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.181276 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kzbbk" event={"ID":"b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad","Type":"ContainerStarted","Data":"42a0e38d623793a681f339555e58c9ea6f9067a859f017dc659c3fe6f6e420e4"} Oct 30 19:31:17.183390 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.183367 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vqzdh" event={"ID":"a2cace4b-3a50-4fe5-8ead-c9765b9540d9","Type":"ContainerStarted","Data":"ad622f0d7340410ea5ffcf586d39ff66d514b44ba6c7502e4564e69c8b7bca67"} Oct 30 19:31:17.184603 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.184580 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h6ssv" event={"ID":"f031b635-0cf3-4670-a9c6-278f14922b37","Type":"ContainerStarted","Data":"00501ced5606129d0345f27bee1112c7caab5f0d7a6b940d1eb3316f85a839c8"} Oct 30 19:31:17.186737 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.186714 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rgfw9-smvbs" event={"ID":"9e4295e8-368d-4799-b1c4-70355610378c","Type":"ContainerStarted","Data":"72d3814b00adb675b07b69585e936c4b6c61aa78a497d5ddbb3f621d675d026b"} Oct 30 19:31:17.186834 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.186753 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rgfw9-smvbs" event={"ID":"9e4295e8-368d-4799-b1c4-70355610378c","Type":"ContainerStarted","Data":"d23691ccb7c2c62b4dfb0b27a3ff430ab94d1b198b3876a7ebd5e846ed247f4c"} Oct 30 19:31:17.234765 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.234720 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-p98lt" podStartSLOduration=34.439641908 podStartE2EDuration="38.23470285s" totalImagesPullingTime="3.795060942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:17.233399937 +0000 UTC m=+39.024473578" watchObservedRunningTime="2026-10-30 19:31:17.23470285 +0000 UTC m=+39.025776491" Oct 30 19:31:17.253718 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.253663 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rgfw9-smvbs" podStartSLOduration=2.253645004 podStartE2EDuration="2.253645004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:17.252045074 +0000 UTC m=+39.043118710" watchObservedRunningTime="2026-10-30 19:31:17.253645004 +0000 UTC m=+39.044718646" Oct 30 19:31:17.263998 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.263921 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:17.275165 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.275096 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kzbbk" podStartSLOduration=366.784821332 podStartE2EDuration="6m13.275084212s" totalImagesPullingTime="6.49026288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:17.274020245 +0000 UTC m=+39.065093885" watchObservedRunningTime="2026-10-30 19:31:17.275084212 +0000 UTC m=+39.066157896" Oct 30 19:31:17.474339 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.474260 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:17.474339 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:17.474331 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:17.474592 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:17.474411 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:31:17.474592 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:17.474483 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm podName:b6e429ff-3fb2-48da-a77b-7806b90139e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:19.474468158 +0000 UTC m=+41.265541789 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fndt2" (UID: "b6e429ff-3fb2-48da-a77b-7806b90139e5") : configmap "kube-rbac-proxy" not found Oct 30 19:31:17.474592 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:17.474498 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:31:17.474592 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:17.474544 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls podName:b6e429ff-3fb2-48da-a77b-7806b90139e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:19.474532173 +0000 UTC m=+41.265605807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fndt2" (UID: "b6e429ff-3fb2-48da-a77b-7806b90139e5") : secret "insights-runtime-extractor-tls" not found Oct 30 19:31:18.048678 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:18.048587 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:18.201296 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:18.201264 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:31:18.204544 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:18.201790 2579 scope.go:122] "RemoveContainer" containerID="d2b9813548f63ce8ff5ee1d96c7d773bedbaa3663d02970290b3b3f268e977b6" Oct 30 19:31:18.204544 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:18.202086 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4kpqr_openshift-console-operator(067a4436-690d-4b2c-aaf2-1d84830b7287)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" podUID="067a4436-690d-4b2c-aaf2-1d84830b7287" Oct 30 19:31:18.204544 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:18.204395 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" event={"ID":"22599c63-1821-4c98-9096-9d4c43549ab7","Type":"ContainerStarted","Data":"f635a984c982a7902780c8dc253de73a9649127d08541c0a8e006b007d2a97df"} Oct 30 19:31:18.238896 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:18.238864 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:18.272840 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:18.272790 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" podStartSLOduration=2.272775359 podStartE2EDuration="2.272775359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:18.270787494 +0000 UTC m=+40.061861135" watchObservedRunningTime="2026-10-30 19:31:18.272775359 +0000 UTC m=+40.063848999" Oct 30 19:31:18.391278 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:18.391097 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:18.536356 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:18.536314 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:18.924873 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:18.924848 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9g689_7fa163f8-129c-4f64-ad4b-ad3cc08bed13/dns-node-resolver/0.log" Oct 30 19:31:19.209844 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:19.209762 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vqzdh" event={"ID":"a2cace4b-3a50-4fe5-8ead-c9765b9540d9","Type":"ContainerStarted","Data":"e28c62b4f544e91d15e1550f4c5fe60f6139090d3b548a98db86699b70939907"} Oct 30 19:31:19.211272 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:19.211243 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:19.211715 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:19.211691 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h6ssv" event={"ID":"f031b635-0cf3-4670-a9c6-278f14922b37","Type":"ContainerStarted","Data":"dada1eee3f6981579f3a9661881d8fcbc93e9aac5d5ec911f3ac8a951261d867"} Oct 30 19:31:19.245611 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:19.245569 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h6ssv" podStartSLOduration=1.235324583 podStartE2EDuration="3.245553877s" totalImagesPullingTime="2.010229294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:19.245341723 +0000 UTC m=+41.036415363" watchObservedRunningTime="2026-10-30 19:31:19.245553877 +0000 UTC m=+41.036627515" Oct 30 19:31:19.246513 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:19.246470 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vqzdh" podStartSLOduration=1.26871866 podStartE2EDuration="3.246461561s" totalImagesPullingTime="1.977742901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:19.228693286 +0000 UTC m=+41.019766926" watchObservedRunningTime="2026-10-30 19:31:19.246461561 +0000 UTC m=+41.037535200" Oct 30 19:31:19.353919 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:19.353875 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:19.443519 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:19.443485 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:19.443879 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:19.443862 2579 scope.go:122] "RemoveContainer" containerID="d2b9813548f63ce8ff5ee1d96c7d773bedbaa3663d02970290b3b3f268e977b6" Oct 30 19:31:19.444083 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:19.444063 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4kpqr_openshift-console-operator(067a4436-690d-4b2c-aaf2-1d84830b7287)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" podUID="067a4436-690d-4b2c-aaf2-1d84830b7287" Oct 30 19:31:19.444461 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:19.444443 2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:19.502748 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:19.502097 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:19.502748 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:19.502224 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Oct 30 19:31:19.502748 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:19.502309 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm podName:b6e429ff-3fb2-48da-a77b-7806b90139e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:23.50228914 +0000 UTC m=+45.293362764 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm") pod "insights-runtime-extractor-fndt2" (UID: "b6e429ff-3fb2-48da-a77b-7806b90139e5") : configmap "kube-rbac-proxy" not found Oct 30 19:31:19.502748 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:19.502311 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:31:19.502748 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:19.502226 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:19.502748 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:19.502367 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls podName:b6e429ff-3fb2-48da-a77b-7806b90139e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:23.502352016 +0000 UTC m=+45.293425651 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fndt2" (UID: "b6e429ff-3fb2-48da-a77b-7806b90139e5") : secret "insights-runtime-extractor-tls" not found Oct 30 19:31:19.682663 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:19.682635 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:19.889005 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:19.888939 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:20.067338 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:20.067314 2579 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:20.124423 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:20.124397 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m77bc_fb5d13ef-8197-4960-b142-d8a1bdd025e2/node-ca/0.log" Oct 30 19:31:20.222377 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:20.222296 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mm6ld" event={"ID":"5bfb73f4-9290-4440-938a-feb9fafa3287","Type":"ContainerStarted","Data":"ac12330a7ead4f95400044aa65d74d86b7d975a4fa5db44ec45f2b40a226a9b3"} Oct 30 19:31:20.222850 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:20.222832 2579 scope.go:122] "RemoveContainer" containerID="d2b9813548f63ce8ff5ee1d96c7d773bedbaa3663d02970290b3b3f268e977b6" Oct 30 19:31:20.222983 ip-10-0-132-214 kubenswrapper[2579]: E1030 19:31:20.222967 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-4kpqr_openshift-console-operator(067a4436-690d-4b2c-aaf2-1d84830b7287)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" podUID="067a4436-690d-4b2c-aaf2-1d84830b7287" Oct 30 19:31:20.242606 ip-10-0-132-214 kubenswrapper[2579]: I1030 19:31:20.242563 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mm6ld" podStartSLOduration=34.342333453 podStartE2EDuration="41.242551919s" totalImagesPullingTime="6.900218466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:20.241530411 +0000 UTC m=+42.032604051" watchObservedRunningTime="2026-10-30 19:31:20.242551919 +0000 UTC m=+42.033625558" Oct 30 19:31:20.251927 ip-10-0-132-214 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 19:31:20.262512 ip-10-0-132-214 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 19:31:20.262682 ip-10-0-132-214 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 19:31:20.262730 ip-10-0-132-214 systemd[1]: kubelet.service: Consumed 3.199s CPU time. Oct 30 19:31:20.274698 ip-10-0-132-214 systemd[1]: Starting Kubernetes Kubelet... Oct 30 19:31:20.354350 ip-10-0-132-214 kubenswrapper[5091]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:31:20.354350 ip-10-0-132-214 kubenswrapper[5091]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 19:31:20.354350 ip-10-0-132-214 kubenswrapper[5091]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:31:20.354350 ip-10-0-132-214 kubenswrapper[5091]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 19:31:20.356430 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356339 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:31:20.356430 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356427 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:31:20.356430 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356432 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:31:20.356430 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356435 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356438 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356441 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356444 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356448 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356453 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356456 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356467 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356470 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356473 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356476 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356479 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356482 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356485 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356488 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356490 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356493 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356496 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356499 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:31:20.356620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356501 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356504 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356508 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356510 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356513 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356516 5091 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356519 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356522 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356524 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356527 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356530 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356543 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356546 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356548 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356552 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356555 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356557 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356560 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356563 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356565 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:31:20.357098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356568 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356571 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356574 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356577 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356580 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356582 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356585 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356593 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356596 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356599 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356602 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356604 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356607 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356610 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356613 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356616 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356618 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356621 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356624 5091 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356627 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:31:20.357622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356630 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356632 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356635 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356638 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356640 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356643 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356646 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356648 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356651 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356654 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356657 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356659 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356662 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356664 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356675 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356678 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356681 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356684 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356687 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:31:20.358135 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356690 5091 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356693 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356696 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356698 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356701 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356703 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356706 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356708 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356711 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356715 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356718 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356721 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356724 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356727 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356730 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356732 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356735 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356738 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356740 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:31:20.358658 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356743 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356746 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356748 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356750 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356753 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.356756 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357537 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357545 5091 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357548 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357551 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357555 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357558 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357560 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357563 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357566 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357568 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357571 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357574 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357576 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357579 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:31:20.359160 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357582 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357586 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357590 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357594 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357599 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357602 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357605 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357608 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357612 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357616 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357623 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357628 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357631 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357635 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357637 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357640 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357643 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357646 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357649 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:31:20.359774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357652 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357654 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357657 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357661 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357664 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357666 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357669 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357672 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357674 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357678 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357682 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357685 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357688 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357690 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357693 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357696 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357698 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357701 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357704 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357706 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:31:20.360487 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357709 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357712 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357715 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357717 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357720 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357723 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357725 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357728 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357730 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357733 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357736 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357739 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357742 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357745 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357747 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357750 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357753 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357756 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357759 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357761 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:31:20.361319 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357764 5091 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357766 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357770 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357772 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357775 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357777 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357780 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357783 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357785 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357788 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357790 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357793 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357795 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357798 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357801 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357803 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357806 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357809 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357811 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:31:20.362191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357814 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357817 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357819 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357822 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357825 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357828 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357830 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357833 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357839 5091 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357841 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357845 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357848 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357851 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.357853 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358335 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358340 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358344 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358347 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358349 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:31:20.362975 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358352 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358355 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358357 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358360 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358363 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358366 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358368 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358372 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358376 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358379 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358381 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358384 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358386 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358389 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358392 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358395 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358397 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358400 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358402 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:31:20.363758 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358405 5091 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358408 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358411 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358413 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358417 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358419 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358422 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358425 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358427 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358430 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358432 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358435 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358437 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358440 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358443 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358445 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358449 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358453 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358456 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:31:20.364585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358459 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358463 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358466 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358469 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358472 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358475 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358478 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358481 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358483 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358486 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358488 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358491 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358494 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358496 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358499 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358502 5091 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358504 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358507 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358510 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:31:20.365377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358513 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358515 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358518 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358520 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358523 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358525 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358528 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358531 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358533 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358536 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358538 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358541 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358544 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358546 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358549 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358551 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358553 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358556 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358559 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358562 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:31:20.366191 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358564 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358567 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358569 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358572 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358575 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358577 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358580 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358582 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358585 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358587 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358590 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358592 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358595 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358598 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358601 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358607 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358610 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358613 5091 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358615 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358618 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:31:20.367024 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358621 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358623 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358627 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.358629 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358726 5091 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358737 5091 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358745 5091 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358750 5091 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358755 5091 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358759 5091 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358763 5091 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358767 5091 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358770 5091 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358774 5091 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358777 5091 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358781 5091 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358784 5091 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358787 5091 flags.go:64] FLAG: --cgroup-root="" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358789 5091 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358793 5091 flags.go:64] FLAG: --client-ca-file="" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358796 5091 flags.go:64] FLAG: --cloud-provider="external" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358799 5091 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358803 5091 flags.go:64] FLAG: --cluster-domain="" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358806 5091 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 19:31:20.367851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358809 5091 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358813 5091 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358816 5091 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358823 5091 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358826 5091 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358829 5091 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358834 5091 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358837 5091 flags.go:64] FLAG: --contention-profiling="false" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358840 5091 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358843 5091 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358846 5091 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358849 5091 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358853 5091 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358857 5091 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358860 5091 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358862 5091 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358866 5091 flags.go:64] FLAG: --enable-server="true" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358869 5091 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358873 5091 flags.go:64] FLAG: --event-burst="100" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358876 5091 flags.go:64] FLAG: --event-qps="50" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358879 5091 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358882 5091 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358885 5091 flags.go:64] FLAG: --eviction-hard="" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358889 5091 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 19:31:20.368817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358892 5091 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358895 5091 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358898 5091 flags.go:64] FLAG: --eviction-soft="" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358901 5091 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358904 5091 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358907 5091 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358910 5091 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358913 5091 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358916 5091 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358919 5091 flags.go:64] FLAG: --feature-gates="" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358926 5091 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358932 5091 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358935 5091 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358938 5091 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358942 5091 flags.go:64] FLAG: --healthz-port="10248" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358946 5091 flags.go:64] FLAG: --help="false" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358948 5091 flags.go:64] FLAG: --hostname-override="ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358952 5091 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358955 5091 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358958 5091 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358962 5091 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358965 5091 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358969 5091 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 19:31:20.369867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358972 5091 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358975 5091 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358978 5091 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358981 5091 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358984 5091 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358987 5091 flags.go:64] FLAG: --kube-reserved="" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358990 5091 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358993 5091 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358996 5091 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.358998 5091 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359001 5091 flags.go:64] FLAG: --lock-file="" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359004 5091 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359007 5091 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359010 5091 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359015 5091 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359018 5091 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359021 5091 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359024 5091 flags.go:64] FLAG: --logging-format="text" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359027 5091 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359031 5091 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359035 5091 flags.go:64] FLAG: --manifest-url="" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359039 5091 flags.go:64] FLAG: --manifest-url-header="" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359043 5091 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359048 5091 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359052 5091 flags.go:64] FLAG: --max-pods="110" Oct 30 19:31:20.370808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359055 5091 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359058 5091 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359061 5091 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359063 5091 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359067 5091 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359070 5091 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359072 5091 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359080 5091 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359083 5091 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359086 5091 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359089 5091 flags.go:64] FLAG: --pod-cidr="" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359092 5091 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359094 5091 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359098 5091 flags.go:64] FLAG: --pods-per-core="0" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359101 5091 flags.go:64] FLAG: --port="10250" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359104 5091 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359107 5091 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-082e3b01570ab4aae" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359110 5091 flags.go:64] FLAG: --qos-reserved="" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359113 5091 flags.go:64] FLAG: --read-only-port="10255" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359116 5091 flags.go:64] FLAG: --register-node="true" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359119 5091 flags.go:64] FLAG: --register-with-taints="" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359123 5091 flags.go:64] FLAG: --registry-burst="10" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359125 5091 flags.go:64] FLAG: --registry-qps="5" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359128 5091 flags.go:64] FLAG: --reserved-cpus="" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359131 5091 flags.go:64] FLAG: --reserved-memory="" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359135 5091 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 19:31:20.371834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359138 5091 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359159 5091 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359164 5091 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359167 5091 flags.go:64] FLAG: --runonce="false" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359172 5091 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359175 5091 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359178 5091 flags.go:64] FLAG: --seccomp-default="false" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359181 5091 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359184 5091 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359188 5091 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359190 5091 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359193 5091 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359196 5091 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359199 5091 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359202 5091 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359205 5091 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359208 5091 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359211 5091 flags.go:64] FLAG: --system-cgroups="" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359214 5091 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359220 5091 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359224 5091 flags.go:64] FLAG: --tls-cert-file="" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359226 5091 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359231 5091 flags.go:64] FLAG: --tls-min-version="" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359234 5091 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359237 5091 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 19:31:20.372935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359240 5091 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359243 5091 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359246 5091 flags.go:64] FLAG: --v="2" Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359251 5091 flags.go:64] FLAG: --version="false" Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359255 5091 flags.go:64] FLAG: --vmodule="" Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359259 5091 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359262 5091 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359454 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359463 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359468 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359473 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359482 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359486 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359491 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359495 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359499 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359502 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359506 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359510 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359514 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359519 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:31:20.373571 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359523 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359527 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359531 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359535 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359539 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359542 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359547 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359551 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359555 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359557 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359560 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359565 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359571 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359576 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359582 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359587 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359591 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359594 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359597 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:31:20.374086 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359601 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359604 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359607 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359612 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359614 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359618 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359620 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359623 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359626 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359628 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359631 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359634 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359636 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359639 5091 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359642 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359644 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359647 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359650 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359652 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359655 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:31:20.374568 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359658 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359660 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359663 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359665 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359668 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359670 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359673 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359676 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359678 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359681 5091 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359683 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359686 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359689 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359692 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359695 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359698 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359701 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359704 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359706 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359709 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:31:20.375063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359712 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359715 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359718 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359720 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359723 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359725 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359728 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359730 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359733 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359736 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359738 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359741 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359744 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359746 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359749 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359751 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359754 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359757 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359759 5091 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:31:20.375569 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359762 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359766 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359769 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359772 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359776 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359780 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359783 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359786 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359790 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359793 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359795 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359798 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359801 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359803 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.359831 5091 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359866 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:31:20.376042 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359871 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359873 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359876 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359879 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359881 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359884 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359887 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359889 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359892 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359895 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359897 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359900 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359903 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359906 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359908 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359911 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359913 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359916 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359919 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:31:20.376508 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359921 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359924 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359926 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359930 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359932 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359937 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359939 5091 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359942 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359946 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359951 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359954 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359956 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359959 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359962 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359965 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359968 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359970 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359973 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359976 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359978 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:31:20.377010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359981 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359983 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359986 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359988 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359991 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359993 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359996 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.359998 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360002 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360006 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360010 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360013 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360018 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360022 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360026 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360029 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360033 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360037 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360040 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:31:20.377781 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360042 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360045 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360048 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360051 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360054 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360056 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360059 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360063 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360067 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360072 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360076 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360081 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360083 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360086 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360089 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360091 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360094 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360097 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360099 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:31:20.378290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360102 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360105 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360107 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360110 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360112 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360115 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360117 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360120 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360122 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360125 5091 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360127 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360131 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360134 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360137 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360139 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360157 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360163 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360167 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360170 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360173 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:31:20.378766 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360175 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360179 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360182 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360184 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360187 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360190 5091 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360192 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.360195 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.363759 5091 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.363773 5091 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.363791 5091 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.363795 5091 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363847 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363852 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363856 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363859 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363862 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363865 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363867 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:31:20.379354 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363870 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363873 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363875 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363878 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363880 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363883 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363886 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363888 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363891 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363895 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363898 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363901 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363904 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363906 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363909 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363912 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363915 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363918 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363920 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363923 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:31:20.379913 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363926 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363928 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363931 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363933 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363936 5091 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363939 5091 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363943 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363946 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363950 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363952 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363955 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363958 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363961 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363964 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363968 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363971 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363974 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363977 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363980 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:31:20.380598 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363982 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363985 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363988 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363991 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363994 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363997 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.363999 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364002 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364004 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364007 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364009 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364012 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364014 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364017 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364019 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364022 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364024 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364027 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364029 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364032 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:31:20.381326 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364035 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364038 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364041 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364043 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364046 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364049 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364051 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364054 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364056 5091 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364059 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364061 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364064 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364066 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364069 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364072 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364075 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364078 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364080 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364083 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:31:20.382115 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364086 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364088 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364090 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364093 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364096 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364098 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364101 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364103 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364106 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364108 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364111 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364113 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364116 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364119 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364122 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364124 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364127 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364130 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364132 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364135 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:31:20.382777 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.364137 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.364184 5091 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365083 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365128 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365139 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365162 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365167 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365173 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365178 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365182 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365187 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365192 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365197 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365202 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365206 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365210 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:31:20.383782 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365221 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365225 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365229 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365234 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365239 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365243 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365247 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365254 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365262 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365267 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365272 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365278 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365289 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365294 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365298 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365303 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365307 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365312 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365317 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:31:20.384473 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365345 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365350 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365354 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365359 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365363 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365373 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365377 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365382 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365386 5091 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365390 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365395 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365400 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365404 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365409 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365413 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365417 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365421 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365425 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365435 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:31:20.385203 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365439 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365443 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365448 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365452 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365456 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365460 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365465 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365470 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365474 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365478 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365483 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365487 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365496 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365500 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365530 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365535 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365539 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365543 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365547 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365552 5091 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:31:20.385712 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365556 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365560 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365564 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365568 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365573 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365582 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365586 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365590 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365594 5091 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365598 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365602 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365605 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365609 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365614 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365618 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365633 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365637 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365642 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365651 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365655 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:31:20.386610 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365661 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365667 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365671 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365675 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365713 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365876 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365881 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365887 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365892 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365896 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365900 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365904 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365908 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.365913 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.365961 5091 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366031 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 19:31:20.387228 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366039 5091 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366045 5091 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366052 5091 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366063 5091 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366068 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366075 5091 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366082 5091 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366088 5091 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366092 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366096 5091 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366100 5091 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366105 5091 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366108 5091 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366110 5091 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366114 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366116 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366119 5091 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366122 5091 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366125 5091 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 19:31:20.387622 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366128 5091 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366130 5091 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366133 5091 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366136 5091 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366139 5091 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366161 5091 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366167 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366170 5091 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366173 5091 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366175 5091 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366178 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366181 5091 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366183 5091 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366186 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366189 5091 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366191 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366196 5091 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366198 5091 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366202 5091 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366205 5091 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 19:31:20.388078 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366208 5091 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366210 5091 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366213 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366216 5091 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366218 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366221 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366223 5091 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366226 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366229 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366231 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366234 5091 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366236 5091 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366239 5091 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366242 5091 feature_gate.go:386] unrecognized feature gate: Example Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366244 5091 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366247 5091 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366250 5091 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366253 5091 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366255 5091 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366258 5091 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 19:31:20.388630 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366261 5091 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366263 5091 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366266 5091 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366268 5091 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366271 5091 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366274 5091 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366276 5091 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366279 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366282 5091 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366285 5091 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366289 5091 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366291 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366294 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366297 5091 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366299 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366304 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366307 5091 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366310 5091 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366312 5091 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366315 5091 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 19:31:20.389113 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366318 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366321 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366323 5091 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366326 5091 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366329 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366331 5091 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366334 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366336 5091 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366339 5091 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366342 5091 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366344 5091 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366347 5091 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366349 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366352 5091 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366355 5091 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366358 5091 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366360 5091 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366364 5091 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366367 5091 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366370 5091 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 19:31:20.389638 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366374 5091 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366377 5091 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366381 5091 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366383 5091 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366386 5091 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:20.366389 5091 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.366449 5091 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.367384 5091 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.367442 5091 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.367629 5091 server.go:1014] "Starting client certificate rotation" Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.367768 5091 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.367825 5091 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 19:25:39 +0000 UTC" deadline="2028-04-03 08:05:57.675568066 +0000 UTC" Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.367852 5091 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12492h34m37.307717709s" Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.368118 5091 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.369176 5091 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.370334 5091 log.go:25] "Validated CRI v1 runtime API" Oct 30 19:31:20.390121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.372038 5091 log.go:25] "Validated CRI v1 image API" Oct 30 19:31:20.390534 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.372812 5091 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 19:31:20.390534 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.374801 5091 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 e3b0b617-352c-4958-8430-22033d746ba5:/dev/nvme0n1p3 e9817087-0aab-4af7-853b-f3ea4b800d4b:/dev/nvme0n1p4] Oct 30 19:31:20.390618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.374815 5091 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00501ced5606129d0345f27bee1112c7caab5f0d7a6b940d1eb3316f85a839c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00501ced5606129d0345f27bee1112c7caab5f0d7a6b940d1eb3316f85a839c8/userdata/shm major:0 minor:330 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/013f0924a531f605911690a8934cbfa44dc6804bf0631ef95546eda783d9e48f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/013f0924a531f605911690a8934cbfa44dc6804bf0631ef95546eda783d9e48f/userdata/shm major:0 minor:238 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07e65c955f517b432c546179e4db92833bc0e84cf063c74ee199eca9f315ef21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07e65c955f517b432c546179e4db92833bc0e84cf063c74ee199eca9f315ef21/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0eb71ea148ca645b71236656f8c56e4aa6e92120dfee48fcb650f0d005b3b047/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0eb71ea148ca645b71236656f8c56e4aa6e92120dfee48fcb650f0d005b3b047/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f92a5095fb5066e5da408999533fdb735868525098e3e3f9e008bf9b3f3003e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f92a5095fb5066e5da408999533fdb735868525098e3e3f9e008bf9b3f3003e/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17e9bb78e607eda1bf5e72de4044bbd3579c2fd9b4221af48e9c779d8de68156/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17e9bb78e607eda1bf5e72de4044bbd3579c2fd9b4221af48e9c779d8de68156/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2759d713bb56bcdeba60dd019aafb131dfe684d5b2c0bf8a711247cfa6333336/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2759d713bb56bcdeba60dd019aafb131dfe684d5b2c0bf8a711247cfa6333336/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f065122df4f28ace6aa0054779e9e09fdcb31b0093ad0f24d81cfb30ab25f9b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f065122df4f28ace6aa0054779e9e09fdcb31b0093ad0f24d81cfb30ab25f9b/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/515e6a514b5bac59facab7b59d56084b49d1112f3e585cb8f258afd2c416e556/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/515e6a514b5bac59facab7b59d56084b49d1112f3e585cb8f258afd2c416e556/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5deed405f40cce56eb0d6e42ae157a656bea9637c3f425661d727b051239159f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5deed405f40cce56eb0d6e42ae157a656bea9637c3f425661d727b051239159f/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6b36afa016de24b8aa10c69c1f397522ee3d58e603a05adf04708b9df644e170/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6b36afa016de24b8aa10c69c1f397522ee3d58e603a05adf04708b9df644e170/userdata/shm major:0 minor:93 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/892e9f6edfc7c956d0e78c4669fa1043e08537af5400263662721c7373b31963/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/892e9f6edfc7c956d0e78c4669fa1043e08537af5400263662721c7373b31963/userdata/shm major:0 minor:354 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96394dbd894cfd2ad3a566b2b843e24a858c2f2f669a455f5b7f2d27397c973b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96394dbd894cfd2ad3a566b2b843e24a858c2f2f669a455f5b7f2d27397c973b/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/97a89a6ee7741dfeefcac263c266168d14613ae7ff6fcd0bbfd2d3badc97cdb4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/97a89a6ee7741dfeefcac263c266168d14613ae7ff6fcd0bbfd2d3badc97cdb4/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a2f0e1459db066802ca9b3b5072041807b613908bdf9d00ee78221c5d7fc3730/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a2f0e1459db066802ca9b3b5072041807b613908bdf9d00ee78221c5d7fc3730/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a6161bc1168b86cd57c41a6ff433a7fd1aaf76078435b1e6134db395ecfad2d5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a6161bc1168b86cd57c41a6ff433a7fd1aaf76078435b1e6134db395ecfad2d5/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ad622f0d7340410ea5ffcf586d39ff66d514b44ba6c7502e4564e69c8b7bca67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ad622f0d7340410ea5ffcf586d39ff66d514b44ba6c7502e4564e69c8b7bca67/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b348f37bb5e67bf1200b8567986992f023d9f6f3e021d4a1004dae81ef5fc873/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b348f37bb5e67bf1200b8567986992f023d9f6f3e021d4a1004dae81ef5fc873/userdata/shm major:0 minor:236 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c50974d77e3047f90b03bdb6069b22f656f65aff6ae918799c0669b692cf5e79/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c50974d77e3047f90b03bdb6069b22f656f65aff6ae918799c0669b692cf5e79/userdata/shm major:0 minor:234 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c722584061fea70d48ce3aaddd579e3b67277faa4f0b3800de3bb2e73c882fb6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c722584061fea70d48ce3aaddd579e3b67277faa4f0b3800de3bb2e73c882fb6/userdata/shm major:0 minor:218 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c951fe4859bfa6a7da5e6428c2209e6ae00baa8cae45386104660f22be56c53f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c951fe4859bfa6a7da5e6428c2209e6ae00baa8cae45386104660f22be56c53f/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d1365cbce7525d7206cba488262330f62c465d65a50e638864581050077f6ce1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d1365cbce7525d7206cba488262330f62c465d65a50e638864581050077f6ce1/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d23691ccb7c2c62b4dfb0b27a3ff430ab94d1b198b3876a7ebd5e846ed247f4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d23691ccb7c2c62b4dfb0b27a3ff430ab94d1b198b3876a7ebd5e846ed247f4c/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db6295ed5bfdd69e1ede730aeec1442244a828472e1c56df5afc8724f5307958/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db6295ed5bfdd69e1ede730aeec1442244a828472e1c56df5afc8724f5307958/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee2cb1522ccc3e004a8be9b1aaa2a11a50e3cba75920aee3076e57dd2dc664f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee2cb1522ccc3e004a8be9b1aaa2a11a50e3cba75920aee3076e57dd2dc664f6/userdata/shm major:0 minor:240 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f9b6276abba369dffcce5917f8594ebe32418e7876b9d4048757aa2aaee0f04e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f9b6276abba369dffcce5917f8594ebe32418e7876b9d4048757aa2aaee0f04e/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/01188e4b-c110-477a-ba88-d9f48741c0a2/volumes/kubernetes.io~projected/kube-api-access-5ghq4:{mountpoint:/var/lib/kubelet/pods/01188e4b-c110-477a-ba88-d9f48741c0a2/volumes/kubernetes.io~projected/kube-api-access-5ghq4 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02832de1-9584-4e12-9b94-8b6b52a71941/volumes/kubernetes.io~projected/kube-api-access-2dqtb:{mountpoint:/var/lib/kubelet/pods/02832de1-9584-4e12-9b94-8b6b52a71941/volumes/kubernetes.io~projected/kube-api-access-2dqtb major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/067a4436-690d-4b2c-aaf2-1d84830b7287/volumes/kubernetes.io~projected/kube-api-access-5cvnm:{mountpoint:/var/lib/kubelet/pods/067a4436-690d-4b2c-aaf2-1d84830b7287/volumes/kubernetes.io~projected/kube-api-access-5cvnm major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/067a4436-690d-4b2c-aaf2-1d84830b7287/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/067a4436-690d-4b2c-aaf2-1d84830b7287/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/06bbc899-e828-4a45-bd9a-3e703c4416e7/volumes/kubernetes.io~projected/kube-api-access-847qk:{mountpoint:/var/lib/kubelet/pods/06bbc899-e828-4a45-bd9a-3e703c4416e7/volumes/kubernetes.io~projected/kube-api-access-847qk major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22599c63-1821-4c98-9096-9d4c43549ab7/volumes/kubernetes.io~projected/kube-api-access-kk5qr:{mountpoint:/var/lib/kubelet/pods/22599c63-1821-4c98-9096-9d4c43549ab7/volumes/kubernetes.io~projected/kube-api-access-kk5qr major:0 minor:347 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/22599c63-1821-4c98-9096-9d4c43549ab7/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/22599c63-1821-4c98-9096-9d4c43549ab7/volumes/kubernetes.io~secret/signing-key major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24ac5496-c680-4b2a-8362-f12b662df997/volumes/kubernetes.io~projected/kube-api-access-qvx5x:{mountpoint:/var/lib/kubelet/pods/24ac5496-c680-4b2a-8362-f12b662df997/volumes/kubernetes.io~projected/kube-api-access-qvx5x major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/24ac5496-c680-4b2a-8362-f12b662df997/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/24ac5496-c680-4b2a-8362-f12b662df997/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2f2ee082-2070-4390-baf3-55fa09089049/volumes/kubernetes.io~projected/kube-api-access-6vkzm:{mountpoint:/var/lib/kubelet/pods/2f2ee082-2070-4390-baf3-55fa09089049/volumes/kubernetes.io~projected/kube-api-access-6vkzm major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35066f63-29d2-42da-83b7-edf8894d8e5c/volumes/kubernetes.io~projected/kube-api-access-24nrk:{mountpoint:/var/lib/kubelet/pods/35066f63-29d2-42da-83b7-edf8894d8e5c/volumes/kubernetes.io~projected/kube-api-access-24nrk major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41c1116a-2830-4ea1-aae5-3a052f073293/volumes/kubernetes.io~projected/kube-api-access-rh66m:{mountpoint:/var/lib/kubelet/pods/41c1116a-2830-4ea1-aae5-3a052f073293/volumes/kubernetes.io~projected/kube-api-access-rh66m major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/41c1116a-2830-4ea1-aae5-3a052f073293/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/41c1116a-2830-4ea1-aae5-3a052f073293/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/534aacd9-7102-4a50-8696-c50362ce87e4/volumes/kubernetes.io~projected/kube-api-access-ql7gq:{mountpoint:/var/lib/kubelet/pods/534aacd9-7102-4a50-8696-c50362ce87e4/volumes/kubernetes.io~projected/kube-api-access-ql7gq major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/534aacd9-7102-4a50-8696-c50362ce87e4/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/534aacd9-7102-4a50-8696-c50362ce87e4/volumes/kubernetes.io~secret/cert major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bfb73f4-9290-4440-938a-feb9fafa3287/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5bfb73f4-9290-4440-938a-feb9fafa3287/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:276 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6693a492-433f-4614-a8e3-71ee75c177c2/volumes/kubernetes.io~projected/kube-api-access-jhzl5:{mountpoint:/var/lib/kubelet/pods/6693a492-433f-4614-a8e3-71ee75c177c2/volumes/kubernetes.io~projected/kube-api-access-jhzl5 major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c260520-55b0-44ca-8ccb-e15cab54fa90/volumes/kubernetes.io~projected/kube-api-access-dtmz6:{mountpoint:/var/lib/kubelet/pods/6c260520-55b0-44ca-8ccb-e15cab54fa90/volumes/kubernetes.io~projected/kube-api-access-dtmz6 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c260520-55b0-44ca-8ccb-e15cab54fa90/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/6c260520-55b0-44ca-8ccb-e15cab54fa90/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6c633f50-fd32-4440-aefc-79e44203424e/volumes/kubernetes.io~projected/kube-api-access-j9vjq:{mountpoint:/var/lib/kubelet/pods/6c633f50-fd32-4440-aefc-79e44203424e/volumes/kubernetes.io~projected/kube-api-access-j9vjq major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f0f5eb3-92c0-4822-867c-cf802c9e8490/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/6f0f5eb3-92c0-4822-867c-cf802c9e8490/volumes/kubernetes.io~secret/agent-certs major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/731f78f9-9938-4170-a8ff-10ba7a863759/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/731f78f9-9938-4170-a8ff-10ba7a863759/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/731f78f9-9938-4170-a8ff-10ba7a863759/volumes/kubernetes.io~projected/kube-api-access-rhvp8:{mountpoint:/var/lib/kubelet/pods/731f78f9-9938-4170-a8ff-10ba7a863759/volumes/kubernetes.io~projected/kube-api-access-rhvp8 major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/731f78f9-9938-4170-a8ff-10ba7a863759/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/731f78f9-9938-4170-a8ff-10ba7a863759/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/731f78f9-9938-4170-a8ff-10ba7a863759/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/731f78f9-9938-4170-a8ff-10ba7a863759/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d903311-c71c-4197-8ac3-486702bd46e9/volumes/kubernetes.io~projected/kube-api-access-qbtrc:{mountpoint:/var/lib/kubelet/pods/7d903311-c71c-4197-8ac3-486702bd46e9/volumes/kubernetes.io~projected/kube-api-access-qbtrc major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7fa163f8-129c-4f64-ad4b-ad3cc08bed13/volumes/kubernetes.io~projected/kube-api-access-5t8r2:{mountpoint:/var/lib/kubelet/pods/7fa163f8-129c-4f64-ad4b-ad3cc08bed13/volumes/kubernetes.io~projected/kube-api-access-5t8r2 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8161b0bd-5184-44fc-864e-869dc7737a55/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/8161b0bd-5184-44fc-864e-869dc7737a55/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8161b0bd-5184-44fc-864e-869dc7737a55/volumes/kubernetes.io~projected/kube-api-access-jvxqn:{mountpoint:/var/lib/kubelet/pods/8161b0bd-5184-44fc-864e-869dc7737a55/volumes/kubernetes.io~projected/kube-api-access-jvxqn major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8161b0bd-5184-44fc-864e-869dc7737a55/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/8161b0bd-5184-44fc-864e-869dc7737a55/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8161b0bd-5184-44fc-864e-869dc7737a55/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/8161b0bd-5184-44fc-864e-869dc7737a55/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89bcca43-c158-47fb-9cb9-1442abc2cd45/volumes/kubernetes.io~projected/kube-api-access-g5c84:{mountpoint:/var/lib/kubelet/pods/89bcca43-c158-47fb-9cb9-1442abc2cd45/volumes/kubernetes.io~projected/kube-api-access-g5c84 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d7792ad-14ae-482b-9def-eda8e029dcda/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/8d7792ad-14ae-482b-9def-eda8e029dcda/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d7792ad-14ae-482b-9def-eda8e029dcda/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/8d7792ad-14ae-482b-9def-eda8e029dcda/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d7792ad-14ae-482b-9def-eda8e029dcda/volumes/kubernetes.io~projected/kube-api-access-52n6r:{mountpoint:/var/lib/kubelet/pods/8d7792ad-14ae-482b-9def-eda8e029dcda/volumes/kubernetes.io~projected/kube-api-access-52n6r major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9458eb02-2921-4f34-9578-60056e3505a4/volumes/kubernetes.io~projected/kube-api-access-cnpd5:{mountpoint:/var/lib/kubelet/pods/9458eb02-2921-4f34-9578-60056e3505a4/volumes/kubernetes.io~projected/kube-api-access-cnpd5 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9e4295e8-368d-4799-b1c4-70355610378c/volumes/kubernetes.io~projected/kube-api-access-phptb:{mountpoint:/var/lib/kubelet/pods/9e4295e8-368d-4799-b1c4-70355610378c/volumes/kubernetes.io~projected/kube-api-access-phptb major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2cace4b-3a50-4fe5-8ead-c9765b9540d9/volumes/kubernetes.io~projected/kube-api-access-t9v4h:{mountpoint:/var/lib/kubelet/pods/a2cace4b-3a50-4fe5-8ead-c9765b9540d9/volumes/kubernetes.io~projected/kube-api-access-t9v4h major:0 minor:334 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8e91049-4a10-4472-922d-7b7875e7de3a/volumes/kubernetes.io~projected/kube-api-access-rrblb:{mountpoint:/var/lib/kubelet/pods/a8e91049-4a10-4472-922d-7b7875e7de3a/volumes/kubernetes.io~projected/kube-api-access-rrblb major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8e91049-4a10-4472-922d-7b7875e7de3a/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/a8e91049-4a10-4472-922d-7b7875e7de3a/volumes/kubernetes.io~secret/default-certificate major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a8e91049-4a10-4472-922d-7b7875e7de3a/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/a8e91049-4a10-4472-922d-7b7875e7de3a/volumes/kubernetes.io~secret/stats-auth major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ad103ddc-523c-4881-8526-00458255bb6c/volumes/kubernetes.io~projected/kube-api-access-qfqd8:{mountpoint:/var/lib/kubelet/pods/ad103ddc-523c-4881-8526-00458255bb6c/volumes/kubernetes.io~projected/kube-api-access-qfqd8 major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/af2922a1-2b0f-4ee1-8ef3-4911d029e93d/volumes/kubernetes.io~projected/kube-api-access-hwf6q:{mountpoint:/var/lib/kubelet/pods/af2922a1-2b0f-4ee1-8ef3-4911d029e93d/volumes/kubernetes.io~projected/kube-api-access-hwf6q major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad/volumes/kubernetes.io~projected/kube-api-access-bzlcx:{mountpoint:/var/lib/kubelet/pods/b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad/volumes/kubernetes.io~projected/kube-api-access-bzlcx major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b6e429ff-3fb2-48da-a77b-7806b90139e5/volumes/kubernetes.io~projected/kube-api-access-wzv8w:{mountpoint:/var/lib/kubelet/pods/b6e429ff-3fb2-48da-a77b-7806b90139e5/volumes/kubernetes.io~projected/kube-api-access-wzv8w major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc37d377-9cb8-4f67-aa5d-1368a886533c/volumes/kubernetes.io~projected/kube-api-access-5fcwx:{mountpoint:/var/lib/kubelet/pods/cc37d377-9cb8-4f67-aa5d-1368a886533c/volumes/kubernetes.io~projected/kube-api-access-5fcwx major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d95b4719-782e-47f3-9a7d-ea04268f82da/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d95b4719-782e-47f3-9a7d-ea04268f82da/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d95b4719-782e-47f3-9a7d-ea04268f82da/volumes/kubernetes.io~projected/kube-api-access-vglqr:{mountpoint:/var/lib/kubelet/pods/d95b4719-782e-47f3-9a7d-ea04268f82da/volumes/kubernetes.io~projected/kube-api-access-vglqr major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d95b4719-782e-47f3-9a7d-ea04268f82da/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d95b4719-782e-47f3-9a7d-ea04268f82da/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f031b635-0cf3-4670-a9c6-278f14922b37/volumes/kubernetes.io~projected/kube-api-access-glnjk:{mountpoint:/var/lib/kubelet/pods/f031b635-0cf3-4670-a9c6-278f14922b37/volumes/kubernetes.io~projected/kube-api-access-glnjk major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb5d13ef-8197-4960-b142-d8a1bdd025e2/volumes/kubernetes.io~projected/kube-api-access-7fnzv:{mountpoint:/var/lib/kubelet/pods/fb5d13ef-8197-4960-b142-d8a1bdd025e2/volumes/kubernetes.io~projected/kube-api-access-7fnzv major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/81662b4940745b95b7680ee839b870df8b4c3f32a355b893c9be7329acc204ac/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/00ae9f10ffbf6c4bdaf29860dbac214c9a7e0ba8a5931f3049826e381a6357d8/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/33a681da063ee214eba9eb44dc45909fbc675969c70ac7b8e91502e8849011de/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/da0aecf334dc67c6c0152ab943f8a7d6b271e1cf54284d0486daff1e46d2f071/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/972cd0b388631530ee15d21d9719ceae4b9cf5cffabbb1a3668d46a6870f7f1d/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/4f0b48896664bae6457fc627a70f26ddf850835f647052ef2d084f9139eeab8e/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e1dee513322f66505399fb3d8c88c6c9eb607a491ce554a642c7c75c1caa94d6/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/16f7a6d9b3e2a70762f23d7989c26650a954c6fe444e3164ad84cd6b19e7652d/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/498b911da579b692fd6769748dda0720d70e9ed3a6e3adce455a473e342f56fa/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/8fe042d07f5780fe1a008c1ca73431f34cf247cd6df9a2d593c448991b1e1d4c/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/6602d8e1c1dfcb3fea731a8384f15f399a2e6d7211c888fcadddfe81be47d468/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/c66669dd1a7abe6aa66e8e317950c7eb678efecccda00dc6d508e1adda387a97/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/32475f52c4a5ef8d68d7f3a2d68c32b2d2ff8836c584ac44c87aa8b64b56b11a/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/ec063ac102fe82c827e88b851629e60d0f62b1eb99ebc9b09daa4ce1e2046e24/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/30bbca2e7651e1176de861a708d1fa4b4760ed042394321670287f59081785dd/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/581c76741fab618c9b0be6cf29bdc52853599ea892b5d3c25ed7f2e4eb52383f/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/09e7b3c427dbca6b37c6f1ac0c1f96ce5954ad44e053deeed6a32d83e60d6ebc/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/3bf4b495968d7111d8e940297dbd4f8735fafa39b25d41adc633c0b0bfe07632/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/4d7eb1ed8d1f75a6ea46d9321c812bb229ad8b0a42d129afec085156e12ee55a/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/1247353feaa492dbcc42c9e832eca017916c3107becb2250e0b02566bb89db07/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/de4dc91f0dd352c90aba2cd0040b1e006ad83eca0b29af07fcc7bde2db862b40/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/3c09d41595565d72e8841720fc5acb73aef8b88aacda3641bb2bc66d4a0ee6b4/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/08f59909bf5dd515ecb6f8b743eddff97958be445de8ccb227699da11c71fa0a/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/601d5b7f43d45c284c5b9fee8543728892cef3ba0be71a9f4938bdc7f59700e1/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/22254a5bce4eba92e88cbc85456609560cd1acf42c772996026dd7500bc605e9/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/2bfc8c8146a57935047de1bffeee33b022fdfdf180545344a70b2251e772b1e1/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/433fddb0285d66e0519a9d38538432a54e75aa1bcfa150a5c9ec34b01f029380/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/13cd1d5122d06e72cf64226a791dcbddde32920b819582dbf0bbb2fd3e78b7b1/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/e85bfbe2ef520967a9772678594e4361950e64c7edd1fffc1337db7c05de9a3b/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/781d866eeeb79b620dfcb96340c85b0bc2e8c2c52420035f9b111ad35c578ef1/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/9bc2a48a1effe69642e9f71dbbd4150b88f3c8154a926709d6909ac3431c8636/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/45cd83d1bc0fed4aed5c8603ac2f007bb135de7e6268000d68eb2d9b9af616e2/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/6af11149dd96a43218b099420f4736469914a5cb87946993fe04b5259dd03b01/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/a2617a8812e4d079929766c8b257564b8e625c83da83a478f4728d941bdfb968/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/1647bcdc7baf3e05fd0d8f1732d748311e6d24a6b4b0b5e1f81960322ca629d5/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/7f44c39d5408151d95b7902f375593cd133f27d370966225652041372e307444/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/9cfc140d2dead8d1b8ba346f737595c275b26793169f3673e1041c1b30a025a3/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/185df132a8aaf5c02c3e4990b71d1151152590b018de8b2c8100d932518a7165/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/f04d9ba65597d2f312a82194084eeeca5d7efd2c1499145cf6b8194a2a849a90/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/7df46ed6c0c3359cded8d2e5495843db6edf8e94383c1db808acfd4327830972/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-308:{mountpoint:/var/lib/containers/storage/overlay/919cb1b8f165bb9bcb5989e895a4f5fab3ca8da714f0a8a5e6b704fc5ca9c723/merged major:0 minor:308 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/b1d7d661622a676b4a9f57740cec64dff1dfea8a44762ce11e70b433755b491c/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/fbb2590408a982e6cbcb8c13b9879aaa0252831a9659bd7421d81d57b72d288d/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/ced96cc544ab3de4104281092dd4cfd913bb0c7a137ff56b4835673e12c71fee/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/a30be611d45a541865cb5c74c7d4367b9054f9d2e5ebeb69bc561042309cfb9f/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/f33c9dfebff841f9ab092b4907e009b579a341adf883082789c0323c5c717426/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/0ceba1362a9c53ff3ebee7bb8d3995fc46c1871dbffda9aaafe61410079b46ea/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-363:{mountpoint:/var/lib/containers/storage/overlay/d3084025e2b78aec6b1cfdf9cf7861122ea396cc71b2c393f962b5ada6b33a98/merged major:0 minor:363 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/bb442199452834611ca58f2a5ba13a12bb9eddbfcc63b1a213d833deb18c80a7/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-368:{mountpoint:/var/lib/containers/storage/overlay/c6ff08a9421a9ea18df0fe9f6dead209c7536f570811fb4d401e76af91b54f0c/merged major:0 minor:368 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/e6e2521c4a5fda27cc868922de606ed540505b0238a446d9bb965c59606020ac/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/41a696d9245865ea68951cd1c26ce57639bd4f217f6f81b117f15143286f0a30/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/e48deecfbb845e6ff6816fd8154c32e84be7a5ca6707df6d79f2dd389621f159/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/379e3b3bbe859bb3411426243018f410c88b944de361ec4d2415836badceb105/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/6a2d6c825723585849af78e2cf27ff55115470674f2512bb2e9662d1035b505b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/6ecad4638720b743c6375d5cc05a6b49bb19ca3b06529b03d37fe3717a2afa83/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/b3b33d685b1b9cf5a41843669c25e9bd802605d21358aa66c646c14591c885bc/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/bd38237c8de7f9566901c84be4c5c2e28f6132c57dac5698ac90899f4096739d/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/131bd17a26112157e7c0dc83734d425a97fe9bca5244a72c68f43e0f299bc647/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/8ba4ab6f33f0a42494bfed8cbbdb26644d1284ce66e9f579d2a62d48e392898b/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/5e120a682f999dd2ea0481c9746e30884f8a87c4f863017b64d5e7e28315d3a9/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/cb6e6492e95d0afc9ca6473cfacb17b9483b35f5bad74b3ca69dd2ea85223e52/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.384136 5091 manager.go:217] Machine: {Timestamp:2026-10-30 19:31:20.383342539 +0000 UTC m=+0.078550490 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099190 MemoryCapacity:33164349440 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29767042ffddc5ac88c5bda08f429f SystemUUID:ec297670-42ff-ddc5-ac88-c5bda08f429f BootID:b10e7dd1-5056-4667-8d62-ff94d832cfaf Filesystems:[{Device:/var/lib/kubelet/pods/fb5d13ef-8197-4960-b142-d8a1bdd025e2/volumes/kubernetes.io~projected/kube-api-access-7fnzv DeviceMajor:0 DeviceMinor:66 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/01188e4b-c110-477a-ba88-d9f48741c0a2/volumes/kubernetes.io~projected/kube-api-access-5ghq4 DeviceMajor:0 DeviceMinor:230 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/6693a492-433f-4614-a8e3-71ee75c177c2/volumes/kubernetes.io~projected/kube-api-access-jhzl5 DeviceMajor:0 DeviceMinor:231 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b348f37bb5e67bf1200b8567986992f023d9f6f3e021d4a1004dae81ef5fc873/userdata/shm DeviceMajor:0 DeviceMinor:236 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d23691ccb7c2c62b4dfb0b27a3ff430ab94d1b198b3876a7ebd5e846ed247f4c/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/534aacd9-7102-4a50-8696-c50362ce87e4/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:215 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22599c63-1821-4c98-9096-9d4c43549ab7/volumes/kubernetes.io~projected/kube-api-access-kk5qr DeviceMajor:0 DeviceMinor:347 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582176768 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/8d7792ad-14ae-482b-9def-eda8e029dcda/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07e65c955f517b432c546179e4db92833bc0e84cf063c74ee199eca9f315ef21/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/515e6a514b5bac59facab7b59d56084b49d1112f3e585cb8f258afd2c416e556/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17e9bb78e607eda1bf5e72de4044bbd3579c2fd9b4221af48e9c779d8de68156/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad/volumes/kubernetes.io~projected/kube-api-access-bzlcx DeviceMajor:0 DeviceMinor:232 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/9e4295e8-368d-4799-b1c4-70355610378c/volumes/kubernetes.io~projected/kube-api-access-phptb DeviceMajor:0 DeviceMinor:310 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/8d7792ad-14ae-482b-9def-eda8e029dcda/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/534aacd9-7102-4a50-8696-c50362ce87e4/volumes/kubernetes.io~projected/kube-api-access-ql7gq DeviceMajor:0 DeviceMinor:224 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/7d903311-c71c-4197-8ac3-486702bd46e9/volumes/kubernetes.io~projected/kube-api-access-qbtrc DeviceMajor:0 DeviceMinor:226 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c633f50-fd32-4440-aefc-79e44203424e/volumes/kubernetes.io~projected/kube-api-access-j9vjq DeviceMajor:0 DeviceMinor:293 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-363 DeviceMajor:0 DeviceMinor:363 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8161b0bd-5184-44fc-864e-869dc7737a55/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:212 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/067a4436-690d-4b2c-aaf2-1d84830b7287/volumes/kubernetes.io~projected/kube-api-access-5cvnm DeviceMajor:0 DeviceMinor:227 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/013f0924a531f605911690a8934cbfa44dc6804bf0631ef95546eda783d9e48f/userdata/shm DeviceMajor:0 DeviceMinor:238 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/97a89a6ee7741dfeefcac263c266168d14613ae7ff6fcd0bbfd2d3badc97cdb4/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/89bcca43-c158-47fb-9cb9-1442abc2cd45/volumes/kubernetes.io~projected/kube-api-access-g5c84 DeviceMajor:0 DeviceMinor:58 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582172672 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0eb71ea148ca645b71236656f8c56e4aa6e92120dfee48fcb650f0d005b3b047/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/a8e91049-4a10-4472-922d-7b7875e7de3a/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:202 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/8161b0bd-5184-44fc-864e-869dc7737a55/volumes/kubernetes.io~projected/kube-api-access-jvxqn DeviceMajor:0 DeviceMinor:222 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-368 DeviceMajor:0 DeviceMinor:368 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9458eb02-2921-4f34-9578-60056e3505a4/volumes/kubernetes.io~projected/kube-api-access-cnpd5 DeviceMajor:0 DeviceMinor:207 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/731f78f9-9938-4170-a8ff-10ba7a863759/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:214 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a6161bc1168b86cd57c41a6ff433a7fd1aaf76078435b1e6134db395ecfad2d5/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c951fe4859bfa6a7da5e6428c2209e6ae00baa8cae45386104660f22be56c53f/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f9b6276abba369dffcce5917f8594ebe32418e7876b9d4048757aa2aaee0f04e/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6b36afa016de24b8aa10c69c1f397522ee3d58e603a05adf04708b9df644e170/userdata/shm DeviceMajor:0 DeviceMinor:93 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/731f78f9-9938-4170-a8ff-10ba7a863759/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:213 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/af2922a1-2b0f-4ee1-8ef3-4911d029e93d/volumes/kubernetes.io~projected/kube-api-access-hwf6q DeviceMajor:0 DeviceMinor:65 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8161b0bd-5184-44fc-864e-869dc7737a55/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:225 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/731f78f9-9938-4170-a8ff-10ba7a863759/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:229 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96394dbd894cfd2ad3a566b2b843e24a858c2f2f669a455f5b7f2d27397c973b/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/02832de1-9584-4e12-9b94-8b6b52a71941/volumes/kubernetes.io~projected/kube-api-access-2dqtb DeviceMajor:0 DeviceMinor:209 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/892e9f6edfc7c956d0e78c4669fa1043e08537af5400263662721c7373b31963/userdata/shm DeviceMajor:0 DeviceMinor:354 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f065122df4f28ace6aa0054779e9e09fdcb31b0093ad0f24d81cfb30ab25f9b/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2759d713bb56bcdeba60dd019aafb131dfe684d5b2c0bf8a711247cfa6333336/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/067a4436-690d-4b2c-aaf2-1d84830b7287/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/8161b0bd-5184-44fc-864e-869dc7737a55/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:217 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ee2cb1522ccc3e004a8be9b1aaa2a11a50e3cba75920aee3076e57dd2dc664f6/userdata/shm DeviceMajor:0 DeviceMinor:240 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f92a5095fb5066e5da408999533fdb735868525098e3e3f9e008bf9b3f3003e/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d95b4719-782e-47f3-9a7d-ea04268f82da/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/41c1116a-2830-4ea1-aae5-3a052f073293/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/b6e429ff-3fb2-48da-a77b-7806b90139e5/volumes/kubernetes.io~projected/kube-api-access-wzv8w DeviceMajor:0 DeviceMinor:323 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/24ac5496-c680-4b2a-8362-f12b662df997/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/06bbc899-e828-4a45-bd9a-3e703c4416e7/volumes/kubernetes.io~projected/kube-api-access-847qk DeviceMajor:0 DeviceMinor:204 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5bfb73f4-9290-4440-938a-feb9fafa3287/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:276 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-308 DeviceMajor:0 DeviceMinor:308 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2cace4b-3a50-4fe5-8ead-c9765b9540d9/volumes/kubernetes.io~projected/kube-api-access-t9v4h DeviceMajor:0 DeviceMinor:334 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/d95b4719-782e-47f3-9a7d-ea04268f82da/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00501ced5606129d0345f27bee1112c7caab5f0d7a6b940d1eb3316f85a839c8/userdata/shm DeviceMajor:0 DeviceMinor:330 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/f031b635-0cf3-4670-a9c6-278f14922b37/volumes/kubernetes.io~projected/kube-api-access-glnjk DeviceMajor:0 DeviceMinor:335 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/731f78f9-9938-4170-a8ff-10ba7a863759/volumes/kubernetes.io~projected/kube-api-access-rhvp8 DeviceMajor:0 DeviceMinor:228 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/35066f63-29d2-42da-83b7-edf8894d8e5c/volumes/kubernetes.io~projected/kube-api-access-24nrk DeviceMajor:0 DeviceMinor:60 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8e91049-4a10-4472-922d-7b7875e7de3a/volumes/kubernetes.io~projected/kube-api-access-rrblb DeviceMajor:0 DeviceMinor:206 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/6f0f5eb3-92c0-4822-867c-cf802c9e8490/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:56 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/7fa163f8-129c-4f64-ad4b-ad3cc08bed13/volumes/kubernetes.io~projected/kube-api-access-5t8r2 DeviceMajor:0 DeviceMinor:63 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d95b4719-782e-47f3-9a7d-ea04268f82da/volumes/kubernetes.io~projected/kube-api-access-vglqr DeviceMajor:0 DeviceMinor:59 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/41c1116a-2830-4ea1-aae5-3a052f073293/volumes/kubernetes.io~projected/kube-api-access-rh66m DeviceMajor:0 DeviceMinor:205 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5deed405f40cce56eb0d6e42ae157a656bea9637c3f425661d727b051239159f/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db6295ed5bfdd69e1ede730aeec1442244a828472e1c56df5afc8724f5307958/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d1365cbce7525d7206cba488262330f62c465d65a50e638864581050077f6ce1/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6c260520-55b0-44ca-8ccb-e15cab54fa90/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/ad103ddc-523c-4881-8526-00458255bb6c/volumes/kubernetes.io~projected/kube-api-access-qfqd8 DeviceMajor:0 DeviceMinor:211 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/22599c63-1821-4c98-9096-9d4c43549ab7/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:346 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/8d7792ad-14ae-482b-9def-eda8e029dcda/volumes/kubernetes.io~projected/kube-api-access-52n6r DeviceMajor:0 DeviceMinor:67 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c722584061fea70d48ce3aaddd579e3b67277faa4f0b3800de3bb2e73c882fb6/userdata/shm DeviceMajor:0 DeviceMinor:218 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2f2ee082-2070-4390-baf3-55fa09089049/volumes/kubernetes.io~projected/kube-api-access-6vkzm DeviceMajor:0 DeviceMinor:57 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/cc37d377-9cb8-4f67-aa5d-1368a886533c/volumes/kubernetes.io~projected/kube-api-access-5fcwx DeviceMajor:0 DeviceMinor:64 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a2f0e1459db066802ca9b3b5072041807b613908bdf9d00ee78221c5d7fc3730/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/6c260520-55b0-44ca-8ccb-e15cab54fa90/volumes/kubernetes.io~projected/kube-api-access-dtmz6 DeviceMajor:0 DeviceMinor:208 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/var/lib/kubelet/pods/24ac5496-c680-4b2a-8362-f12b662df997/volumes/kubernetes.io~projected/kube-api-access-qvx5x DeviceMajor:0 DeviceMinor:210 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c50974d77e3047f90b03bdb6069b22f656f65aff6ae918799c0669b692cf5e79/userdata/shm DeviceMajor:0 DeviceMinor:234 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ad622f0d7340410ea5ffcf586d39ff66d514b44ba6c7502e4564e69c8b7bca67/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4048382 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a8e91049-4a10-4472-922d-7b7875e7de3a/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:45 Capacity:31985750016 Type:vfs Inodes:4048382 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:00501ced5606129 MacAddress:e6:c7:f1:66:6c:7a Speed:10000 Mtu:8901} {Name:013f0924a531f60 MacAddress:ea:de:29:58:86:6e Speed:10000 Mtu:8901} {Name:0f92a5095fb5066 MacAddress:b2:a0:9a:76:a6:0b Speed:10000 Mtu:8901} {Name:515e6a514b5bac5 MacAddress:32:85:3c:50:6b:6b Speed:10000 Mtu:8901} {Name:5deed405f40cce5 MacAddress:6e:44:91:4f:b9:b3 Speed:10000 Mtu:8901} {Name:892e9f6edfc7c95 MacAddress:82:75:00:65:8f:00 Speed:10000 Mtu:8901} {Name:96394dbd894cfd2 MacAddress:fa:38:40:d4:9c:2d Speed:10000 Mtu:8901} {Name:97a89a6ee7741df MacAddress:ae:f1:66:81:e3:70 Speed:10000 Mtu:8901} {Name:a6161bc1168b86c MacAddress:9e:19:4c:88:0e:c4 Speed:10000 Mtu:8901} {Name:ad622f0d7340410 MacAddress:16:78:55:94:45:0f Speed:10000 Mtu:8901} {Name:b348f37bb5e67bf MacAddress:86:fa:94:f1:a1:8f Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ee:71:79:5d Speed:0 Mtu:9001} {Name:br-int MacAddress:56:e4:39:6a:ad:ae Speed:0 Mtu:8901} {Name:c50974d77e3047f MacAddress:66:71:a4:05:05:e7 Speed:10000 Mtu:8901} {Name:c722584061fea70 MacAddress:0a:4e:44:f9:6f:33 Speed:10000 Mtu:8901} {Name:d23691ccb7c2c62 MacAddress:3e:8f:c7:2f:c8:e9 Speed:10000 Mtu:8901} {Name:db6295ed5bfdd69 MacAddress:fa:5d:b2:5e:ae:b7 Speed:10000 Mtu:8901} {Name:ee2cb1522ccc3e0 MacAddress:86:2b:09:34:b4:b0 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ee:71:79:5d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:f6:13:71:95:e1:2e Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:be:76:10:bb:a1:c3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164349440 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.384797 5091 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.384862 5091 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.385185 5091 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.385206 5091 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-214.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.385334 5091 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.385341 5091 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.385354 5091 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.385368 5091 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.385387 5091 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.385526 5091 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.385644 5091 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.385680 5091 kubelet.go:493] "Attempting to sync node with API server" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.385691 5091 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.385705 5091 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.385714 5091 kubelet.go:405] "Adding apiserver pod source" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.385723 5091 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.386472 5091 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.386481 5091 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.386977 5091 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390307 5091 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390316 5091 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390410 5091 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390582 5091 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390612 5091 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390717 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390734 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390749 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390761 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390772 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390777 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390782 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390788 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390800 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390806 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390820 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390828 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 19:31:20.390843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390856 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 19:31:20.392283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.390864 5091 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 19:31:20.394476 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.394459 5091 server.go:1263] "Started kubelet" Oct 30 19:31:20.394570 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.394508 5091 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 19:31:20.395003 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.394948 5091 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 19:31:20.395084 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.395023 5091 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 19:31:20.395184 ip-10-0-132-214 systemd[1]: Started Kubernetes Kubelet. Oct 30 19:31:20.403767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.396322 5091 server.go:317] "Adding debug handlers to kubelet server" Oct 30 19:31:20.403767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.396706 5091 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 19:31:20.406632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.405205 5091 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 19:31:20.406632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.405346 5091 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 19:31:20.406632 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:20.405755 5091 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-214.us-east-2.compute.internal\" not found" Oct 30 19:31:20.406632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.405810 5091 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 19:31:20.406632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.405819 5091 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 19:31:20.406632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.406001 5091 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 19:25:41 +0000 UTC" deadline="2028-06-15 12:14:27.044541508 +0000 UTC" Oct 30 19:31:20.406632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.406023 5091 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14248h43m6.638521997s" Oct 30 19:31:20.406632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.406039 5091 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 19:31:20.408835 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408805 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24ac5496-c680-4b2a-8362-f12b662df997" volumeName="kubernetes.io/empty-dir/24ac5496-c680-4b2a-8362-f12b662df997-snapshots" seLinuxMountContext="" Oct 30 19:31:20.408835 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408833 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35066f63-29d2-42da-83b7-edf8894d8e5c" volumeName="kubernetes.io/projected/35066f63-29d2-42da-83b7-edf8894d8e5c-kube-api-access-24nrk" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408842 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6693a492-433f-4614-a8e3-71ee75c177c2" volumeName="kubernetes.io/projected/6693a492-433f-4614-a8e3-71ee75c177c2-kube-api-access-jhzl5" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408849 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d903311-c71c-4197-8ac3-486702bd46e9" volumeName="kubernetes.io/configmap/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-config" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408857 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24ac5496-c680-4b2a-8362-f12b662df997" volumeName="kubernetes.io/secret/24ac5496-c680-4b2a-8362-f12b662df997-serving-cert" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408864 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="731f78f9-9938-4170-a8ff-10ba7a863759" volumeName="kubernetes.io/empty-dir/731f78f9-9938-4170-a8ff-10ba7a863759-ca-trust-extracted" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408871 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="731f78f9-9938-4170-a8ff-10ba7a863759" volumeName="kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-installation-pull-secrets" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408878 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fa163f8-129c-4f64-ad4b-ad3cc08bed13" volumeName="kubernetes.io/empty-dir/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-tmp-dir" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408886 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7fa163f8-129c-4f64-ad4b-ad3cc08bed13" volumeName="kubernetes.io/projected/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-kube-api-access-5t8r2" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408892 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e4295e8-368d-4799-b1c4-70355610378c" volumeName="kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408900 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="534aacd9-7102-4a50-8696-c50362ce87e4" volumeName="kubernetes.io/secret/534aacd9-7102-4a50-8696-c50362ce87e4-cert" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408908 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01188e4b-c110-477a-ba88-d9f48741c0a2" volumeName="kubernetes.io/projected/01188e4b-c110-477a-ba88-d9f48741c0a2-kube-api-access-5ghq4" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408914 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2f2ee082-2070-4390-baf3-55fa09089049" volumeName="kubernetes.io/projected/2f2ee082-2070-4390-baf3-55fa09089049-kube-api-access-6vkzm" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408932 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6e429ff-3fb2-48da-a77b-7806b90139e5" volumeName="kubernetes.io/empty-dir/b6e429ff-3fb2-48da-a77b-7806b90139e5-data-volume" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408946 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d95b4719-782e-47f3-9a7d-ea04268f82da" volumeName="kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-env-overrides" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408953 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02832de1-9584-4e12-9b94-8b6b52a71941" volumeName="kubernetes.io/configmap/02832de1-9584-4e12-9b94-8b6b52a71941-telemetry-config" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408974 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bfb73f4-9290-4440-938a-feb9fafa3287" volumeName="kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408983 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c260520-55b0-44ca-8ccb-e15cab54fa90" volumeName="kubernetes.io/configmap/6c260520-55b0-44ca-8ccb-e15cab54fa90-config" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408990 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d903311-c71c-4197-8ac3-486702bd46e9" volumeName="kubernetes.io/projected/7d903311-c71c-4197-8ac3-486702bd46e9-kube-api-access-qbtrc" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.408998 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b6e429ff-3fb2-48da-a77b-7806b90139e5" volumeName="kubernetes.io/projected/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-api-access-wzv8w" seLinuxMountContext="" Oct 30 19:31:20.408995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409005 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc37d377-9cb8-4f67-aa5d-1368a886533c" volumeName="kubernetes.io/configmap/cc37d377-9cb8-4f67-aa5d-1368a886533c-iptables-alerter-script" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409012 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc37d377-9cb8-4f67-aa5d-1368a886533c" volumeName="kubernetes.io/projected/cc37d377-9cb8-4f67-aa5d-1368a886533c-kube-api-access-5fcwx" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409020 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22599c63-1821-4c98-9096-9d4c43549ab7" volumeName="kubernetes.io/projected/22599c63-1821-4c98-9096-9d4c43549ab7-kube-api-access-kk5qr" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409029 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="731f78f9-9938-4170-a8ff-10ba7a863759" volumeName="kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-image-registry-private-configuration" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409040 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89bcca43-c158-47fb-9cb9-1442abc2cd45" volumeName="kubernetes.io/projected/89bcca43-c158-47fb-9cb9-1442abc2cd45-kube-api-access-g5c84" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409047 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="067a4436-690d-4b2c-aaf2-1d84830b7287" volumeName="kubernetes.io/projected/067a4436-690d-4b2c-aaf2-1d84830b7287-kube-api-access-5cvnm" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409055 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89bcca43-c158-47fb-9cb9-1442abc2cd45" volumeName="kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409062 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2cace4b-3a50-4fe5-8ead-c9765b9540d9" volumeName="kubernetes.io/projected/a2cace4b-3a50-4fe5-8ead-c9765b9540d9-kube-api-access-t9v4h" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409073 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24ac5496-c680-4b2a-8362-f12b662df997" volumeName="kubernetes.io/configmap/24ac5496-c680-4b2a-8362-f12b662df997-service-ca-bundle" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409090 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb5d13ef-8197-4960-b142-d8a1bdd025e2" volumeName="kubernetes.io/configmap/fb5d13ef-8197-4960-b142-d8a1bdd025e2-serviceca" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409103 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01188e4b-c110-477a-ba88-d9f48741c0a2" volumeName="kubernetes.io/empty-dir/01188e4b-c110-477a-ba88-d9f48741c0a2-tmp-dir" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409114 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e13582a-7d64-443a-a62f-c14ecc655f18" volumeName="kubernetes.io/configmap/6e13582a-7d64-443a-a62f-c14ecc655f18-nginx-conf" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409126 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f0f5eb3-92c0-4822-867c-cf802c9e8490" volumeName="kubernetes.io/secret/6f0f5eb3-92c0-4822-867c-cf802c9e8490-agent-certs" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409136 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8161b0bd-5184-44fc-864e-869dc7737a55" volumeName="kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-registry-certificates" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409162 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d95b4719-782e-47f3-9a7d-ea04268f82da" volumeName="kubernetes.io/projected/d95b4719-782e-47f3-9a7d-ea04268f82da-kube-api-access-vglqr" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409174 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24ac5496-c680-4b2a-8362-f12b662df997" volumeName="kubernetes.io/configmap/24ac5496-c680-4b2a-8362-f12b662df997-trusted-ca-bundle" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409187 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41c1116a-2830-4ea1-aae5-3a052f073293" volumeName="kubernetes.io/configmap/41c1116a-2830-4ea1-aae5-3a052f073293-config" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409200 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41c1116a-2830-4ea1-aae5-3a052f073293" volumeName="kubernetes.io/secret/41c1116a-2830-4ea1-aae5-3a052f073293-serving-cert" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409214 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="af2922a1-2b0f-4ee1-8ef3-4911d029e93d" volumeName="kubernetes.io/projected/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-kube-api-access-hwf6q" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409226 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f031b635-0cf3-4670-a9c6-278f14922b37" volumeName="kubernetes.io/projected/f031b635-0cf3-4670-a9c6-278f14922b37-kube-api-access-glnjk" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409238 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22599c63-1821-4c98-9096-9d4c43549ab7" volumeName="kubernetes.io/configmap/22599c63-1821-4c98-9096-9d4c43549ab7-signing-cabundle" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409249 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c260520-55b0-44ca-8ccb-e15cab54fa90" volumeName="kubernetes.io/secret/6c260520-55b0-44ca-8ccb-e15cab54fa90-serving-cert" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409262 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8161b0bd-5184-44fc-864e-869dc7737a55" volumeName="kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-image-registry-private-configuration" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409275 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9458eb02-2921-4f34-9578-60056e3505a4" volumeName="kubernetes.io/projected/9458eb02-2921-4f34-9578-60056e3505a4-kube-api-access-cnpd5" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409291 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad" volumeName="kubernetes.io/projected/b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad-kube-api-access-bzlcx" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409315 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8161b0bd-5184-44fc-864e-869dc7737a55" volumeName="kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-installation-pull-secrets" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409327 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="067a4436-690d-4b2c-aaf2-1d84830b7287" volumeName="kubernetes.io/configmap/067a4436-690d-4b2c-aaf2-1d84830b7287-trusted-ca" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409339 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24ac5496-c680-4b2a-8362-f12b662df997" volumeName="kubernetes.io/empty-dir/24ac5496-c680-4b2a-8362-f12b662df997-tmp" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409351 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="24ac5496-c680-4b2a-8362-f12b662df997" volumeName="kubernetes.io/projected/24ac5496-c680-4b2a-8362-f12b662df997-kube-api-access-qvx5x" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409363 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="41c1116a-2830-4ea1-aae5-3a052f073293" volumeName="kubernetes.io/projected/41c1116a-2830-4ea1-aae5-3a052f073293-kube-api-access-rh66m" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409375 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="731f78f9-9938-4170-a8ff-10ba7a863759" volumeName="kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-trusted-ca" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409387 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e4295e8-368d-4799-b1c4-70355610378c" volumeName="kubernetes.io/empty-dir/9e4295e8-368d-4799-b1c4-70355610378c-archives-path" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409401 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d95b4719-782e-47f3-9a7d-ea04268f82da" volumeName="kubernetes.io/secret/d95b4719-782e-47f3-9a7d-ea04268f82da-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409413 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="01188e4b-c110-477a-ba88-d9f48741c0a2" volumeName="kubernetes.io/configmap/01188e4b-c110-477a-ba88-d9f48741c0a2-config-volume" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409421 5091 factory.go:55] Registering systemd factory Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409441 5091 factory.go:223] Registration of the systemd container factory successfully Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409426 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="067a4436-690d-4b2c-aaf2-1d84830b7287" volumeName="kubernetes.io/secret/067a4436-690d-4b2c-aaf2-1d84830b7287-serving-cert" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409502 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35066f63-29d2-42da-83b7-edf8894d8e5c" volumeName="kubernetes.io/configmap/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-daemon-config" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409544 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8161b0bd-5184-44fc-864e-869dc7737a55" volumeName="kubernetes.io/empty-dir/8161b0bd-5184-44fc-864e-869dc7737a55-ca-trust-extracted" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409570 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ad103ddc-523c-4881-8526-00458255bb6c" volumeName="kubernetes.io/projected/ad103ddc-523c-4881-8526-00458255bb6c-kube-api-access-qfqd8" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409584 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb5d13ef-8197-4960-b142-d8a1bdd025e2" volumeName="kubernetes.io/projected/fb5d13ef-8197-4960-b142-d8a1bdd025e2-kube-api-access-7fnzv" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409596 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="22599c63-1821-4c98-9096-9d4c43549ab7" volumeName="kubernetes.io/secret/22599c63-1821-4c98-9096-9d4c43549ab7-signing-key" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409621 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35066f63-29d2-42da-83b7-edf8894d8e5c" volumeName="kubernetes.io/configmap/35066f63-29d2-42da-83b7-edf8894d8e5c-cni-binary-copy" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409634 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="731f78f9-9938-4170-a8ff-10ba7a863759" volumeName="kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-registry-certificates" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409646 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89bcca43-c158-47fb-9cb9-1442abc2cd45" volumeName="kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-cni-binary-copy" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409658 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d7792ad-14ae-482b-9def-eda8e029dcda" volumeName="kubernetes.io/projected/8d7792ad-14ae-482b-9def-eda8e029dcda-kube-api-access-52n6r" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409670 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8e91049-4a10-4472-922d-7b7875e7de3a" volumeName="kubernetes.io/projected/a8e91049-4a10-4472-922d-7b7875e7de3a-kube-api-access-rrblb" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409682 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d95b4719-782e-47f3-9a7d-ea04268f82da" volumeName="kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-ovnkube-script-lib" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409693 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c633f50-fd32-4440-aefc-79e44203424e" volumeName="kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409705 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f0f5eb3-92c0-4822-867c-cf802c9e8490" volumeName="kubernetes.io/configmap/6f0f5eb3-92c0-4822-867c-cf802c9e8490-konnectivity-ca" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409717 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8161b0bd-5184-44fc-864e-869dc7737a55" volumeName="kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-kube-api-access-jvxqn" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409728 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8161b0bd-5184-44fc-864e-869dc7737a55" volumeName="kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-bound-sa-token" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409740 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d7792ad-14ae-482b-9def-eda8e029dcda" volumeName="kubernetes.io/empty-dir/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-tuned" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409753 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d7792ad-14ae-482b-9def-eda8e029dcda" volumeName="kubernetes.io/empty-dir/8d7792ad-14ae-482b-9def-eda8e029dcda-tmp" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409772 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e4295e8-368d-4799-b1c4-70355610378c" volumeName="kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-service-ca-bundle" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409789 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8e91049-4a10-4472-922d-7b7875e7de3a" volumeName="kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-default-certificate" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409802 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02832de1-9584-4e12-9b94-8b6b52a71941" volumeName="kubernetes.io/projected/02832de1-9584-4e12-9b94-8b6b52a71941-kube-api-access-2dqtb" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409812 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="06bbc899-e828-4a45-bd9a-3e703c4416e7" volumeName="kubernetes.io/projected/06bbc899-e828-4a45-bd9a-3e703c4416e7-kube-api-access-847qk" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409830 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="731f78f9-9938-4170-a8ff-10ba7a863759" volumeName="kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-kube-api-access-rhvp8" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409841 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="731f78f9-9938-4170-a8ff-10ba7a863759" volumeName="kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-bound-sa-token" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409853 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8161b0bd-5184-44fc-864e-869dc7737a55" volumeName="kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-trusted-ca" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409865 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d95b4719-782e-47f3-9a7d-ea04268f82da" volumeName="kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-ovnkube-config" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409879 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="067a4436-690d-4b2c-aaf2-1d84830b7287" volumeName="kubernetes.io/configmap/067a4436-690d-4b2c-aaf2-1d84830b7287-config" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409892 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="534aacd9-7102-4a50-8696-c50362ce87e4" volumeName="kubernetes.io/projected/534aacd9-7102-4a50-8696-c50362ce87e4-kube-api-access-ql7gq" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409911 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6c260520-55b0-44ca-8ccb-e15cab54fa90" volumeName="kubernetes.io/projected/6c260520-55b0-44ca-8ccb-e15cab54fa90-kube-api-access-dtmz6" seLinuxMountContext="" Oct 30 19:31:20.409874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409923 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89bcca43-c158-47fb-9cb9-1442abc2cd45" volumeName="kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 19:31:20.415022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409936 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9e4295e8-368d-4799-b1c4-70355610378c" volumeName="kubernetes.io/projected/9e4295e8-368d-4799-b1c4-70355610378c-kube-api-access-phptb" seLinuxMountContext="" Oct 30 19:31:20.415022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409948 5091 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a8e91049-4a10-4472-922d-7b7875e7de3a" volumeName="kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-stats-auth" seLinuxMountContext="" Oct 30 19:31:20.415022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409960 5091 reconstruct.go:97] "Volume reconstruction finished" Oct 30 19:31:20.415022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.409966 5091 reconciler.go:29] "Reconciler: start to sync state" Oct 30 19:31:20.415022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.410470 5091 factory.go:153] Registering CRI-O factory Oct 30 19:31:20.415022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.410489 5091 factory.go:223] Registration of the crio container factory successfully Oct 30 19:31:20.415022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.410540 5091 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 19:31:20.415022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.410561 5091 factory.go:103] Registering Raw factory Oct 30 19:31:20.415022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.410575 5091 manager.go:1199] Started watching for new ooms in manager Oct 30 19:31:20.415022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.410887 5091 manager.go:319] Starting recovery of all containers Oct 30 19:31:20.416905 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.416886 5091 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 19:31:20.419285 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.419106 5091 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:31:20.419285 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.419239 5091 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:31:20.423837 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.423816 5091 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:31:20.425265 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.425245 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:31:20.427532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.427514 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:31:20.431331 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.431293 5091 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 19:31:20.432873 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.432853 5091 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 19:31:20.432873 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.432868 5091 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 19:31:20.433036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.432888 5091 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 19:31:20.433036 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:20.432924 5091 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 19:31:20.439326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.438853 5091 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 19:31:20.443085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.443051 5091 generic.go:356] "Generic (PLEG): container finished" podID="89bcca43-c158-47fb-9cb9-1442abc2cd45" containerID="e03d7e4cb0518deec2e709c10e25ee41f13b22fc2ff47e735e9f1e87448b2e9e" exitCode=0 Oct 30 19:31:20.443085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.443082 5091 generic.go:356] "Generic (PLEG): container finished" podID="89bcca43-c158-47fb-9cb9-1442abc2cd45" containerID="060c949aa60750f9348b948945a565e1a28e12014c2aa067ddfbee714d255af4" exitCode=0 Oct 30 19:31:20.443247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.443093 5091 generic.go:356] "Generic (PLEG): container finished" podID="89bcca43-c158-47fb-9cb9-1442abc2cd45" containerID="5cf01851dbd4be000a8b097af076fe8224184a348b076ccad6286b5adffabbc6" exitCode=0 Oct 30 19:31:20.443247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.443104 5091 generic.go:356] "Generic (PLEG): container finished" podID="89bcca43-c158-47fb-9cb9-1442abc2cd45" containerID="c2dddcc3a8880575c112705ba19f68d6ed3363c7bc7eac35757ee21400b86a1c" exitCode=0 Oct 30 19:31:20.443247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.443110 5091 generic.go:356] "Generic (PLEG): container finished" podID="89bcca43-c158-47fb-9cb9-1442abc2cd45" containerID="811ce339461a265908c5efa81df75e9c7129d84caecb6d3c0cb0a641b5d90499" exitCode=0 Oct 30 19:31:20.443247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.443115 5091 generic.go:356] "Generic (PLEG): container finished" podID="89bcca43-c158-47fb-9cb9-1442abc2cd45" containerID="4f796438ba42579b9fe3e1124e9ad3be8a18c82865803b7959d12f37547e0a3b" exitCode=0 Oct 30 19:31:20.446709 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.446685 5091 generic.go:356] "Generic (PLEG): container finished" podID="f5f4b9c4d5c0325968d9812f1693d009" containerID="4e36fa05873ca0e43e3644e018d6334ff5bf46c0c538d13038328de655aabf8a" exitCode=0 Oct 30 19:31:20.454171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.453080 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:31:20.454171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.453566 5091 generic.go:356] "Generic (PLEG): container finished" podID="067a4436-690d-4b2c-aaf2-1d84830b7287" containerID="d2b9813548f63ce8ff5ee1d96c7d773bedbaa3663d02970290b3b3f268e977b6" exitCode=255 Oct 30 19:31:20.504608 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.504589 5091 manager.go:324] Recovery completed Oct 30 19:31:20.525645 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.525613 5091 cpu_manager.go:226] "Starting" policy="none" Oct 30 19:31:20.525645 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.525639 5091 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 19:31:20.525854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.525656 5091 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 19:31:20.525854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.525767 5091 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 19:31:20.526259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.525777 5091 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 19:31:20.526298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.526260 5091 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 19:31:20.526298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.526267 5091 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 19:31:20.526298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.526273 5091 policy_none.go:50] "Start" Oct 30 19:31:20.526298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.526282 5091 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 19:31:20.526298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.526296 5091 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 19:31:20.526439 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.526429 5091 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 19:31:20.526472 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.526439 5091 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 19:31:20.526472 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.526445 5091 policy_none.go:44] "Start" Oct 30 19:31:20.530919 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.530892 5091 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 19:31:20.533510 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:20.533490 5091 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 19:31:20.533616 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.533601 5091 manager.go:341] "Starting Device Plugin manager" Oct 30 19:31:20.533670 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:20.533630 5091 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 19:31:20.533670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.533643 5091 server.go:87] "Starting device plugin registration server" Oct 30 19:31:20.533937 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.533922 5091 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 19:31:20.534014 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.533940 5091 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 19:31:20.534065 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.534013 5091 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 19:31:20.534179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.534123 5091 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 19:31:20.534233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.534186 5091 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 19:31:20.534233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.534194 5091 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 19:31:20.534320 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.534247 5091 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T19:31:20.534176121Z","UUID":"d869618e-e187-46c9-8655-632b850970b2","Handler":null,"Name":"","Endpoint":""} Oct 30 19:31:20.535770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.535754 5091 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 19:31:20.535865 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.535815 5091 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 19:31:20.535865 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:20.535840 5091 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 19:31:20.634503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.634474 5091 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 19:31:20.635835 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.635819 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 19:31:20.635922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.635852 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 19:31:20.635922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.635864 5091 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-214.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 19:31:20.635922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.635904 5091 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.653300 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.653280 5091 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.653405 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.653346 5091 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.734258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.734217 5091 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal"] Oct 30 19:31:20.734809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.734794 5091 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.734809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.734787 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" event={"ID":"f5f4b9c4d5c0325968d9812f1693d009","Type":"ContainerStarted","Data":"34def9a07570e749a94faad670dccb763d05de3c26a245752d0515623fcf800b"} Oct 30 19:31:20.734897 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.734828 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" event={"ID":"f5f4b9c4d5c0325968d9812f1693d009","Type":"ContainerDied","Data":"4e36fa05873ca0e43e3644e018d6334ff5bf46c0c538d13038328de655aabf8a"} Oct 30 19:31:20.734897 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.734843 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" event={"ID":"f5f4b9c4d5c0325968d9812f1693d009","Type":"ContainerStarted","Data":"07e65c955f517b432c546179e4db92833bc0e84cf063c74ee199eca9f315ef21"} Oct 30 19:31:20.734961 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.734921 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" event={"ID":"84cfec117dc1db7ebfa66a1a705b525e","Type":"ContainerStarted","Data":"c19a97b6bededb7d4a9b35efed5664bfff1864e300287ad0237fc45de3c65370"} Oct 30 19:31:20.734961 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.734935 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" event={"ID":"84cfec117dc1db7ebfa66a1a705b525e","Type":"ContainerStarted","Data":"17e9bb78e607eda1bf5e72de4044bbd3579c2fd9b4221af48e9c779d8de68156"} Oct 30 19:31:20.735036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.735012 5091 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.751274 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.751257 5091 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:31:20.751315 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:20.751293 5091 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.752266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.752251 5091 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:31:20.752311 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:20.752283 5091 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.819634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.819556 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84cfec117dc1db7ebfa66a1a705b525e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"84cfec117dc1db7ebfa66a1a705b525e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.819634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.819589 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f5f4b9c4d5c0325968d9812f1693d009-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"f5f4b9c4d5c0325968d9812f1693d009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.819634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.819606 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5f4b9c4d5c0325968d9812f1693d009-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"f5f4b9c4d5c0325968d9812f1693d009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.920230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.920131 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84cfec117dc1db7ebfa66a1a705b525e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"84cfec117dc1db7ebfa66a1a705b525e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.920369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.920255 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f5f4b9c4d5c0325968d9812f1693d009-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"f5f4b9c4d5c0325968d9812f1693d009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.920369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.920274 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5f4b9c4d5c0325968d9812f1693d009-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"f5f4b9c4d5c0325968d9812f1693d009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.920369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.920199 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/84cfec117dc1db7ebfa66a1a705b525e-config\") pod \"kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"84cfec117dc1db7ebfa66a1a705b525e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.920369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.920324 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f5f4b9c4d5c0325968d9812f1693d009-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"f5f4b9c4d5c0325968d9812f1693d009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:20.920369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:20.920339 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f5f4b9c4d5c0325968d9812f1693d009-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal\" (UID: \"f5f4b9c4d5c0325968d9812f1693d009\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:21.386378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.386339 5091 apiserver.go:52] "Watching apiserver" Oct 30 19:31:21.410800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.410771 5091 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 19:31:21.411477 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.411451 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-kzbbk","kube-system/kas-connection-checker-685895c7c9-fkjv2","openshift-insights/insights-operator-7c4986d449-7jhj7","kube-system/global-pull-secret-syncer-mm6ld","openshift-console-operator/console-operator-7cd56b8dbc-4kpqr","openshift-insights/periodic-gathering-rgfw9-smvbs","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg","openshift-network-operator/iptables-alerter-pcsxp","openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal","openshift-network-console/networking-console-plugin-86c478bb88-twdf7","openshift-dns/node-resolver-9g689","openshift-image-registry/node-ca-m77bc","openshift-ingress-canary/ingress-canary-77ms2","kube-system/kas-connection-checker-685895c7c9-rtf4t","kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vblv9","openshift-multus/multus-zzxkl","openshift-kube-storage-version-migrator/migrator-66df5d496b-vqzdh","openshift-dns/dns-default-j2h9z","openshift-network-diagnostics/network-check-target-p98lt","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp","openshift-kube-storage-version-migrator/migrator-66df5d496b-h6ssv","kube-system/konnectivity-agent-9hgsn","openshift-image-registry/image-registry-7f6d8b54bc-kmdpv","openshift-multus/multus-additional-cni-plugins-kvwqx","openshift-cluster-node-tuning-operator/tuned-mnf6c","openshift-ovn-kubernetes/ovnkube-node-fg2zw","kube-system/kas-connection-checker-685895c7c9-pggqx","openshift-ingress/router-default-787bddbb47-hfcvw","openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl","openshift-service-ca/service-ca-5fffc97bdf-nfhdn","openshift-multus/network-metrics-daemon-t4fw7","openshift-image-registry/image-registry-549665644b-9xg4z","openshift-insights/insights-runtime-extractor-fndt2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d"] Oct 30 19:31:21.412826 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.412805 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:21.413544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.413524 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.414118 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.414101 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:21.417050 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.417021 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.417159 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.417036 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.418821 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.418799 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:21.418972 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.418951 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.418972 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.418965 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Oct 30 19:31:21.419196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.418968 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Oct 30 19:31:21.419196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.418999 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Oct 30 19:31:21.419196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.419036 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Oct 30 19:31:21.419196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.419064 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.419196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.419103 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Oct 30 19:31:21.419412 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.419284 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:21.419412 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.419309 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.420473 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.420447 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Oct 30 19:31:21.420615 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.420552 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-t79gm\"" Oct 30 19:31:21.420758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.420740 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:31:21.421345 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.421317 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Oct 30 19:31:21.421489 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.421475 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-2q7gs\"" Oct 30 19:31:21.423672 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.423653 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-dp4dn\"" Oct 30 19:31:21.423864 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.423848 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.426224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.426207 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.426790 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.426774 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:21.428169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.428125 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:21.429076 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.429034 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 19:31:21.429076 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.429035 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Oct 30 19:31:21.429239 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.429082 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Oct 30 19:31:21.429239 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.429032 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.429239 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.429055 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Oct 30 19:31:21.429239 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.429113 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.429239 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.429050 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.429239 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.429058 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Oct 30 19:31:21.431311 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.431237 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.431892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.431869 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.431980 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.431942 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Oct 30 19:31:21.432060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.432046 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.432117 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.432098 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.432249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.432233 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.432863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.432842 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.432965 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.432948 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Oct 30 19:31:21.433404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.433385 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-jflbh\"" Oct 30 19:31:21.433519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.433496 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-n522r\"" Oct 30 19:31:21.433629 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.433406 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Oct 30 19:31:21.433850 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.433832 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Oct 30 19:31:21.434211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.434190 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-9h8tn\"" Oct 30 19:31:21.434287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.434263 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Oct 30 19:31:21.434369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.434352 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.434521 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.434504 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.434614 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.434597 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.434742 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.434728 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.434832 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.434815 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-64274\"" Oct 30 19:31:21.434970 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.434953 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-5j45v\"" Oct 30 19:31:21.435061 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.435048 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Oct 30 19:31:21.435115 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.435060 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.439198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.439140 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Oct 30 19:31:21.439198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.439186 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.439336 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.439200 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.439336 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.439276 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Oct 30 19:31:21.439427 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.439352 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.439427 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.439384 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.441664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.441643 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.441664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.441645 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 19:31:21.441811 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.441708 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.441811 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.441775 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 19:31:21.441911 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.441834 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.441911 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.441836 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.441911 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.441858 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.441911 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.441889 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.443791 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.443775 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 19:31:21.444304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.444261 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Oct 30 19:31:21.444304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.444285 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-52tnm\"" Oct 30 19:31:21.444466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.444316 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Oct 30 19:31:21.444466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.444346 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-dvfnr\"" Oct 30 19:31:21.444466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.444348 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-q99f8\"" Oct 30 19:31:21.444466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.444386 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-5r9wt\"" Oct 30 19:31:21.444466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.444388 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ztltl\"" Oct 30 19:31:21.444466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.444395 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 19:31:21.446094 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.446076 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vcqml\"" Oct 30 19:31:21.446094 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.446080 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wr97x\"" Oct 30 19:31:21.446449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.446435 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7c6bv\"" Oct 30 19:31:21.446741 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.446723 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-jznq8\"" Oct 30 19:31:21.446812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.446757 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 19:31:21.446812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.446774 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.446812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.446776 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 19:31:21.446812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.446763 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-79gww\"" Oct 30 19:31:21.446812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.446795 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 19:31:21.446812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.446775 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.447037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.446824 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 19:31:21.447037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.446845 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.447037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.446859 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-g4c9f\"" Oct 30 19:31:21.447037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.446904 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 19:31:21.447037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.446912 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-x9qfs\"" Oct 30 19:31:21.447037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.447005 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 19:31:21.447037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.447012 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 19:31:21.447292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.447047 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 19:31:21.447292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.447052 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.447292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.447057 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 19:31:21.447292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.447059 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 19:31:21.447292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.447201 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rpkw7\"" Oct 30 19:31:21.449372 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.449354 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 19:31:21.449449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.449384 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.449449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.449384 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.449449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.449412 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 19:31:21.449449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.449428 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.449593 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.449466 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 19:31:21.449593 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.449482 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 19:31:21.449659 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.449623 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4mthb\"" Oct 30 19:31:21.449738 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.449721 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-7mr7r\"" Oct 30 19:31:21.449822 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.449801 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-h46gl\"" Oct 30 19:31:21.452396 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.452379 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-f2mgm\"" Oct 30 19:31:21.452474 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.452405 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.452474 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.452422 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Oct 30 19:31:21.452474 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.452406 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-scxl6\"" Oct 30 19:31:21.452474 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.452450 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-w9bpk\"" Oct 30 19:31:21.452684 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.452497 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Oct 30 19:31:21.452684 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.452504 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Oct 30 19:31:21.452684 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.452540 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-7c7hk\"" Oct 30 19:31:21.452684 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.452543 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Oct 30 19:31:21.452684 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.452587 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Oct 30 19:31:21.455058 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.455038 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 19:31:21.455594 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.455560 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Oct 30 19:31:21.455808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.455789 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Oct 30 19:31:21.491486 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.491466 5091 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:21.491597 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.491581 5091 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:21.505705 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.505668 5091 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:31:21.505857 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.505721 5091 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:21.508784 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.508767 5091 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 19:31:21.509354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.509337 5091 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 19:31:21.509438 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.509383 5091 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-214.us-east-2.compute.internal" Oct 30 19:31:21.523743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.523719 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/731f78f9-9938-4170-a8ff-10ba7a863759-ca-trust-extracted\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.523838 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.523749 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-trusted-ca\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.523838 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.523797 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-cnibin\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.523838 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.523822 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysctl-d\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.523968 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.523839 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvp8\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-kube-api-access-rhvp8\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.523968 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.523858 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6e13582a-7d64-443a-a62f-c14ecc655f18-nginx-conf\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:21.523968 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.523886 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52n6r\" (UniqueName: \"kubernetes.io/projected/8d7792ad-14ae-482b-9def-eda8e029dcda-kube-api-access-52n6r\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.523968 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.523891 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/731f78f9-9938-4170-a8ff-10ba7a863759-ca-trust-extracted\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.524229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524024 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35066f63-29d2-42da-83b7-edf8894d8e5c-cni-binary-copy\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.524229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524059 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwf6q\" (UniqueName: \"kubernetes.io/projected/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-kube-api-access-hwf6q\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:31:21.524229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524094 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6e13582a-7d64-443a-a62f-c14ecc655f18-nginx-conf\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:21.524229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524096 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phptb\" (UniqueName: \"kubernetes.io/projected/9e4295e8-368d-4799-b1c4-70355610378c-kube-api-access-phptb\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:21.524229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524131 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-var-lib-kubelet\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.524229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524176 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-trusted-ca\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.524229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524181 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:21.524229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524207 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.524670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524243 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.524670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524283 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-socket-dir-parent\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.524670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524313 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb5d13ef-8197-4960-b142-d8a1bdd025e2-host\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:31:21.524670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524336 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:21.524670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524373 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d7792ad-14ae-482b-9def-eda8e029dcda-tmp\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.524670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524419 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c260520-55b0-44ca-8ccb-e15cab54fa90-serving-cert\") pod \"service-ca-operator-544d749c8c-dqnkb\" (UID: \"6c260520-55b0-44ca-8ccb-e15cab54fa90\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:21.524670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524444 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5bfb73f4-9290-4440-938a-feb9fafa3287-original-pull-secret\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:21.524670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524463 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35066f63-29d2-42da-83b7-edf8894d8e5c-cni-binary-copy\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.524670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524488 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-cni-bin\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.524670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524507 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8d7792ad-14ae-482b-9def-eda8e029dcda-tmp\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.524670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524510 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.524670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524547 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fcwx\" (UniqueName: \"kubernetes.io/projected/cc37d377-9cb8-4f67-aa5d-1368a886533c-kube-api-access-5fcwx\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:31:21.524670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524596 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-socket-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.524670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524628 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-modprobe-d\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.524670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524668 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-slash\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524694 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-var-lib-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524722 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524765 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-run-ovn-kubernetes\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524797 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnpd5\" (UniqueName: \"kubernetes.io/projected/9458eb02-2921-4f34-9578-60056e3505a4-kube-api-access-cnpd5\") pod \"kas-connection-checker-685895c7c9-pggqx\" (UID: \"9458eb02-2921-4f34-9578-60056e3505a4\") " pod="kube-system/kas-connection-checker-685895c7c9-pggqx" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524806 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6c260520-55b0-44ca-8ccb-e15cab54fa90-serving-cert\") pod \"service-ca-operator-544d749c8c-dqnkb\" (UID: \"6c260520-55b0-44ca-8ccb-e15cab54fa90\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524835 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/24ac5496-c680-4b2a-8362-f12b662df997-snapshots\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524854 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01188e4b-c110-477a-ba88-d9f48741c0a2-tmp-dir\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524954 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.524983 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-systemd\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.525001 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-systemd-units\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.525049 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01188e4b-c110-477a-ba88-d9f48741c0a2-tmp-dir\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.525052 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.525090 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/24ac5496-c680-4b2a-8362-f12b662df997-snapshots\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.525090 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ac5496-c680-4b2a-8362-f12b662df997-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.525206 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbtrc\" (UniqueName: \"kubernetes.io/projected/7d903311-c71c-4197-8ac3-486702bd46e9-kube-api-access-qbtrc\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:21.525314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.525260 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-host\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.525889 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.525390 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-kubelet\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.525889 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.525454 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24nrk\" (UniqueName: \"kubernetes.io/projected/35066f63-29d2-42da-83b7-edf8894d8e5c-kube-api-access-24nrk\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.525889 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.525506 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ghq4\" (UniqueName: \"kubernetes.io/projected/01188e4b-c110-477a-ba88-d9f48741c0a2-kube-api-access-5ghq4\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:21.525889 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.525568 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:21.525889 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.525667 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f0f5eb3-92c0-4822-867c-cf802c9e8490-konnectivity-ca\") pod \"konnectivity-agent-9hgsn\" (UID: \"6f0f5eb3-92c0-4822-867c-cf802c9e8490\") " pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:31:21.525889 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.525703 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysctl-conf\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.525889 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.525735 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-log-socket\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.525889 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.525782 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d95b4719-782e-47f3-9a7d-ea04268f82da-ovn-node-metrics-cert\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.526194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.526097 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ac5496-c680-4b2a-8362-f12b662df997-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:21.526194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.526123 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/6f0f5eb3-92c0-4822-867c-cf802c9e8490-konnectivity-ca\") pod \"konnectivity-agent-9hgsn\" (UID: \"6f0f5eb3-92c0-4822-867c-cf802c9e8490\") " pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:31:21.526290 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.526200 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vglqr\" (UniqueName: \"kubernetes.io/projected/d95b4719-782e-47f3-9a7d-ea04268f82da-kube-api-access-vglqr\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.526365 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.526345 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-cni-bin\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.526449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.526406 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d95b4719-782e-47f3-9a7d-ea04268f82da-ovn-node-metrics-cert\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.526504 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.526485 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:21.526552 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.526539 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-bound-sa-token\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.526646 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.526624 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvxqn\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-kube-api-access-jvxqn\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.526710 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.526693 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-tmp-dir\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:31:21.526816 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.526803 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-tmp-dir\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:31:21.526867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.526833 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:21.526916 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.526877 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc37d377-9cb8-4f67-aa5d-1368a886533c-iptables-alerter-script\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:31:21.526962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.526947 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc37d377-9cb8-4f67-aa5d-1368a886533c-host-slash\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:31:21.527011 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.526995 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cvnm\" (UniqueName: \"kubernetes.io/projected/067a4436-690d-4b2c-aaf2-1d84830b7287-kube-api-access-5cvnm\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:21.527077 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527063 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb5d13ef-8197-4960-b142-d8a1bdd025e2-serviceca\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:31:21.527127 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527099 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-kubernetes\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.527195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527160 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-sys\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.527268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527253 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41c1116a-2830-4ea1-aae5-3a052f073293-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg\" (UID: \"41c1116a-2830-4ea1-aae5-3a052f073293\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:21.527324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527305 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-conf-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.527368 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527336 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-stats-auth\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:21.527411 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527381 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/067a4436-690d-4b2c-aaf2-1d84830b7287-config\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:21.527524 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527504 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9v4h\" (UniqueName: \"kubernetes.io/projected/a2cace4b-3a50-4fe5-8ead-c9765b9540d9-kube-api-access-t9v4h\") pod \"migrator-66df5d496b-vqzdh\" (UID: \"a2cace4b-3a50-4fe5-8ead-c9765b9540d9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vqzdh" Oct 30 19:31:21.527570 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527541 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/22599c63-1821-4c98-9096-9d4c43549ab7-signing-cabundle\") pod \"service-ca-5fffc97bdf-nfhdn\" (UID: \"22599c63-1821-4c98-9096-9d4c43549ab7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:21.527620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527577 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-multus-certs\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.527620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527610 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmz6\" (UniqueName: \"kubernetes.io/projected/6c260520-55b0-44ca-8ccb-e15cab54fa90-kube-api-access-dtmz6\") pod \"service-ca-operator-544d749c8c-dqnkb\" (UID: \"6c260520-55b0-44ca-8ccb-e15cab54fa90\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:21.527706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527641 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t8r2\" (UniqueName: \"kubernetes.io/projected/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-kube-api-access-5t8r2\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:31:21.527706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527665 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vkzm\" (UniqueName: \"kubernetes.io/projected/2f2ee082-2070-4390-baf3-55fa09089049-kube-api-access-6vkzm\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.527706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527698 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-netns\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.527830 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527713 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-stats-auth\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:21.527830 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527794 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/067a4436-690d-4b2c-aaf2-1d84830b7287-serving-cert\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:21.527914 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527826 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:21.527914 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527857 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-env-overrides\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.527914 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527889 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-image-registry-private-configuration\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.528041 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527914 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-device-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.528041 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527946 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-registry-certificates\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.528041 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.527980 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-etc-selinux\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.528041 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.528019 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.528234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.528071 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-cni-multus\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.528281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.528264 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/067a4436-690d-4b2c-aaf2-1d84830b7287-config\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:21.528593 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.528574 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/cc37d377-9cb8-4f67-aa5d-1368a886533c-iptables-alerter-script\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:31:21.528832 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.528813 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fb5d13ef-8197-4960-b142-d8a1bdd025e2-serviceca\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:31:21.528885 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.528837 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/067a4436-690d-4b2c-aaf2-1d84830b7287-serving-cert\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:21.528997 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.528977 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/41c1116a-2830-4ea1-aae5-3a052f073293-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg\" (UID: \"41c1116a-2830-4ea1-aae5-3a052f073293\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:21.529043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.529030 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-env-overrides\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.529285 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.529266 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-image-registry-private-configuration\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.529454 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.529428 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/22599c63-1821-4c98-9096-9d4c43549ab7-signing-cabundle\") pod \"service-ca-5fffc97bdf-nfhdn\" (UID: \"22599c63-1821-4c98-9096-9d4c43549ab7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:21.529454 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.529447 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-registry-certificates\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.529559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.529492 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/067a4436-690d-4b2c-aaf2-1d84830b7287-trusted-ca\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:21.529559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.529523 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f0f5eb3-92c0-4822-867c-cf802c9e8490-agent-certs\") pod \"konnectivity-agent-9hgsn\" (UID: \"6f0f5eb3-92c0-4822-867c-cf802c9e8490\") " pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:31:21.529644 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.529601 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/22599c63-1821-4c98-9096-9d4c43549ab7-signing-key\") pod \"service-ca-5fffc97bdf-nfhdn\" (UID: \"22599c63-1821-4c98-9096-9d4c43549ab7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:21.529644 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.529636 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-registration-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.529733 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.529664 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-cni-binary-copy\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.529733 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.529680 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-tuned\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.530003 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.529982 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/067a4436-690d-4b2c-aaf2-1d84830b7287-trusted-ca\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:21.530266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.530247 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/22599c63-1821-4c98-9096-9d4c43549ab7-signing-key\") pod \"service-ca-5fffc97bdf-nfhdn\" (UID: \"22599c63-1821-4c98-9096-9d4c43549ab7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:21.530346 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.530332 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-tuned\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.530398 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.530378 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfqd8\" (UniqueName: \"kubernetes.io/projected/ad103ddc-523c-4881-8526-00458255bb6c-kube-api-access-qfqd8\") pod \"kas-connection-checker-685895c7c9-fkjv2\" (UID: \"ad103ddc-523c-4881-8526-00458255bb6c\") " pod="kube-system/kas-connection-checker-685895c7c9-fkjv2" Oct 30 19:31:21.530452 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.530411 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhzl5\" (UniqueName: \"kubernetes.io/projected/6693a492-433f-4614-a8e3-71ee75c177c2-kube-api-access-jhzl5\") pod \"volume-data-source-validator-58c9f5f46d-vblv9\" (UID: \"6693a492-433f-4614-a8e3-71ee75c177c2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vblv9" Oct 30 19:31:21.530516 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.530502 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ac5496-c680-4b2a-8362-f12b662df997-service-ca-bundle\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:21.530564 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.530541 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-hostroot\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.530609 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.530576 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzv8w\" (UniqueName: \"kubernetes.io/projected/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-api-access-wzv8w\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:21.530845 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.530823 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/24ac5496-c680-4b2a-8362-f12b662df997-service-ca-bundle\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:21.530845 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.530830 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c260520-55b0-44ca-8ccb-e15cab54fa90-config\") pod \"service-ca-operator-544d749c8c-dqnkb\" (UID: \"6c260520-55b0-44ca-8ccb-e15cab54fa90\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:21.530951 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.530843 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/6f0f5eb3-92c0-4822-867c-cf802c9e8490-agent-certs\") pod \"konnectivity-agent-9hgsn\" (UID: \"6f0f5eb3-92c0-4822-867c-cf802c9e8490\") " pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:31:21.530951 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.530927 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-ovn\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.531037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.530977 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-ovnkube-script-lib\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.531037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.531014 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.531131 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.531074 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:21.531131 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.531072 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c260520-55b0-44ca-8ccb-e15cab54fa90-config\") pod \"service-ca-operator-544d749c8c-dqnkb\" (UID: \"6c260520-55b0-44ca-8ccb-e15cab54fa90\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:21.531257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.531131 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:21.531291 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.531263 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-ovnkube-script-lib\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.531291 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.531275 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-run-netns\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.531348 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.531302 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-cnibin\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.531348 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.531330 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvx5x\" (UniqueName: \"kubernetes.io/projected/24ac5496-c680-4b2a-8362-f12b662df997-kube-api-access-qvx5x\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:21.531484 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.531468 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-etc-kubernetes\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.531537 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.531503 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-default-certificate\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:21.531577 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.531537 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6e429ff-3fb2-48da-a77b-7806b90139e5-data-volume\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:21.531985 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.531958 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24ac5496-c680-4b2a-8362-f12b662df997-tmp\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:21.531985 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.531970 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/b6e429ff-3fb2-48da-a77b-7806b90139e5-data-volume\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:21.532100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532003 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24ac5496-c680-4b2a-8362-f12b662df997-tmp\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:21.532100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532020 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-default-certificate\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:21.532100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532044 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.532100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532066 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8161b0bd-5184-44fc-864e-869dc7737a55-ca-trust-extracted\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.532100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532098 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-sys-fs\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.532269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532163 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24ac5496-c680-4b2a-8362-f12b662df997-serving-cert\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:21.532269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532185 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8161b0bd-5184-44fc-864e-869dc7737a55-ca-trust-extracted\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.532269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532200 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-installation-pull-secrets\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.532269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532224 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrblb\" (UniqueName: \"kubernetes.io/projected/a8e91049-4a10-4472-922d-7b7875e7de3a-kube-api-access-rrblb\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:21.532269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532239 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/02832de1-9584-4e12-9b94-8b6b52a71941-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:21.532269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532253 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/534aacd9-7102-4a50-8696-c50362ce87e4-cert\") pod \"ingress-canary-77ms2\" (UID: \"534aacd9-7102-4a50-8696-c50362ce87e4\") " pod="openshift-ingress-canary/ingress-canary-77ms2" Oct 30 19:31:21.532555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532273 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-hosts-file\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:31:21.532555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532296 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-system-cni-dir\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.532555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532320 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-daemon-config\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.532555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532345 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41c1116a-2830-4ea1-aae5-3a052f073293-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg\" (UID: \"41c1116a-2830-4ea1-aae5-3a052f073293\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:21.532555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532372 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kk5qr\" (UniqueName: \"kubernetes.io/projected/22599c63-1821-4c98-9096-9d4c43549ab7-kube-api-access-kk5qr\") pod \"service-ca-5fffc97bdf-nfhdn\" (UID: \"22599c63-1821-4c98-9096-9d4c43549ab7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:21.532555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532397 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01188e4b-c110-477a-ba88-d9f48741c0a2-config-volume\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:21.532555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532431 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnjk\" (UniqueName: \"kubernetes.io/projected/f031b635-0cf3-4670-a9c6-278f14922b37-kube-api-access-glnjk\") pod \"migrator-66df5d496b-h6ssv\" (UID: \"f031b635-0cf3-4670-a9c6-278f14922b37\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h6ssv" Oct 30 19:31:21.532555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532460 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:31:21.532555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532492 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/02832de1-9584-4e12-9b94-8b6b52a71941-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:21.532555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532485 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-trusted-ca-bundle\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:21.532555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532495 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/534aacd9-7102-4a50-8696-c50362ce87e4-cert\") pod \"ingress-canary-77ms2\" (UID: \"534aacd9-7102-4a50-8696-c50362ce87e4\") " pod="openshift-ingress-canary/ingress-canary-77ms2" Oct 30 19:31:21.532555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532497 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-installation-pull-secrets\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.532990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532625 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-daemon-config\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.532990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532627 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/24ac5496-c680-4b2a-8362-f12b662df997-serving-cert\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:21.532990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532706 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-lib-modules\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.532990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532736 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-systemd\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.532990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532744 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/01188e4b-c110-477a-ba88-d9f48741c0a2-config-volume\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:21.532990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532761 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-etc-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.532990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532814 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/41c1116a-2830-4ea1-aae5-3a052f073293-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg\" (UID: \"41c1116a-2830-4ea1-aae5-3a052f073293\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:21.532990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532819 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-cni-netd\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.532990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532877 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bfb73f4-9290-4440-938a-feb9fafa3287-dbus\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:21.532990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532916 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-image-registry-private-configuration\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.532990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.532954 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzlcx\" (UniqueName: \"kubernetes.io/projected/b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad-kube-api-access-bzlcx\") pod \"network-check-source-6f78d59f99-kzbbk\" (UID: \"b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kzbbk" Oct 30 19:31:21.533369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533028 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-trusted-ca-bundle\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:21.533369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533061 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-kubelet\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.533369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533084 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9e4295e8-368d-4799-b1c4-70355610378c-archives-path\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:21.533369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533096 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-image-registry-private-configuration\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.533369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533101 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-847qk\" (UniqueName: \"kubernetes.io/projected/06bbc899-e828-4a45-bd9a-3e703c4416e7-kube-api-access-847qk\") pod \"kas-connection-checker-685895c7c9-rtf4t\" (UID: \"06bbc899-e828-4a45-bd9a-3e703c4416e7\") " pod="kube-system/kas-connection-checker-685895c7c9-rtf4t" Oct 30 19:31:21.533369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533124 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-ovnkube-config\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.533369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533160 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-cni-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.533369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533186 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9vjq\" (UniqueName: \"kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq\") pod \"network-check-target-p98lt\" (UID: \"6c633f50-fd32-4440-aefc-79e44203424e\") " pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:31:21.533369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533199 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9e4295e8-368d-4799-b1c4-70355610378c-archives-path\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:21.533369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533253 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-system-cni-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.533369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533272 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-run\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.533369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533297 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-trusted-ca\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.533369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533314 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh66m\" (UniqueName: \"kubernetes.io/projected/41c1116a-2830-4ea1-aae5-3a052f073293-kube-api-access-rh66m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg\" (UID: \"41c1116a-2830-4ea1-aae5-3a052f073293\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:21.533369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533334 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d95b4719-782e-47f3-9a7d-ea04268f82da-ovnkube-config\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.533856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533377 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bfb73f4-9290-4440-938a-feb9fafa3287-kubelet-config\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:21.533856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533449 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.533856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533500 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5c84\" (UniqueName: \"kubernetes.io/projected/89bcca43-c158-47fb-9cb9-1442abc2cd45-kube-api-access-g5c84\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.533856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533629 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-bound-sa-token\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.533856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533647 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:21.533856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533674 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fnzv\" (UniqueName: \"kubernetes.io/projected/fb5d13ef-8197-4960-b142-d8a1bdd025e2-kube-api-access-7fnzv\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:31:21.533856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533681 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-trusted-ca\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.533856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533698 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dqtb\" (UniqueName: \"kubernetes.io/projected/02832de1-9584-4e12-9b94-8b6b52a71941-kube-api-access-2dqtb\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:21.533856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533723 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql7gq\" (UniqueName: \"kubernetes.io/projected/534aacd9-7102-4a50-8696-c50362ce87e4-kube-api-access-ql7gq\") pod \"ingress-canary-77ms2\" (UID: \"534aacd9-7102-4a50-8696-c50362ce87e4\") " pod="openshift-ingress-canary/ingress-canary-77ms2" Oct 30 19:31:21.533856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533741 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6e429ff-3fb2-48da-a77b-7806b90139e5-crio-socket\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:21.533856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533759 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:21.533856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533776 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-node-log\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.533856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533827 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-registry-certificates\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.534386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533891 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-installation-pull-secrets\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.534386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533926 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.534386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533961 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-k8s-cni-cncf-io\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.534386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.533993 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-service-ca-bundle\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:21.534386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.534011 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-registry-certificates\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.534386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.534027 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysconfig\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.534386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.534060 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-installation-pull-secrets\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.534386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.534184 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-service-ca-bundle\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:21.534386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.534189 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/89bcca43-c158-47fb-9cb9-1442abc2cd45-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.540802 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.540781 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwf6q\" (UniqueName: \"kubernetes.io/projected/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-kube-api-access-hwf6q\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:31:21.541421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.541386 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fcwx\" (UniqueName: \"kubernetes.io/projected/cc37d377-9cb8-4f67-aa5d-1368a886533c-kube-api-access-5fcwx\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:31:21.541582 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.541561 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnpd5\" (UniqueName: \"kubernetes.io/projected/9458eb02-2921-4f34-9578-60056e3505a4-kube-api-access-cnpd5\") pod \"kas-connection-checker-685895c7c9-pggqx\" (UID: \"9458eb02-2921-4f34-9578-60056e3505a4\") " pod="kube-system/kas-connection-checker-685895c7c9-pggqx" Oct 30 19:31:21.541671 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.541595 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vglqr\" (UniqueName: \"kubernetes.io/projected/d95b4719-782e-47f3-9a7d-ea04268f82da-kube-api-access-vglqr\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.541671 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.541595 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52n6r\" (UniqueName: \"kubernetes.io/projected/8d7792ad-14ae-482b-9def-eda8e029dcda-kube-api-access-52n6r\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.541774 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.541747 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbtrc\" (UniqueName: \"kubernetes.io/projected/7d903311-c71c-4197-8ac3-486702bd46e9-kube-api-access-qbtrc\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:21.541827 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.541804 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phptb\" (UniqueName: \"kubernetes.io/projected/9e4295e8-368d-4799-b1c4-70355610378c-kube-api-access-phptb\") pod \"periodic-gathering-rgfw9-smvbs\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:31:21.541956 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.541939 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvxqn\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-kube-api-access-jvxqn\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.542303 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.542284 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24nrk\" (UniqueName: \"kubernetes.io/projected/35066f63-29d2-42da-83b7-edf8894d8e5c-kube-api-access-24nrk\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.543035 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.543020 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-bound-sa-token\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.543301 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.543286 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cvnm\" (UniqueName: \"kubernetes.io/projected/067a4436-690d-4b2c-aaf2-1d84830b7287-kube-api-access-5cvnm\") pod \"console-operator-7cd56b8dbc-4kpqr\" (UID: \"067a4436-690d-4b2c-aaf2-1d84830b7287\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:21.544107 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.544087 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvp8\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-kube-api-access-rhvp8\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.545589 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.545573 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9v4h\" (UniqueName: \"kubernetes.io/projected/a2cace4b-3a50-4fe5-8ead-c9765b9540d9-kube-api-access-t9v4h\") pod \"migrator-66df5d496b-vqzdh\" (UID: \"a2cace4b-3a50-4fe5-8ead-c9765b9540d9\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-vqzdh" Oct 30 19:31:21.546368 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.546350 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmz6\" (UniqueName: \"kubernetes.io/projected/6c260520-55b0-44ca-8ccb-e15cab54fa90-kube-api-access-dtmz6\") pod \"service-ca-operator-544d749c8c-dqnkb\" (UID: \"6c260520-55b0-44ca-8ccb-e15cab54fa90\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" Oct 30 19:31:21.547733 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.547717 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ghq4\" (UniqueName: \"kubernetes.io/projected/01188e4b-c110-477a-ba88-d9f48741c0a2-kube-api-access-5ghq4\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:21.549392 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.549372 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glnjk\" (UniqueName: \"kubernetes.io/projected/f031b635-0cf3-4670-a9c6-278f14922b37-kube-api-access-glnjk\") pod \"migrator-66df5d496b-h6ssv\" (UID: \"f031b635-0cf3-4670-a9c6-278f14922b37\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-h6ssv" Oct 30 19:31:21.549482 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.549407 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t8r2\" (UniqueName: \"kubernetes.io/projected/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-kube-api-access-5t8r2\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:31:21.549578 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.549559 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kk5qr\" (UniqueName: \"kubernetes.io/projected/22599c63-1821-4c98-9096-9d4c43549ab7-kube-api-access-kk5qr\") pod \"service-ca-5fffc97bdf-nfhdn\" (UID: \"22599c63-1821-4c98-9096-9d4c43549ab7\") " pod="openshift-service-ca/service-ca-5fffc97bdf-nfhdn" Oct 30 19:31:21.549628 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.549555 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzlcx\" (UniqueName: \"kubernetes.io/projected/b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad-kube-api-access-bzlcx\") pod \"network-check-source-6f78d59f99-kzbbk\" (UID: \"b5ecde06-7797-4df8-8cf8-1b30a5f0a8ad\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-kzbbk" Oct 30 19:31:21.550004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.549988 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfqd8\" (UniqueName: \"kubernetes.io/projected/ad103ddc-523c-4881-8526-00458255bb6c-kube-api-access-qfqd8\") pod \"kas-connection-checker-685895c7c9-fkjv2\" (UID: \"ad103ddc-523c-4881-8526-00458255bb6c\") " pod="kube-system/kas-connection-checker-685895c7c9-fkjv2" Oct 30 19:31:21.550066 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.550013 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhzl5\" (UniqueName: \"kubernetes.io/projected/6693a492-433f-4614-a8e3-71ee75c177c2-kube-api-access-jhzl5\") pod \"volume-data-source-validator-58c9f5f46d-vblv9\" (UID: \"6693a492-433f-4614-a8e3-71ee75c177c2\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-vblv9" Oct 30 19:31:21.551444 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.551426 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvx5x\" (UniqueName: \"kubernetes.io/projected/24ac5496-c680-4b2a-8362-f12b662df997-kube-api-access-qvx5x\") pod \"insights-operator-7c4986d449-7jhj7\" (UID: \"24ac5496-c680-4b2a-8362-f12b662df997\") " pod="openshift-insights/insights-operator-7c4986d449-7jhj7" Oct 30 19:31:21.551489 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.551462 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrblb\" (UniqueName: \"kubernetes.io/projected/a8e91049-4a10-4472-922d-7b7875e7de3a-kube-api-access-rrblb\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:21.551737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.551722 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dqtb\" (UniqueName: \"kubernetes.io/projected/02832de1-9584-4e12-9b94-8b6b52a71941-kube-api-access-2dqtb\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:21.552554 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.552525 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fnzv\" (UniqueName: \"kubernetes.io/projected/fb5d13ef-8197-4960-b142-d8a1bdd025e2-kube-api-access-7fnzv\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:31:21.552724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.552700 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh66m\" (UniqueName: \"kubernetes.io/projected/41c1116a-2830-4ea1-aae5-3a052f073293-kube-api-access-rh66m\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg\" (UID: \"41c1116a-2830-4ea1-aae5-3a052f073293\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" Oct 30 19:31:21.553927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.553910 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-847qk\" (UniqueName: \"kubernetes.io/projected/06bbc899-e828-4a45-bd9a-3e703c4416e7-kube-api-access-847qk\") pod \"kas-connection-checker-685895c7c9-rtf4t\" (UID: \"06bbc899-e828-4a45-bd9a-3e703c4416e7\") " pod="kube-system/kas-connection-checker-685895c7c9-rtf4t" Oct 30 19:31:21.554098 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.554080 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-bound-sa-token\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.554500 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.554480 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9vjq\" (UniqueName: \"kubernetes.io/projected/6c633f50-fd32-4440-aefc-79e44203424e-kube-api-access-j9vjq\") pod \"network-check-target-p98lt\" (UID: \"6c633f50-fd32-4440-aefc-79e44203424e\") " pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:31:21.554619 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.554603 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vkzm\" (UniqueName: \"kubernetes.io/projected/2f2ee082-2070-4390-baf3-55fa09089049-kube-api-access-6vkzm\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.554684 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.554605 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5c84\" (UniqueName: \"kubernetes.io/projected/89bcca43-c158-47fb-9cb9-1442abc2cd45-kube-api-access-g5c84\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.554757 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.554739 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzv8w\" (UniqueName: \"kubernetes.io/projected/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-api-access-wzv8w\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:21.555194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.555180 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql7gq\" (UniqueName: \"kubernetes.io/projected/534aacd9-7102-4a50-8696-c50362ce87e4-kube-api-access-ql7gq\") pod \"ingress-canary-77ms2\" (UID: \"534aacd9-7102-4a50-8696-c50362ce87e4\") " pod="openshift-ingress-canary/ingress-canary-77ms2" Oct 30 19:31:21.635016 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.634984 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-hostroot\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.635016 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635018 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-ovn\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.635284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635036 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:21.635284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635072 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:21.635284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635095 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-hostroot\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.635284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635104 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:21.635284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635161 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-run-netns\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.635284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635173 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-ovn\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.635284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635187 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-cnibin\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.635284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635211 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-etc-kubernetes\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.635284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635212 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-run-netns\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.635284 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.635224 5091 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:31:21.635284 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.635243 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle podName:a8e91049-4a10-4472-922d-7b7875e7de3a nodeName:}" failed. No retries permitted until 2026-10-30 19:31:22.135223355 +0000 UTC m=+1.830431310 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle") pod "router-default-787bddbb47-hfcvw" (UID: "a8e91049-4a10-4472-922d-7b7875e7de3a") : configmap references non-existent config key: service-ca.crt Oct 30 19:31:21.635284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635261 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-etc-kubernetes\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.635284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635244 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-cnibin\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.635284 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.635282 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls podName:01188e4b-c110-477a-ba88-d9f48741c0a2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:22.135267993 +0000 UTC m=+1.830475940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls") pod "dns-default-j2h9z" (UID: "01188e4b-c110-477a-ba88-d9f48741c0a2") : secret "dns-default-metrics-tls" not found Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.635310 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.635320 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f6d8b54bc-kmdpv: secret "image-registry-tls" not found Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635328 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.635352 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls podName:731f78f9-9938-4170-a8ff-10ba7a863759 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:22.135340616 +0000 UTC m=+1.830548564 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls") pod "image-registry-7f6d8b54bc-kmdpv" (UID: "731f78f9-9938-4170-a8ff-10ba7a863759") : secret "image-registry-tls" not found Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635367 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635375 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-sys-fs\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635402 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-hosts-file\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635417 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-system-cni-dir\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635436 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635484 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-lib-modules\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635507 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-systemd\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635532 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-etc-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635537 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-sys-fs\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635484 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-system-cni-dir\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635555 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-cni-netd\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635578 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bfb73f4-9290-4440-938a-feb9fafa3287-dbus\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635581 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/7fa163f8-129c-4f64-ad4b-ad3cc08bed13-hosts-file\") pod \"node-resolver-9g689\" (UID: \"7fa163f8-129c-4f64-ad4b-ad3cc08bed13\") " pod="openshift-dns/node-resolver-9g689" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635604 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-kubelet\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635613 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-etc-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635625 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-cni-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635643 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-system-cni-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635645 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-lib-modules\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635604 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-cni-netd\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635654 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5bfb73f4-9290-4440-938a-feb9fafa3287-dbus\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635669 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-run\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635674 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-systemd\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635698 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bfb73f4-9290-4440-938a-feb9fafa3287-kubelet-config\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635708 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-run\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635707 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-cni-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635712 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-system-cni-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635724 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635755 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5bfb73f4-9290-4440-938a-feb9fafa3287-kubelet-config\") pod \"global-pull-secret-syncer-mm6ld\" (UID: \"5bfb73f4-9290-4440-938a-feb9fafa3287\") " pod="kube-system/global-pull-secret-syncer-mm6ld" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635759 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-kubelet\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635760 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635773 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/89bcca43-c158-47fb-9cb9-1442abc2cd45-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kvwqx\" (UID: \"89bcca43-c158-47fb-9cb9-1442abc2cd45\") " pod="openshift-multus/multus-additional-cni-plugins-kvwqx" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635802 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6e429ff-3fb2-48da-a77b-7806b90139e5-crio-socket\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.635824 5091 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635833 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635857 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-node-log\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.635843 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.635864 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert podName:6e13582a-7d64-443a-a62f-c14ecc655f18 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:22.135854859 +0000 UTC m=+1.831062795 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-twdf7" (UID: "6e13582a-7d64-443a-a62f-c14ecc655f18") : secret "networking-console-plugin-cert" not found Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635798 5091 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635891 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-k8s-cni-cncf-io\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635894 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/b6e429ff-3fb2-48da-a77b-7806b90139e5-crio-socket\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635917 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysconfig\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635941 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-cnibin\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.635891 5091 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635962 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysctl-d\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635988 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-k8s-cni-cncf-io\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.636000 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls podName:b6e429ff-3fb2-48da-a77b-7806b90139e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:22.135985485 +0000 UTC m=+1.831193438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fndt2" (UID: "b6e429ff-3fb2-48da-a77b-7806b90139e5") : secret "insights-runtime-extractor-tls" not found Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.635887 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-node-log\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636010 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysconfig\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636015 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-cnibin\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636020 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-var-lib-kubelet\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636058 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-var-lib-kubelet\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636068 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636102 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysctl-d\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636108 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-socket-dir-parent\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636121 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-kubelet-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636133 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb5d13ef-8197-4960-b142-d8a1bdd025e2-host\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636176 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636180 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fb5d13ef-8197-4960-b142-d8a1bdd025e2-host\") pod \"node-ca-m77bc\" (UID: \"fb5d13ef-8197-4960-b142-d8a1bdd025e2\") " pod="openshift-image-registry/node-ca-m77bc" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636196 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-socket-dir-parent\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636221 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-cni-bin\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636247 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-socket-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636269 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-modprobe-d\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636285 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-cni-bin\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636291 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-slash\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636315 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-var-lib-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636326 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-socket-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636338 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636350 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-slash\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636362 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-run-ovn-kubernetes\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636390 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636413 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-modprobe-d\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636413 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-systemd\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636439 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-systemd-units\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636450 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-systemd\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636462 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636484 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-host\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636487 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-var-lib-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636503 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-systemd-units\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636508 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-kubelet\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636545 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-kubelet\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636525 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-host-run-ovn-kubernetes\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.636567 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.636581 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-549665644b-9xg4z: secret "image-registry-tls" not found Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636594 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-host\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.636623 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls podName:8161b0bd-5184-44fc-864e-869dc7737a55 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:22.136602327 +0000 UTC m=+1.831810267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls") pod "image-registry-549665644b-9xg4z" (UID: "8161b0bd-5184-44fc-864e-869dc7737a55") : secret "image-registry-tls" not found Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636563 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.636646 5091 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636652 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysctl-conf\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636670 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-log-socket\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636686 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-cni-bin\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:21.636700 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs podName:a8e91049-4a10-4472-922d-7b7875e7de3a nodeName:}" failed. No retries permitted until 2026-10-30 19:31:22.136683736 +0000 UTC m=+1.831891672 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs") pod "router-default-787bddbb47-hfcvw" (UID: "a8e91049-4a10-4472-922d-7b7875e7de3a") : secret "router-metrics-certs-default" not found Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636714 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-cni-bin\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636648 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-run-openvswitch\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636737 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc37d377-9cb8-4f67-aa5d-1368a886533c-host-slash\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636744 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d95b4719-782e-47f3-9a7d-ea04268f82da-log-socket\") pod \"ovnkube-node-fg2zw\" (UID: \"d95b4719-782e-47f3-9a7d-ea04268f82da\") " pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636768 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-kubernetes\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636776 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/cc37d377-9cb8-4f67-aa5d-1368a886533c-host-slash\") pod \"iptables-alerter-pcsxp\" (UID: \"cc37d377-9cb8-4f67-aa5d-1368a886533c\") " pod="openshift-network-operator/iptables-alerter-pcsxp" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636769 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-sysctl-conf\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636792 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-sys\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636810 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-etc-kubernetes\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636821 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-conf-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636829 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d7792ad-14ae-482b-9def-eda8e029dcda-sys\") pod \"tuned-mnf6c\" (UID: \"8d7792ad-14ae-482b-9def-eda8e029dcda\") " pod="openshift-cluster-node-tuning-operator/tuned-mnf6c" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636850 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-multus-certs\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636878 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-multus-conf-dir\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636877 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-netns\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636916 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636925 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-multus-certs\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636945 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-device-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636972 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-etc-selinux\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.636995 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-cni-multus\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.637020 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-registration-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.637102 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-etc-selinux\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.637113 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-registration-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.637173 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-var-lib-cni-multus\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.637184 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2f2ee082-2070-4390-baf3-55fa09089049-device-dir\") pod \"aws-ebs-csi-driver-node-x756d\" (UID: \"2f2ee082-2070-4390-baf3-55fa09089049\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-x756d" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.637204 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35066f63-29d2-42da-83b7-edf8894d8e5c-host-run-netns\") pod \"multus-zzxkl\" (UID: \"35066f63-29d2-42da-83b7-edf8894d8e5c\") " pod="openshift-multus/multus-zzxkl" Oct 30 19:31:21.637770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.637414 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/b6e429ff-3fb2-48da-a77b-7806b90139e5-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:21.640579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.639131 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af2922a1-2b0f-4ee1-8ef3-4911d029e93d-metrics-certs\") pod \"network-metrics-daemon-t4fw7\" (UID: \"af2922a1-2b0f-4ee1-8ef3-4911d029e93d\") " pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:31:21.640579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.639162 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/7d903311-c71c-4197-8ac3-486702bd46e9-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b8bsp\" (UID: \"7d903311-c71c-4197-8ac3-486702bd46e9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:21.640579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.639364 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02832de1-9584-4e12-9b94-8b6b52a71941-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-fcjbl\" (UID: \"02832de1-9584-4e12-9b94-8b6b52a71941\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:21.665290 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.665251 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:21.714372 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.714353 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" Oct 30 19:31:21.718477 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.718449 5091 scope.go:122] "RemoveContainer" containerID="d2b9813548f63ce8ff5ee1d96c7d773bedbaa3663d02970290b3b3f268e977b6" Oct 30 19:31:21.719396 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.719379 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" Oct 30 19:31:21.721207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.721190 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-t4fw7" Oct 30 19:31:21.824111 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.824032 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:21.970181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:21.970085 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:22.044396 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.044372 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp"] Oct 30 19:31:22.063837 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.063816 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl"] Oct 30 19:31:22.066361 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:22.066336 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02832de1_9584_4e12_9b94_8b6b52a71941.slice/crio-55a878348e140bd04648b18f8593ca17dd959c10deab5f4ef5a9bdd279eef395 WatchSource:0}: Error finding container 55a878348e140bd04648b18f8593ca17dd959c10deab5f4ef5a9bdd279eef395: Status 404 returned error can't find the container with id 55a878348e140bd04648b18f8593ca17dd959c10deab5f4ef5a9bdd279eef395 Oct 30 19:31:22.074408 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.069374 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:31:22.122922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.122900 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-t4fw7"] Oct 30 19:31:22.124895 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:22.124866 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2922a1_2b0f_4ee1_8ef3_4911d029e93d.slice/crio-fc749058591b6a1106b0b21a935889be1bae417946bfe37c7a42bdb098b9c39c WatchSource:0}: Error finding container fc749058591b6a1106b0b21a935889be1bae417946bfe37c7a42bdb098b9c39c: Status 404 returned error can't find the container with id fc749058591b6a1106b0b21a935889be1bae417946bfe37c7a42bdb098b9c39c Oct 30 19:31:22.133436 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.133413 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:22.141280 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.141258 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:22.141368 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.141289 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:22.141368 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.141312 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:22.141466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.141373 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:22.141466 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:22.141394 5091 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:31:22.141466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.141403 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:22.141466 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:22.141399 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:31:22.141466 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:22.141441 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7f6d8b54bc-kmdpv: secret "image-registry-tls" not found Oct 30 19:31:22.141466 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:22.141448 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls podName:01188e4b-c110-477a-ba88-d9f48741c0a2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:23.141430008 +0000 UTC m=+2.836637945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls") pod "dns-default-j2h9z" (UID: "01188e4b-c110-477a-ba88-d9f48741c0a2") : secret "dns-default-metrics-tls" not found Oct 30 19:31:22.141466 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:22.141456 5091 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:31:22.141791 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:22.141480 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls podName:731f78f9-9938-4170-a8ff-10ba7a863759 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:23.141466169 +0000 UTC m=+2.836674121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls") pod "image-registry-7f6d8b54bc-kmdpv" (UID: "731f78f9-9938-4170-a8ff-10ba7a863759") : secret "image-registry-tls" not found Oct 30 19:31:22.141791 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:22.141496 5091 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:31:22.141791 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:22.141511 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle podName:a8e91049-4a10-4472-922d-7b7875e7de3a nodeName:}" failed. No retries permitted until 2026-10-30 19:31:23.141493118 +0000 UTC m=+2.836701060 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle") pod "router-default-787bddbb47-hfcvw" (UID: "a8e91049-4a10-4472-922d-7b7875e7de3a") : configmap references non-existent config key: service-ca.crt Oct 30 19:31:22.141791 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:22.141538 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert podName:6e13582a-7d64-443a-a62f-c14ecc655f18 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:23.141527355 +0000 UTC m=+2.836735307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-twdf7" (UID: "6e13582a-7d64-443a-a62f-c14ecc655f18") : secret "networking-console-plugin-cert" not found Oct 30 19:31:22.141791 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:22.141557 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls podName:b6e429ff-3fb2-48da-a77b-7806b90139e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:23.141547812 +0000 UTC m=+2.836755755 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fndt2" (UID: "b6e429ff-3fb2-48da-a77b-7806b90139e5") : secret "insights-runtime-extractor-tls" not found Oct 30 19:31:22.141791 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.141602 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:22.141791 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.141637 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:22.141791 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:22.141699 5091 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Oct 30 19:31:22.141791 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:22.141718 5091 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-549665644b-9xg4z: secret "image-registry-tls" not found Oct 30 19:31:22.141791 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:22.141739 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls podName:8161b0bd-5184-44fc-864e-869dc7737a55 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:23.141732014 +0000 UTC m=+2.836939993 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls") pod "image-registry-549665644b-9xg4z" (UID: "8161b0bd-5184-44fc-864e-869dc7737a55") : secret "image-registry-tls" not found Oct 30 19:31:22.141791 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:22.141764 5091 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:31:22.142138 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:22.141803 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs podName:a8e91049-4a10-4472-922d-7b7875e7de3a nodeName:}" failed. No retries permitted until 2026-10-30 19:31:23.141792977 +0000 UTC m=+2.837000929 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs") pod "router-default-787bddbb47-hfcvw" (UID: "a8e91049-4a10-4472-922d-7b7875e7de3a") : secret "router-metrics-certs-default" not found Oct 30 19:31:22.295437 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.295398 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:22.485685 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.485643 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:22.495592 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.495563 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" event={"ID":"7d903311-c71c-4197-8ac3-486702bd46e9","Type":"ContainerStarted","Data":"96a13453e3ac8c6e96703a1d7ac1fa0f034aa30b14cbc2fcf108292280222436"} Oct 30 19:31:22.496628 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.496606 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" event={"ID":"02832de1-9584-4e12-9b94-8b6b52a71941","Type":"ContainerStarted","Data":"55a878348e140bd04648b18f8593ca17dd959c10deab5f4ef5a9bdd279eef395"} Oct 30 19:31:22.498319 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.498304 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:31:22.498397 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.498384 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" event={"ID":"067a4436-690d-4b2c-aaf2-1d84830b7287","Type":"ContainerStarted","Data":"ced65dab5279fdd821fd394054bfbf937213926672b257ddd28184322a023fff"} Oct 30 19:31:22.499533 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.499506 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t4fw7" event={"ID":"af2922a1-2b0f-4ee1-8ef3-4911d029e93d","Type":"ContainerStarted","Data":"fc749058591b6a1106b0b21a935889be1bae417946bfe37c7a42bdb098b9c39c"} Oct 30 19:31:22.636420 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.636347 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:22.783169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.783111 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:22.935771 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:22.935644 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:23.152131 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.152094 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:23.152316 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.152138 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:23.152316 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.152214 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:23.152316 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.152237 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:23.152316 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.152266 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:23.152478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.152323 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:23.152478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.152352 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:23.152478 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:23.152473 5091 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:31:23.152616 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:23.152544 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls podName:b6e429ff-3fb2-48da-a77b-7806b90139e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:25.152518449 +0000 UTC m=+4.847726400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fndt2" (UID: "b6e429ff-3fb2-48da-a77b-7806b90139e5") : secret "insights-runtime-extractor-tls" not found Oct 30 19:31:23.152680 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:23.152623 5091 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:31:23.152680 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:23.152647 5091 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Oct 30 19:31:23.152747 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:23.152689 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls podName:01188e4b-c110-477a-ba88-d9f48741c0a2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:25.152670394 +0000 UTC m=+4.847878350 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls") pod "dns-default-j2h9z" (UID: "01188e4b-c110-477a-ba88-d9f48741c0a2") : secret "dns-default-metrics-tls" not found Oct 30 19:31:23.152747 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:23.152710 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs podName:a8e91049-4a10-4472-922d-7b7875e7de3a nodeName:}" failed. No retries permitted until 2026-10-30 19:31:25.152700856 +0000 UTC m=+4.847908792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs") pod "router-default-787bddbb47-hfcvw" (UID: "a8e91049-4a10-4472-922d-7b7875e7de3a") : secret "router-metrics-certs-default" not found Oct 30 19:31:23.152747 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:23.152720 5091 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Oct 30 19:31:23.152854 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:23.152776 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert podName:6e13582a-7d64-443a-a62f-c14ecc655f18 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:25.152760249 +0000 UTC m=+4.847968192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-twdf7" (UID: "6e13582a-7d64-443a-a62f-c14ecc655f18") : secret "networking-console-plugin-cert" not found Oct 30 19:31:23.153363 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.153339 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8e91049-4a10-4472-922d-7b7875e7de3a-service-ca-bundle\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:23.155891 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.155864 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls\") pod \"image-registry-549665644b-9xg4z\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:23.156940 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.156917 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls\") pod \"image-registry-7f6d8b54bc-kmdpv\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:23.212447 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.212366 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:23.213882 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.213852 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:23.227201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.226994 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:23.355179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.355131 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:23.379912 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.379184 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:31:23.495020 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.494949 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:23.515123 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.515097 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f6d8b54bc-kmdpv"] Oct 30 19:31:23.525176 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:23.524645 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731f78f9_9938_4170_a8ff_10ba7a863759.slice/crio-81d5c0bd3ab58839f8ca49130593d58f2faaf288e79234b364ca45260b8a713a WatchSource:0}: Error finding container 81d5c0bd3ab58839f8ca49130593d58f2faaf288e79234b364ca45260b8a713a: Status 404 returned error can't find the container with id 81d5c0bd3ab58839f8ca49130593d58f2faaf288e79234b364ca45260b8a713a Oct 30 19:31:23.542549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.542523 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-549665644b-9xg4z"] Oct 30 19:31:23.648523 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.643556 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:23.792738 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.792700 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:23.934653 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:23.934618 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:24.081971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.081889 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:24.229139 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.229102 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:24.349787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.349761 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:24.349946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.349931 5091 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:31:24.355979 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.355950 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-4kpqr" Oct 30 19:31:24.433214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.433186 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:24.510929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.510892 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" event={"ID":"7d903311-c71c-4197-8ac3-486702bd46e9","Type":"ContainerStarted","Data":"6307934ba2d93e4c43e568205711626c60b2bcb163cf9e3d0f141603798345f5"} Oct 30 19:31:24.511370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.510952 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b8bsp" event={"ID":"7d903311-c71c-4197-8ac3-486702bd46e9","Type":"ContainerStarted","Data":"966919dab164ff7ab85ef130d6d68782ca88e47fd2adb9d5a34244f6007c0281"} Oct 30 19:31:24.512741 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.512712 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-549665644b-9xg4z" event={"ID":"8161b0bd-5184-44fc-864e-869dc7737a55","Type":"ContainerStarted","Data":"1c72f807c70252639d4f6fac30fadf7c4bef5f8e143c7519cf4e0c3f09bcf4a9"} Oct 30 19:31:24.512853 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.512760 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-549665644b-9xg4z" event={"ID":"8161b0bd-5184-44fc-864e-869dc7737a55","Type":"ContainerStarted","Data":"fd91f9ee7b191add71a312b2a519f1da0cfd79b926e296d651b82fa36697b6b3"} Oct 30 19:31:24.514406 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.514381 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" event={"ID":"731f78f9-9938-4170-a8ff-10ba7a863759","Type":"ContainerStarted","Data":"6feab0f746fc4304bfd398967c1b80673ca06a678d7a030dbd25a2c54ce7b22d"} Oct 30 19:31:24.514617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.514600 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" event={"ID":"731f78f9-9938-4170-a8ff-10ba7a863759","Type":"ContainerStarted","Data":"81d5c0bd3ab58839f8ca49130593d58f2faaf288e79234b364ca45260b8a713a"} Oct 30 19:31:24.515976 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.515953 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-fcjbl" event={"ID":"02832de1-9584-4e12-9b94-8b6b52a71941","Type":"ContainerStarted","Data":"194dc52dae067256687b7b024aadec89c5d145ad4ec4f9dd30d1e2b92f38948b"} Oct 30 19:31:24.518095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.518070 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t4fw7" event={"ID":"af2922a1-2b0f-4ee1-8ef3-4911d029e93d","Type":"ContainerStarted","Data":"a2d1ecdc73d8637a4c0927e2f1d4a97b8a3362ac26e0427900994d7c5369f7f0"} Oct 30 19:31:24.518193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.518117 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-t4fw7" event={"ID":"af2922a1-2b0f-4ee1-8ef3-4911d029e93d","Type":"ContainerStarted","Data":"5e65ce12ae5b1673831d759a019e41af6d862a2fcb468abcd73fbe21668dedec"} Oct 30 19:31:24.588956 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.588893 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:24.732543 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.732499 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:24.872313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:24.872238 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:25.013800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.013763 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:25.172800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.172731 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:25.172800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.172778 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:25.173008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.172813 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:25.173008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.172831 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:25.173008 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:25.172920 5091 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Oct 30 19:31:25.173008 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:25.172932 5091 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Oct 30 19:31:25.173008 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:25.172975 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls podName:b6e429ff-3fb2-48da-a77b-7806b90139e5 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:29.172962357 +0000 UTC m=+8.868170294 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls") pod "insights-runtime-extractor-fndt2" (UID: "b6e429ff-3fb2-48da-a77b-7806b90139e5") : secret "insights-runtime-extractor-tls" not found Oct 30 19:31:25.173008 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:25.172992 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls podName:01188e4b-c110-477a-ba88-d9f48741c0a2 nodeName:}" failed. No retries permitted until 2026-10-30 19:31:29.172983313 +0000 UTC m=+8.868191268 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls") pod "dns-default-j2h9z" (UID: "01188e4b-c110-477a-ba88-d9f48741c0a2") : secret "dns-default-metrics-tls" not found Oct 30 19:31:25.175044 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.175023 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8e91049-4a10-4472-922d-7b7875e7de3a-metrics-certs\") pod \"router-default-787bddbb47-hfcvw\" (UID: \"a8e91049-4a10-4472-922d-7b7875e7de3a\") " pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:25.175252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.175224 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/6e13582a-7d64-443a-a62f-c14ecc655f18-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-twdf7\" (UID: \"6e13582a-7d64-443a-a62f-c14ecc655f18\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:25.314156 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.314118 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:25.320518 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.320495 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" Oct 30 19:31:25.523572 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.523546 5091 generic.go:356] "Generic (PLEG): container finished" podID="24ac5496-c680-4b2a-8362-f12b662df997" containerID="0ba6c02fc5a5d01e405ee62c66025ff3dbd10d48025eb6c710e785a903710bea" exitCode=0 Oct 30 19:31:25.523928 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.523614 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7jhj7" event={"ID":"24ac5496-c680-4b2a-8362-f12b662df997","Type":"ContainerDied","Data":"0ba6c02fc5a5d01e405ee62c66025ff3dbd10d48025eb6c710e785a903710bea"} Oct 30 19:31:25.524000 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.523982 5091 scope.go:122] "RemoveContainer" containerID="0ba6c02fc5a5d01e405ee62c66025ff3dbd10d48025eb6c710e785a903710bea" Oct 30 19:31:25.525204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.525178 5091 generic.go:356] "Generic (PLEG): container finished" podID="6c260520-55b0-44ca-8ccb-e15cab54fa90" containerID="691006c8d5acdb575b9c091b3ec5d9271e8a6d912e6611228c073107f34ae2eb" exitCode=0 Oct 30 19:31:25.525321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.525305 5091 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:31:25.525426 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.525316 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" event={"ID":"6c260520-55b0-44ca-8ccb-e15cab54fa90","Type":"ContainerDied","Data":"691006c8d5acdb575b9c091b3ec5d9271e8a6d912e6611228c073107f34ae2eb"} Oct 30 19:31:25.529182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.526575 5091 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:31:25.529182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.526657 5091 scope.go:122] "RemoveContainer" containerID="691006c8d5acdb575b9c091b3ec5d9271e8a6d912e6611228c073107f34ae2eb" Oct 30 19:31:25.593594 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.593567 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-twdf7"] Oct 30 19:31:25.596622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.596602 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-787bddbb47-hfcvw"] Oct 30 19:31:25.613227 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:25.613199 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e13582a_7d64_443a_a62f_c14ecc655f18.slice/crio-54d5ea9035919df07c8d2629a88838db46c63c17296d42acf7219220a5ef3a0a WatchSource:0}: Error finding container 54d5ea9035919df07c8d2629a88838db46c63c17296d42acf7219220a5ef3a0a: Status 404 returned error can't find the container with id 54d5ea9035919df07c8d2629a88838db46c63c17296d42acf7219220a5ef3a0a Oct 30 19:31:25.613406 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:25.613390 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e91049_4a10_4472_922d_7b7875e7de3a.slice/crio-78fe2979ae638b9faecc9702cf5c4361c8988b027391281cadd9a8814f9c4825 WatchSource:0}: Error finding container 78fe2979ae638b9faecc9702cf5c4361c8988b027391281cadd9a8814f9c4825: Status 404 returned error can't find the container with id 78fe2979ae638b9faecc9702cf5c4361c8988b027391281cadd9a8814f9c4825 Oct 30 19:31:25.670008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.669920 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:25.875907 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:25.875819 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:26.096834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:26.096797 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:26.261400 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:26.261364 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:26.442832 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:26.442542 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:26.444708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:26.444681 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:26.536257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:26.535627 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-dqnkb" event={"ID":"6c260520-55b0-44ca-8ccb-e15cab54fa90","Type":"ContainerStarted","Data":"c24c411cfaae224bd9326b715b0cf28bff879a1bee00545e4bf755a8dd1b106c"} Oct 30 19:31:26.536929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:26.536900 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" event={"ID":"6e13582a-7d64-443a-a62f-c14ecc655f18","Type":"ContainerStarted","Data":"54d5ea9035919df07c8d2629a88838db46c63c17296d42acf7219220a5ef3a0a"} Oct 30 19:31:26.538557 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:26.538529 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-787bddbb47-hfcvw" event={"ID":"a8e91049-4a10-4472-922d-7b7875e7de3a","Type":"ContainerStarted","Data":"ca123af111fcf81b1d50af1d1b39b0c4ef841f71e926720e2603f25321f5c5db"} Oct 30 19:31:26.538630 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:26.538580 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-787bddbb47-hfcvw" event={"ID":"a8e91049-4a10-4472-922d-7b7875e7de3a","Type":"ContainerStarted","Data":"78fe2979ae638b9faecc9702cf5c4361c8988b027391281cadd9a8814f9c4825"} Oct 30 19:31:26.540490 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:26.540467 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7jhj7" event={"ID":"24ac5496-c680-4b2a-8362-f12b662df997","Type":"ContainerStarted","Data":"6ee53317f78eae384b46e7782f411073ab64bba46028c5302fdab485a4e50828"} Oct 30 19:31:26.638403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:26.638367 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:27.315024 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:27.314988 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:27.317706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:27.317681 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:27.549901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:27.549865 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-twdf7" event={"ID":"6e13582a-7d64-443a-a62f-c14ecc655f18","Type":"ContainerStarted","Data":"78ca9c717ddb9ef148a0e8476c3a280499daf1d0ae4b68f0f167960335e3dfcf"} Oct 30 19:31:27.668677 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:27.668558 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:27.689336 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:27.689307 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:27.869558 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:27.869505 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:28.071067 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:28.071033 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:28.281238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:28.281204 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:28.422458 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:28.422379 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:28.554747 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:28.554711 5091 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:31:28.563358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:28.563337 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:28.573208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:28.573182 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:29.074100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:29.074067 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:29.209599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:29.209571 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:29.209758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:29.209620 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:29.211887 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:29.211857 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/b6e429ff-3fb2-48da-a77b-7806b90139e5-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-fndt2\" (UID: \"b6e429ff-3fb2-48da-a77b-7806b90139e5\") " pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:29.212005 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:29.211863 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/01188e4b-c110-477a-ba88-d9f48741c0a2-metrics-tls\") pod \"dns-default-j2h9z\" (UID: \"01188e4b-c110-477a-ba88-d9f48741c0a2\") " pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:29.227847 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:29.227827 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:29.228920 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:29.228897 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-fndt2" Oct 30 19:31:29.271416 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:29.271378 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:29.507487 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:29.507465 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-j2h9z"] Oct 30 19:31:29.509863 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:29.509838 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01188e4b_c110_477a_ba88_d9f48741c0a2.slice/crio-ce853787befaeab3745de1d59acc25bddca5bedce0ba5dc94e1f1b5dc7a8edad WatchSource:0}: Error finding container ce853787befaeab3745de1d59acc25bddca5bedce0ba5dc94e1f1b5dc7a8edad: Status 404 returned error can't find the container with id ce853787befaeab3745de1d59acc25bddca5bedce0ba5dc94e1f1b5dc7a8edad Oct 30 19:31:29.512279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:29.512197 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-fndt2"] Oct 30 19:31:29.514351 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:29.514330 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6e429ff_3fb2_48da_a77b_7806b90139e5.slice/crio-ad9692c97441fb627c7c47581a89672fb5d0848e0ff7f7149942951ac30c6d82 WatchSource:0}: Error finding container ad9692c97441fb627c7c47581a89672fb5d0848e0ff7f7149942951ac30c6d82: Status 404 returned error can't find the container with id ad9692c97441fb627c7c47581a89672fb5d0848e0ff7f7149942951ac30c6d82 Oct 30 19:31:29.560311 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:29.560284 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fndt2" event={"ID":"b6e429ff-3fb2-48da-a77b-7806b90139e5","Type":"ContainerStarted","Data":"ad9692c97441fb627c7c47581a89672fb5d0848e0ff7f7149942951ac30c6d82"} Oct 30 19:31:29.561417 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:29.561392 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j2h9z" event={"ID":"01188e4b-c110-477a-ba88-d9f48741c0a2","Type":"ContainerStarted","Data":"ce853787befaeab3745de1d59acc25bddca5bedce0ba5dc94e1f1b5dc7a8edad"} Oct 30 19:31:30.567625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:30.567590 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fndt2" event={"ID":"b6e429ff-3fb2-48da-a77b-7806b90139e5","Type":"ContainerStarted","Data":"356ca6695aa7e9a44f7a06cf782b690348182593f903dca52fec6853de8daec8"} Oct 30 19:31:30.568032 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:30.567661 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fndt2" event={"ID":"b6e429ff-3fb2-48da-a77b-7806b90139e5","Type":"ContainerStarted","Data":"bbb83a537f2f0b6dcdced02c1a5e511645bd115a2bbf225ce52bda47c79104ef"} Oct 30 19:31:31.126617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:31.126574 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:31.266377 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:31.266339 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:31.344198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:31.344084 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:31.344361 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:31.344348 5091 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:31:31.572228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:31.572195 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:31.572681 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:31.572327 5091 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 19:31:31.573740 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:31.573716 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-787bddbb47-hfcvw" Oct 30 19:31:31.575578 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:31.575552 5091 generic.go:356] "Generic (PLEG): container finished" podID="24ac5496-c680-4b2a-8362-f12b662df997" containerID="6ee53317f78eae384b46e7782f411073ab64bba46028c5302fdab485a4e50828" exitCode=0 Oct 30 19:31:31.575679 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:31.575631 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7jhj7" event={"ID":"24ac5496-c680-4b2a-8362-f12b662df997","Type":"ContainerDied","Data":"6ee53317f78eae384b46e7782f411073ab64bba46028c5302fdab485a4e50828"} Oct 30 19:31:31.575738 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:31.575694 5091 scope.go:122] "RemoveContainer" containerID="0ba6c02fc5a5d01e405ee62c66025ff3dbd10d48025eb6c710e785a903710bea" Oct 30 19:31:31.576015 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:31.575992 5091 scope.go:122] "RemoveContainer" containerID="6ee53317f78eae384b46e7782f411073ab64bba46028c5302fdab485a4e50828" Oct 30 19:31:31.576241 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:31.576220 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-7jhj7_openshift-insights(24ac5496-c680-4b2a-8362-f12b662df997)\"" pod="openshift-insights/insights-operator-7c4986d449-7jhj7" podUID="24ac5496-c680-4b2a-8362-f12b662df997" Oct 30 19:31:31.577715 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:31.577696 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j2h9z" event={"ID":"01188e4b-c110-477a-ba88-d9f48741c0a2","Type":"ContainerStarted","Data":"05bba9a2cb4422d0ab030e1ee591f6fefe23e7b95bbf175fd195bc06f6032568"} Oct 30 19:31:31.577800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:31.577730 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-j2h9z" event={"ID":"01188e4b-c110-477a-ba88-d9f48741c0a2","Type":"ContainerStarted","Data":"18796369e6eff1567ebc6cb4cbe44a4dfe7560689f7c2d02bda4c7273af94176"} Oct 30 19:31:31.577855 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:31.577837 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:32.583343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:32.583306 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-fndt2" event={"ID":"b6e429ff-3fb2-48da-a77b-7806b90139e5","Type":"ContainerStarted","Data":"5741d2bfd357f9d1c7330e0f91bd57e6a82babc0b0c37d332454a3e6e15c7c4e"} Oct 30 19:31:34.829727 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.829681 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-tllvs-gxphx"] Oct 30 19:31:34.833159 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.833122 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:31:34.853445 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.853419 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tllvs-gxphx"] Oct 30 19:31:34.910209 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.910166 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-smbbf"] Oct 30 19:31:34.913934 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.913910 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-556ccfcb78-9mh6t"] Oct 30 19:31:34.914072 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.914052 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-smbbf" Oct 30 19:31:34.917198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.917178 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-xxslt"] Oct 30 19:31:34.917321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.917286 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:34.920730 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.920711 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-xxslt" Oct 30 19:31:34.921187 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.921169 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f6d8b54bc-kmdpv"] Oct 30 19:31:34.929357 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.929338 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 19:31:34.938270 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.937860 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 19:31:34.938270 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.938021 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-7k7ng\"" Oct 30 19:31:34.939059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.938819 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-vm22p\"" Oct 30 19:31:34.939059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.938860 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 19:31:34.939362 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.939326 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 19:31:34.940544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.940515 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 19:31:34.940769 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.940576 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 19:31:34.940866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.940794 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 19:31:34.941007 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.940985 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-f6m98\"" Oct 30 19:31:34.941128 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.941109 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Oct 30 19:31:34.943202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.943098 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556ccfcb78-9mh6t"] Oct 30 19:31:34.943202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.943169 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-smbbf"] Oct 30 19:31:34.956697 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.956670 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-xxslt"] Oct 30 19:31:34.963710 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.963682 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fed516ca-1ba6-4094-8bb4-342bdb41638e-archives-path\") pod \"periodic-gathering-tllvs-gxphx\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:31:34.963835 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.963718 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed516ca-1ba6-4094-8bb4-342bdb41638e-service-ca-bundle\") pod \"periodic-gathering-tllvs-gxphx\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:31:34.963835 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.963736 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed516ca-1ba6-4094-8bb4-342bdb41638e-trusted-ca-bundle\") pod \"periodic-gathering-tllvs-gxphx\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:31:34.963835 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:34.963754 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn2cz\" (UniqueName: \"kubernetes.io/projected/fed516ca-1ba6-4094-8bb4-342bdb41638e-kube-api-access-wn2cz\") pod \"periodic-gathering-tllvs-gxphx\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:31:35.027555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.027525 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-79fdc76dbd-5q48r"] Oct 30 19:31:35.030885 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.030865 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.061760 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.061729 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79fdc76dbd-5q48r"] Oct 30 19:31:35.065121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.065089 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f936e891-6ea7-4d4c-881f-867120559e66-console-serving-cert\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.065350 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.065329 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fed516ca-1ba6-4094-8bb4-342bdb41638e-archives-path\") pod \"periodic-gathering-tllvs-gxphx\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:31:35.065467 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.065453 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed516ca-1ba6-4094-8bb4-342bdb41638e-service-ca-bundle\") pod \"periodic-gathering-tllvs-gxphx\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:31:35.065564 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.065551 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed516ca-1ba6-4094-8bb4-342bdb41638e-trusted-ca-bundle\") pod \"periodic-gathering-tllvs-gxphx\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:31:35.065664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.065649 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn2cz\" (UniqueName: \"kubernetes.io/projected/fed516ca-1ba6-4094-8bb4-342bdb41638e-kube-api-access-wn2cz\") pod \"periodic-gathering-tllvs-gxphx\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:31:35.065758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.065746 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-service-ca\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.065851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.065837 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f936e891-6ea7-4d4c-881f-867120559e66-console-oauth-config\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.065955 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.065943 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69hbt\" (UniqueName: \"kubernetes.io/projected/f936e891-6ea7-4d4c-881f-867120559e66-kube-api-access-69hbt\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.066064 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.066050 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f936e891-6ea7-4d4c-881f-867120559e66-tmp\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.066184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.066170 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3052bdd-cd3b-4d52-998c-538138c6d060-tmp\") pod \"downloads-6dd789c8b7-xxslt\" (UID: \"d3052bdd-cd3b-4d52-998c-538138c6d060\") " pod="openshift-console/downloads-6dd789c8b7-xxslt" Oct 30 19:31:35.066316 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.066304 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-console-config\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.066417 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.066404 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-oauth-serving-cert\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.066512 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.066500 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54drh\" (UniqueName: \"kubernetes.io/projected/d3052bdd-cd3b-4d52-998c-538138c6d060-kube-api-access-54drh\") pod \"downloads-6dd789c8b7-xxslt\" (UID: \"d3052bdd-cd3b-4d52-998c-538138c6d060\") " pod="openshift-console/downloads-6dd789c8b7-xxslt" Oct 30 19:31:35.066635 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.066597 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/216ca825-8a83-4100-ba3b-7d6294c3dadf-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-smbbf\" (UID: \"216ca825-8a83-4100-ba3b-7d6294c3dadf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-smbbf" Oct 30 19:31:35.067236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.067207 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fed516ca-1ba6-4094-8bb4-342bdb41638e-archives-path\") pod \"periodic-gathering-tllvs-gxphx\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:31:35.067679 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.066626 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed516ca-1ba6-4094-8bb4-342bdb41638e-service-ca-bundle\") pod \"periodic-gathering-tllvs-gxphx\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:31:35.067873 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.067828 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed516ca-1ba6-4094-8bb4-342bdb41638e-trusted-ca-bundle\") pod \"periodic-gathering-tllvs-gxphx\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:31:35.072385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.072359 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:35.087636 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.087584 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn2cz\" (UniqueName: \"kubernetes.io/projected/fed516ca-1ba6-4094-8bb4-342bdb41638e-kube-api-access-wn2cz\") pod \"periodic-gathering-tllvs-gxphx\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:31:35.146117 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.146089 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:31:35.168174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.168120 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/216ca825-8a83-4100-ba3b-7d6294c3dadf-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-smbbf\" (UID: \"216ca825-8a83-4100-ba3b-7d6294c3dadf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-smbbf" Oct 30 19:31:35.168328 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.168214 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-registry-certificates\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.168328 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.168245 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-bound-sa-token\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.168328 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.168269 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-554rp\" (UniqueName: \"kubernetes.io/projected/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-kube-api-access-554rp\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.168483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.168346 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f936e891-6ea7-4d4c-881f-867120559e66-console-serving-cert\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.168559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.168541 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-service-ca\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.168616 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.168591 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-ca-trust-extracted\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.168669 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.168622 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f936e891-6ea7-4d4c-881f-867120559e66-console-oauth-config\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.168733 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.168706 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-installation-pull-secrets\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.168820 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.168802 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69hbt\" (UniqueName: \"kubernetes.io/projected/f936e891-6ea7-4d4c-881f-867120559e66-kube-api-access-69hbt\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.168880 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.168854 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f936e891-6ea7-4d4c-881f-867120559e66-tmp\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.168937 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.168894 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3052bdd-cd3b-4d52-998c-538138c6d060-tmp\") pod \"downloads-6dd789c8b7-xxslt\" (UID: \"d3052bdd-cd3b-4d52-998c-538138c6d060\") " pod="openshift-console/downloads-6dd789c8b7-xxslt" Oct 30 19:31:35.168937 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.168922 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-image-registry-private-configuration\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.169043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.168969 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-console-config\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.169043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.168997 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-oauth-serving-cert\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.169043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.169023 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-trusted-ca\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.169235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.169053 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54drh\" (UniqueName: \"kubernetes.io/projected/d3052bdd-cd3b-4d52-998c-538138c6d060-kube-api-access-54drh\") pod \"downloads-6dd789c8b7-xxslt\" (UID: \"d3052bdd-cd3b-4d52-998c-538138c6d060\") " pod="openshift-console/downloads-6dd789c8b7-xxslt" Oct 30 19:31:35.169235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.169077 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-registry-tls\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.170215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.169355 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d3052bdd-cd3b-4d52-998c-538138c6d060-tmp\") pod \"downloads-6dd789c8b7-xxslt\" (UID: \"d3052bdd-cd3b-4d52-998c-538138c6d060\") " pod="openshift-console/downloads-6dd789c8b7-xxslt" Oct 30 19:31:35.170215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.169517 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-service-ca\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.170215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.169740 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f936e891-6ea7-4d4c-881f-867120559e66-tmp\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.170215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.169898 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-console-config\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.170215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.170172 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-oauth-serving-cert\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.171019 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.170994 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/216ca825-8a83-4100-ba3b-7d6294c3dadf-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-smbbf\" (UID: \"216ca825-8a83-4100-ba3b-7d6294c3dadf\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-smbbf" Oct 30 19:31:35.171101 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.171042 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f936e891-6ea7-4d4c-881f-867120559e66-console-serving-cert\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.171892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.171866 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f936e891-6ea7-4d4c-881f-867120559e66-console-oauth-config\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.184270 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.184216 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54drh\" (UniqueName: \"kubernetes.io/projected/d3052bdd-cd3b-4d52-998c-538138c6d060-kube-api-access-54drh\") pod \"downloads-6dd789c8b7-xxslt\" (UID: \"d3052bdd-cd3b-4d52-998c-538138c6d060\") " pod="openshift-console/downloads-6dd789c8b7-xxslt" Oct 30 19:31:35.185227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.185181 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69hbt\" (UniqueName: \"kubernetes.io/projected/f936e891-6ea7-4d4c-881f-867120559e66-kube-api-access-69hbt\") pod \"console-556ccfcb78-9mh6t\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.233990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.233960 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-smbbf" Oct 30 19:31:35.238118 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.238087 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:35.245935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.245911 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-xxslt" Oct 30 19:31:35.267932 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.267897 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:35.270214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.270181 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-image-registry-private-configuration\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.271034 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.271000 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-trusted-ca\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.272385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.271183 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-registry-tls\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.272762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.272738 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-registry-certificates\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.272875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.272774 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-bound-sa-token\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.272875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.272008 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-trusted-ca\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.272875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.272796 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-554rp\" (UniqueName: \"kubernetes.io/projected/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-kube-api-access-554rp\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.272875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.272863 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-ca-trust-extracted\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.273104 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.272886 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-installation-pull-secrets\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.273552 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.273505 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-registry-certificates\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.273682 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.273662 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-ca-trust-extracted\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.275298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.275276 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-image-registry-private-configuration\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.275892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.275871 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-registry-tls\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.276993 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.276952 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-installation-pull-secrets\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.290896 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.290858 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-bound-sa-token\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.291305 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.291280 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-554rp\" (UniqueName: \"kubernetes.io/projected/6fa2a92c-d7cc-45c9-8424-347bf91f45d9-kube-api-access-554rp\") pod \"image-registry-79fdc76dbd-5q48r\" (UID: \"6fa2a92c-d7cc-45c9-8424-347bf91f45d9\") " pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.343499 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.343471 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:35.424125 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.424098 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-tllvs-gxphx"] Oct 30 19:31:35.427053 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:35.427026 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed516ca_1ba6_4094_8bb4_342bdb41638e.slice/crio-e892061abdc62617106bef4ee735c222ce0ec125268a5139906ec5dd1fbc3699 WatchSource:0}: Error finding container e892061abdc62617106bef4ee735c222ce0ec125268a5139906ec5dd1fbc3699: Status 404 returned error can't find the container with id e892061abdc62617106bef4ee735c222ce0ec125268a5139906ec5dd1fbc3699 Oct 30 19:31:35.522034 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.522003 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-smbbf"] Oct 30 19:31:35.523097 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:35.523057 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod216ca825_8a83_4100_ba3b_7d6294c3dadf.slice/crio-46c8f707123efbf059ddc13867f261b8f419962b908e244fe223303a00167c15 WatchSource:0}: Error finding container 46c8f707123efbf059ddc13867f261b8f419962b908e244fe223303a00167c15: Status 404 returned error can't find the container with id 46c8f707123efbf059ddc13867f261b8f419962b908e244fe223303a00167c15 Oct 30 19:31:35.523958 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.523936 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-556ccfcb78-9mh6t"] Oct 30 19:31:35.527585 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:35.527567 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf936e891_6ea7_4d4c_881f_867120559e66.slice/crio-3de3fbbf19574d2a90408990d62c81e5eb0c01ad9bb9ca5a5fd7de09870739e5 WatchSource:0}: Error finding container 3de3fbbf19574d2a90408990d62c81e5eb0c01ad9bb9ca5a5fd7de09870739e5: Status 404 returned error can't find the container with id 3de3fbbf19574d2a90408990d62c81e5eb0c01ad9bb9ca5a5fd7de09870739e5 Oct 30 19:31:35.539711 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.539652 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-xxslt"] Oct 30 19:31:35.541760 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:35.541738 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3052bdd_cd3b_4d52_998c_538138c6d060.slice/crio-977c0e60cd043b6ce4e5f7b0c383aec3d416da42ca8762ca70dd7c3565f48dbb WatchSource:0}: Error finding container 977c0e60cd043b6ce4e5f7b0c383aec3d416da42ca8762ca70dd7c3565f48dbb: Status 404 returned error can't find the container with id 977c0e60cd043b6ce4e5f7b0c383aec3d416da42ca8762ca70dd7c3565f48dbb Oct 30 19:31:35.597534 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.597457 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-smbbf" event={"ID":"216ca825-8a83-4100-ba3b-7d6294c3dadf","Type":"ContainerStarted","Data":"46c8f707123efbf059ddc13867f261b8f419962b908e244fe223303a00167c15"} Oct 30 19:31:35.598723 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.598693 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-xxslt" event={"ID":"d3052bdd-cd3b-4d52-998c-538138c6d060","Type":"ContainerStarted","Data":"977c0e60cd043b6ce4e5f7b0c383aec3d416da42ca8762ca70dd7c3565f48dbb"} Oct 30 19:31:35.600251 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.600227 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tllvs-gxphx" event={"ID":"fed516ca-1ba6-4094-8bb4-342bdb41638e","Type":"ContainerStarted","Data":"396634b039b7992a2db13e1b4ad2e2abf371ab0ecd419097f9fc15555e68d36b"} Oct 30 19:31:35.600358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.600275 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tllvs-gxphx" event={"ID":"fed516ca-1ba6-4094-8bb4-342bdb41638e","Type":"ContainerStarted","Data":"e892061abdc62617106bef4ee735c222ce0ec125268a5139906ec5dd1fbc3699"} Oct 30 19:31:35.602192 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.602169 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556ccfcb78-9mh6t" event={"ID":"f936e891-6ea7-4d4c-881f-867120559e66","Type":"ContainerStarted","Data":"3de3fbbf19574d2a90408990d62c81e5eb0c01ad9bb9ca5a5fd7de09870739e5"} Oct 30 19:31:35.621537 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.621438 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-tllvs-gxphx" podStartSLOduration=1.621422861 podStartE2EDuration="1.621422861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:35.620713961 +0000 UTC m=+15.315921922" watchObservedRunningTime="2026-10-30 19:31:35.621422861 +0000 UTC m=+15.316630820" Oct 30 19:31:35.625774 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:35.625747 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-79fdc76dbd-5q48r"] Oct 30 19:31:35.629123 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:31:35.629099 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa2a92c_d7cc_45c9_8424_347bf91f45d9.slice/crio-c289da4335749b0d01e48ec2d57f9169b63619d33ab6780a66b2669db13800ad WatchSource:0}: Error finding container c289da4335749b0d01e48ec2d57f9169b63619d33ab6780a66b2669db13800ad: Status 404 returned error can't find the container with id c289da4335749b0d01e48ec2d57f9169b63619d33ab6780a66b2669db13800ad Oct 30 19:31:36.342267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:36.342241 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j2h9z_01188e4b-c110-477a-ba88-d9f48741c0a2/dns/0.log" Oct 30 19:31:36.543491 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:36.543460 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j2h9z_01188e4b-c110-477a-ba88-d9f48741c0a2/kube-rbac-proxy/0.log" Oct 30 19:31:36.613309 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:36.613238 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" event={"ID":"6fa2a92c-d7cc-45c9-8424-347bf91f45d9","Type":"ContainerStarted","Data":"0d490cc1c06d87558bbc860b32ae9a5c128b9d81fd8c2003e8df926aaf14d17b"} Oct 30 19:31:36.613309 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:36.613282 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" event={"ID":"6fa2a92c-d7cc-45c9-8424-347bf91f45d9","Type":"ContainerStarted","Data":"c289da4335749b0d01e48ec2d57f9169b63619d33ab6780a66b2669db13800ad"} Oct 30 19:31:36.613539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:36.613315 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:36.645199 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:36.645131 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" podStartSLOduration=2.645112519 podStartE2EDuration="2.645112519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:36.642236249 +0000 UTC m=+16.337444208" watchObservedRunningTime="2026-10-30 19:31:36.645112519 +0000 UTC m=+16.340320477" Oct 30 19:31:37.343775 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:37.343745 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9g689_7fa163f8-129c-4f64-ad4b-ad3cc08bed13/dns-node-resolver/0.log" Oct 30 19:31:37.622369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:37.622293 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-smbbf" event={"ID":"216ca825-8a83-4100-ba3b-7d6294c3dadf","Type":"ContainerStarted","Data":"1163b05138cbad661ae6184db584baca9cb7022942f8e6b49fbc4478db2e663b"} Oct 30 19:31:37.622608 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:37.622582 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-smbbf" Oct 30 19:31:37.628686 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:37.628664 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-smbbf" Oct 30 19:31:37.667318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:37.667263 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-smbbf" podStartSLOduration=2.372816877 podStartE2EDuration="3.667247886s" totalImagesPullingTime="1.294431009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:37.64416663 +0000 UTC m=+17.339374579" watchObservedRunningTime="2026-10-30 19:31:37.667247886 +0000 UTC m=+17.362455846" Oct 30 19:31:37.742206 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:37.742178 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-549665644b-9xg4z_8161b0bd-5184-44fc-864e-869dc7737a55/registry/0.log" Oct 30 19:31:37.941892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:37.941813 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79fdc76dbd-5q48r_6fa2a92c-d7cc-45c9-8424-347bf91f45d9/registry/0.log" Oct 30 19:31:38.012487 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:38.012447 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:38.144279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:38.144250 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f6d8b54bc-kmdpv_731f78f9-9938-4170-a8ff-10ba7a863759/registry/0.log" Oct 30 19:31:38.164759 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:38.164719 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:38.311834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:38.311735 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:38.454656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:38.454615 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:38.570129 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:38.570052 5091 patch_prober.go:29] interesting pod/image-registry-549665644b-9xg4z container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 19:31:38.570312 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:38.570115 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-549665644b-9xg4z" podUID="8161b0bd-5184-44fc-864e-869dc7737a55" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:31:38.741546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:38.741519 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m77bc_fb5d13ef-8197-4960-b142-d8a1bdd025e2/node-ca/0.log" Oct 30 19:31:38.942604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:38.942470 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-787bddbb47-hfcvw_a8e91049-4a10-4472-922d-7b7875e7de3a/router/0.log" Oct 30 19:31:39.141343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:39.141308 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-77ms2_534aacd9-7102-4a50-8696-c50362ce87e4/serve-healthcheck-canary/0.log" Oct 30 19:31:39.633577 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:39.633536 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556ccfcb78-9mh6t" event={"ID":"f936e891-6ea7-4d4c-881f-867120559e66","Type":"ContainerStarted","Data":"4f44c1e2e9174e65162b0ee5afd8d88478b99568dfb6a701f9dca12273a02d02"} Oct 30 19:31:39.697528 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:39.697498 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:40.072683 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:40.072628 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:40.214291 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:40.214241 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:40.358671 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:40.358602 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:41.587419 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:41.587380 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-j2h9z" Oct 30 19:31:41.614585 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:41.614524 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-556ccfcb78-9mh6t" podStartSLOduration=4.057551672 podStartE2EDuration="7.614503512s" totalImagesPullingTime="3.55695184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:39.662075413 +0000 UTC m=+19.357283372" watchObservedRunningTime="2026-10-30 19:31:41.614503512 +0000 UTC m=+21.309711471" Oct 30 19:31:42.221014 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.220984 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:42.385806 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.385737 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:42.445169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.444892 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p6rpl"] Oct 30 19:31:42.452091 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.450968 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.499268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.499241 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 19:31:42.499268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.499265 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-vzztc\"" Oct 30 19:31:42.499268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.499253 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 19:31:42.523517 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.523347 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 19:31:42.523517 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.523471 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 19:31:42.536796 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.536770 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:42.544924 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.544890 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f72b3621-e387-4252-a171-17982ec3af80-root\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.545032 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.544944 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f72b3621-e387-4252-a171-17982ec3af80-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.545032 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.544975 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phsxd\" (UniqueName: \"kubernetes.io/projected/f72b3621-e387-4252-a171-17982ec3af80-kube-api-access-phsxd\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.545179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.545098 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f72b3621-e387-4252-a171-17982ec3af80-node-exporter-textfile\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.545179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.545161 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f72b3621-e387-4252-a171-17982ec3af80-sys\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.545280 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.545239 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f72b3621-e387-4252-a171-17982ec3af80-node-exporter-wtmp\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.545336 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.545301 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f72b3621-e387-4252-a171-17982ec3af80-node-exporter-accelerators-collector-config\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.545394 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.545332 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f72b3621-e387-4252-a171-17982ec3af80-node-exporter-tls\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.545394 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.545354 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f72b3621-e387-4252-a171-17982ec3af80-metrics-client-ca\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.646251 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.646209 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phsxd\" (UniqueName: \"kubernetes.io/projected/f72b3621-e387-4252-a171-17982ec3af80-kube-api-access-phsxd\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.646664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.646268 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f72b3621-e387-4252-a171-17982ec3af80-node-exporter-textfile\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.646664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.646308 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f72b3621-e387-4252-a171-17982ec3af80-sys\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.646664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.646337 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f72b3621-e387-4252-a171-17982ec3af80-node-exporter-wtmp\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.646664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.646393 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f72b3621-e387-4252-a171-17982ec3af80-node-exporter-accelerators-collector-config\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.646664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.646421 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f72b3621-e387-4252-a171-17982ec3af80-node-exporter-tls\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.646664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.646444 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f72b3621-e387-4252-a171-17982ec3af80-metrics-client-ca\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.646664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.646507 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f72b3621-e387-4252-a171-17982ec3af80-root\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.646664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.646542 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f72b3621-e387-4252-a171-17982ec3af80-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.647100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.647068 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f72b3621-e387-4252-a171-17982ec3af80-node-exporter-wtmp\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.647253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.647138 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f72b3621-e387-4252-a171-17982ec3af80-sys\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.647396 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.647366 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f72b3621-e387-4252-a171-17982ec3af80-node-exporter-textfile\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.647503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.647486 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f72b3621-e387-4252-a171-17982ec3af80-root\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.648979 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.648949 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f72b3621-e387-4252-a171-17982ec3af80-metrics-client-ca\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.649086 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.648949 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f72b3621-e387-4252-a171-17982ec3af80-node-exporter-accelerators-collector-config\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.653532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.649228 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f72b3621-e387-4252-a171-17982ec3af80-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.653532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.649616 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f72b3621-e387-4252-a171-17982ec3af80-node-exporter-tls\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.661528 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.661507 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phsxd\" (UniqueName: \"kubernetes.io/projected/f72b3621-e387-4252-a171-17982ec3af80-kube-api-access-phsxd\") pod \"node-exporter-p6rpl\" (UID: \"f72b3621-e387-4252-a171-17982ec3af80\") " pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.705345 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.705310 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:42.777131 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.777059 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p6rpl" Oct 30 19:31:42.847707 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:42.847635 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:43.014480 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:43.014432 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:43.165696 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:43.165611 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:43.305700 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:43.305665 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:43.463116 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:43.462606 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:43.610062 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:43.610019 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:43.655774 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:43.655739 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p6rpl" event={"ID":"f72b3621-e387-4252-a171-17982ec3af80","Type":"ContainerStarted","Data":"1fd25e1892ca0613b2c2de9be1aa7a4ef56a9eb0c9c04a1a444f96d1086a3df7"} Oct 30 19:31:43.791949 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:43.791911 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:43.949230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:43.949190 5091 patch_prober.go:29] interesting pod/image-registry-549665644b-9xg4z container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Oct 30 19:31:43.949397 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:43.949249 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-549665644b-9xg4z" podUID="8161b0bd-5184-44fc-864e-869dc7737a55" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:31:43.992268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:43.992238 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:44.662587 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:44.662552 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p6rpl" event={"ID":"f72b3621-e387-4252-a171-17982ec3af80","Type":"ContainerStarted","Data":"2717b3a6de4e1316380a8bd5c05149a57f6c5e87e39c25231bb04b669808c011"} Oct 30 19:31:44.928310 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:44.928282 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:31:45.239107 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:45.238446 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:45.239107 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:45.238512 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:45.243861 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:45.243836 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:45.261121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:45.261094 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:45.434324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:45.434297 5091 scope.go:122] "RemoveContainer" containerID="6ee53317f78eae384b46e7782f411073ab64bba46028c5302fdab485a4e50828" Oct 30 19:31:45.567121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:45.567099 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-fg2zw" Oct 30 19:31:45.588923 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:45.588896 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:45.670127 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:45.670089 5091 generic.go:356] "Generic (PLEG): container finished" podID="f72b3621-e387-4252-a171-17982ec3af80" containerID="2717b3a6de4e1316380a8bd5c05149a57f6c5e87e39c25231bb04b669808c011" exitCode=0 Oct 30 19:31:45.670546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:45.670446 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p6rpl" event={"ID":"f72b3621-e387-4252-a171-17982ec3af80","Type":"ContainerDied","Data":"2717b3a6de4e1316380a8bd5c05149a57f6c5e87e39c25231bb04b669808c011"} Oct 30 19:31:45.672936 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:45.672903 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-7jhj7" event={"ID":"24ac5496-c680-4b2a-8362-f12b662df997","Type":"ContainerStarted","Data":"e2a3b532a2ec7637040bf0055fa316618b9cd2d6b447ae9f1459f502f07bcd63"} Oct 30 19:31:45.677963 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:45.677922 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:31:45.737049 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:45.737018 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:45.897309 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:45.897278 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:45.980691 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:31:45.980650 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:31:46.132933 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:46.132849 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:46.286023 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:46.285984 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:46.680559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:46.680513 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p6rpl" event={"ID":"f72b3621-e387-4252-a171-17982ec3af80","Type":"ContainerStarted","Data":"585caf3f2609839709462768395a76cccc36c472222700ba19f3152fa965043e"} Oct 30 19:31:46.680996 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:46.680589 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p6rpl" event={"ID":"f72b3621-e387-4252-a171-17982ec3af80","Type":"ContainerStarted","Data":"22515786be035b51a0e32286021f8b012d637213e054226b9f4244777c7ca58c"} Oct 30 19:31:46.705591 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:46.705542 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p6rpl" podStartSLOduration=2.968516859 podStartE2EDuration="4.705529039s" totalImagesPullingTime="1.645298491s" totalInitContainerRuntime="91.713689ms" isStatefulPod=true podCreationTimestamp="2026-10-30 19:31:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:46.705116839 +0000 UTC m=+26.400324800" watchObservedRunningTime="2026-10-30 19:31:46.705529039 +0000 UTC m=+26.400736997" Oct 30 19:31:47.272183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:47.272070 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:47.465458 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:47.465421 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:47.819597 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:47.819560 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:47.964544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:47.964505 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:48.568872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:48.568841 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:31:48.808943 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:48.808908 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:49.020599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:49.020555 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:50.967694 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:50.967654 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9hgsn" Oct 30 19:31:52.713257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:52.713183 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-xxslt" event={"ID":"d3052bdd-cd3b-4d52-998c-538138c6d060","Type":"ContainerStarted","Data":"7962beeb7e3bc5021b54f1a07ae735ee63377db401e9d3296c2c032f0ece920a"} Oct 30 19:31:52.713673 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:52.713428 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-xxslt" Oct 30 19:31:52.715011 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:52.714989 5091 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-xxslt container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.31:8080/\": dial tcp 10.132.0.31:8080: connect: connection refused" start-of-body= Oct 30 19:31:52.715057 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:52.715035 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-xxslt" podUID="d3052bdd-cd3b-4d52-998c-538138c6d060" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.31:8080/\": dial tcp 10.132.0.31:8080: connect: connection refused" Oct 30 19:31:52.744281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:52.744228 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-xxslt" podStartSLOduration=1.829970044 podStartE2EDuration="18.744215727s" totalImagesPullingTime="16.914245683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:31:52.743570417 +0000 UTC m=+32.438778374" watchObservedRunningTime="2026-10-30 19:31:52.744215727 +0000 UTC m=+32.439423682" Oct 30 19:31:53.383032 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:53.382996 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-p98lt" Oct 30 19:31:53.728883 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:53.728788 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-xxslt" Oct 30 19:31:55.854630 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:55.854563 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:56.076507 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:56.076462 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:57.062624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:57.062591 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:57.207343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:57.207306 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:57.462284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:57.462197 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:57.617836 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:57.617646 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:58.632672 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:58.632639 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-79fdc76dbd-5q48r" Oct 30 19:31:58.695120 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:58.695086 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-549665644b-9xg4z"] Oct 30 19:31:58.852687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:58.852654 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:59.009086 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:59.009046 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:59.828482 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:59.828443 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:31:59.945405 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:59.945345 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" podUID="731f78f9-9938-4170-a8ff-10ba7a863759" containerName="registry" containerID="cri-o://6feab0f746fc4304bfd398967c1b80673ca06a678d7a030dbd25a2c54ce7b22d" gracePeriod=30 Oct 30 19:31:59.964916 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:31:59.964881 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:00.243849 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.243820 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:32:00.325075 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.325039 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-image-registry-private-configuration\") pod \"731f78f9-9938-4170-a8ff-10ba7a863759\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " Oct 30 19:32:00.325281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.325094 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-trusted-ca\") pod \"731f78f9-9938-4170-a8ff-10ba7a863759\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " Oct 30 19:32:00.325281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.325128 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-kube-api-access-rhvp8\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-kube-api-access-rhvp8\") pod \"731f78f9-9938-4170-a8ff-10ba7a863759\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " Oct 30 19:32:00.325281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.325183 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-bound-sa-token\") pod \"731f78f9-9938-4170-a8ff-10ba7a863759\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " Oct 30 19:32:00.325281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.325224 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls\") pod \"731f78f9-9938-4170-a8ff-10ba7a863759\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " Oct 30 19:32:00.325281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.325277 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-installation-pull-secrets\") pod \"731f78f9-9938-4170-a8ff-10ba7a863759\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " Oct 30 19:32:00.325558 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.325320 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/731f78f9-9938-4170-a8ff-10ba7a863759-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/731f78f9-9938-4170-a8ff-10ba7a863759-ca-trust-extracted\") pod \"731f78f9-9938-4170-a8ff-10ba7a863759\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " Oct 30 19:32:00.325558 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.325392 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-registry-certificates\") pod \"731f78f9-9938-4170-a8ff-10ba7a863759\" (UID: \"731f78f9-9938-4170-a8ff-10ba7a863759\") " Oct 30 19:32:00.326057 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.325986 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-registry-certificates" pod "731f78f9-9938-4170-a8ff-10ba7a863759" (UID: "731f78f9-9938-4170-a8ff-10ba7a863759"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:00.326676 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.326644 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-trusted-ca" pod "731f78f9-9938-4170-a8ff-10ba7a863759" (UID: "731f78f9-9938-4170-a8ff-10ba7a863759"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:00.328731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.328697 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-image-registry-private-configuration" pod "731f78f9-9938-4170-a8ff-10ba7a863759" (UID: "731f78f9-9938-4170-a8ff-10ba7a863759"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:00.328927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.328897 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-kube-api-access-rhvp8" pod "731f78f9-9938-4170-a8ff-10ba7a863759" (UID: "731f78f9-9938-4170-a8ff-10ba7a863759"). InnerVolumeSpecName "kube-api-access-rhvp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:00.329627 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.329567 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-installation-pull-secrets" pod "731f78f9-9938-4170-a8ff-10ba7a863759" (UID: "731f78f9-9938-4170-a8ff-10ba7a863759"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:00.331009 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.330982 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls" pod "731f78f9-9938-4170-a8ff-10ba7a863759" (UID: "731f78f9-9938-4170-a8ff-10ba7a863759"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:00.334365 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.334335 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-bound-sa-token" pod "731f78f9-9938-4170-a8ff-10ba7a863759" (UID: "731f78f9-9938-4170-a8ff-10ba7a863759"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:00.334599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.334574 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731f78f9-9938-4170-a8ff-10ba7a863759-ca-trust-extracted" pod "731f78f9-9938-4170-a8ff-10ba7a863759" (UID: "731f78f9-9938-4170-a8ff-10ba7a863759"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:00.427024 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.426988 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhvp8\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-kube-api-access-rhvp8\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:00.427024 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.427020 5091 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-bound-sa-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:00.427024 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.427030 5091 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/731f78f9-9938-4170-a8ff-10ba7a863759-registry-tls\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:00.427307 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.427040 5091 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-installation-pull-secrets\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:00.427307 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.427049 5091 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/731f78f9-9938-4170-a8ff-10ba7a863759-ca-trust-extracted\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:00.427307 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.427065 5091 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-registry-certificates\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:00.427307 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.427078 5091 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/731f78f9-9938-4170-a8ff-10ba7a863759-image-registry-private-configuration\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:00.427307 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.427092 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/731f78f9-9938-4170-a8ff-10ba7a863759-trusted-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:00.768058 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.768002 5091 generic.go:356] "Generic (PLEG): container finished" podID="731f78f9-9938-4170-a8ff-10ba7a863759" containerID="6feab0f746fc4304bfd398967c1b80673ca06a678d7a030dbd25a2c54ce7b22d" exitCode=0 Oct 30 19:32:00.768260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.768059 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" Oct 30 19:32:00.768260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.768077 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" event={"ID":"731f78f9-9938-4170-a8ff-10ba7a863759","Type":"ContainerDied","Data":"6feab0f746fc4304bfd398967c1b80673ca06a678d7a030dbd25a2c54ce7b22d"} Oct 30 19:32:00.768260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.768125 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f6d8b54bc-kmdpv" event={"ID":"731f78f9-9938-4170-a8ff-10ba7a863759","Type":"ContainerDied","Data":"81d5c0bd3ab58839f8ca49130593d58f2faaf288e79234b364ca45260b8a713a"} Oct 30 19:32:00.768260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.768174 5091 scope.go:122] "RemoveContainer" containerID="6feab0f746fc4304bfd398967c1b80673ca06a678d7a030dbd25a2c54ce7b22d" Oct 30 19:32:00.777991 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.777893 5091 scope.go:122] "RemoveContainer" containerID="6feab0f746fc4304bfd398967c1b80673ca06a678d7a030dbd25a2c54ce7b22d" Oct 30 19:32:00.778288 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:32:00.778253 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6feab0f746fc4304bfd398967c1b80673ca06a678d7a030dbd25a2c54ce7b22d\": container with ID starting with 6feab0f746fc4304bfd398967c1b80673ca06a678d7a030dbd25a2c54ce7b22d not found: ID does not exist" containerID="6feab0f746fc4304bfd398967c1b80673ca06a678d7a030dbd25a2c54ce7b22d" Oct 30 19:32:00.778393 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.778298 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6feab0f746fc4304bfd398967c1b80673ca06a678d7a030dbd25a2c54ce7b22d"} err="failed to get container status \"6feab0f746fc4304bfd398967c1b80673ca06a678d7a030dbd25a2c54ce7b22d\": rpc error: code = NotFound desc = could not find container \"6feab0f746fc4304bfd398967c1b80673ca06a678d7a030dbd25a2c54ce7b22d\": container with ID starting with 6feab0f746fc4304bfd398967c1b80673ca06a678d7a030dbd25a2c54ce7b22d not found: ID does not exist" Oct 30 19:32:00.794305 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.794277 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7f6d8b54bc-kmdpv"] Oct 30 19:32:00.798382 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:00.798360 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7f6d8b54bc-kmdpv"] Oct 30 19:32:01.124656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:01.124565 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:01.368746 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:01.368718 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:01.518887 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:01.518853 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:02.442818 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:02.442782 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="731f78f9-9938-4170-a8ff-10ba7a863759" path="/var/lib/kubelet/pods/731f78f9-9938-4170-a8ff-10ba7a863759/volumes" Oct 30 19:32:06.068854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:06.068817 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556ccfcb78-9mh6t"] Oct 30 19:32:07.204114 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.204078 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-tllvs-gxphx"] Oct 30 19:32:07.204493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.204322 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-tllvs-gxphx" podUID="fed516ca-1ba6-4094-8bb4-342bdb41638e" containerName="insights-gathering" containerID="cri-o://396634b039b7992a2db13e1b4ad2e2abf371ab0ecd419097f9fc15555e68d36b" gracePeriod=30 Oct 30 19:32:07.208038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.208014 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rgfw9-smvbs"] Oct 30 19:32:07.208237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.208203 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-rgfw9-smvbs" podUID="9e4295e8-368d-4799-b1c4-70355610378c" containerName="insights-gathering" containerID="cri-o://72d3814b00adb675b07b69585e936c4b6c61aa78a497d5ddbb3f621d675d026b" gracePeriod=30 Oct 30 19:32:07.483660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.483583 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:32:07.515988 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.515966 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:32:07.585429 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.585401 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed516ca-1ba6-4094-8bb4-342bdb41638e-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed516ca-1ba6-4094-8bb4-342bdb41638e-service-ca-bundle\") pod \"fed516ca-1ba6-4094-8bb4-342bdb41638e\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " Oct 30 19:32:07.585568 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.585446 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e4295e8-368d-4799-b1c4-70355610378c-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9e4295e8-368d-4799-b1c4-70355610378c-archives-path\") pod \"9e4295e8-368d-4799-b1c4-70355610378c\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " Oct 30 19:32:07.585568 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.585473 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e4295e8-368d-4799-b1c4-70355610378c-kube-api-access-phptb\" (UniqueName: \"kubernetes.io/projected/9e4295e8-368d-4799-b1c4-70355610378c-kube-api-access-phptb\") pod \"9e4295e8-368d-4799-b1c4-70355610378c\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " Oct 30 19:32:07.585568 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.585553 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed516ca-1ba6-4094-8bb4-342bdb41638e-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fed516ca-1ba6-4094-8bb4-342bdb41638e-archives-path\") pod \"fed516ca-1ba6-4094-8bb4-342bdb41638e\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " Oct 30 19:32:07.585740 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.585574 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed516ca-1ba6-4094-8bb4-342bdb41638e-kube-api-access-wn2cz\" (UniqueName: \"kubernetes.io/projected/fed516ca-1ba6-4094-8bb4-342bdb41638e-kube-api-access-wn2cz\") pod \"fed516ca-1ba6-4094-8bb4-342bdb41638e\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " Oct 30 19:32:07.585792 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.585731 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-trusted-ca-bundle\") pod \"9e4295e8-368d-4799-b1c4-70355610378c\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " Oct 30 19:32:07.585845 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.585787 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e4295e8-368d-4799-b1c4-70355610378c-archives-path" pod "9e4295e8-368d-4799-b1c4-70355610378c" (UID: "9e4295e8-368d-4799-b1c4-70355610378c"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:07.585845 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.585818 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-service-ca-bundle\") pod \"9e4295e8-368d-4799-b1c4-70355610378c\" (UID: \"9e4295e8-368d-4799-b1c4-70355610378c\") " Oct 30 19:32:07.585954 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.585858 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed516ca-1ba6-4094-8bb4-342bdb41638e-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed516ca-1ba6-4094-8bb4-342bdb41638e-trusted-ca-bundle\") pod \"fed516ca-1ba6-4094-8bb4-342bdb41638e\" (UID: \"fed516ca-1ba6-4094-8bb4-342bdb41638e\") " Oct 30 19:32:07.585954 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.585820 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed516ca-1ba6-4094-8bb4-342bdb41638e-service-ca-bundle" pod "fed516ca-1ba6-4094-8bb4-342bdb41638e" (UID: "fed516ca-1ba6-4094-8bb4-342bdb41638e"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:07.585954 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.585884 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed516ca-1ba6-4094-8bb4-342bdb41638e-archives-path" pod "fed516ca-1ba6-4094-8bb4-342bdb41638e" (UID: "fed516ca-1ba6-4094-8bb4-342bdb41638e"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:07.586164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.586087 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-service-ca-bundle" pod "9e4295e8-368d-4799-b1c4-70355610378c" (UID: "9e4295e8-368d-4799-b1c4-70355610378c"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:07.586252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.586139 5091 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed516ca-1ba6-4094-8bb4-342bdb41638e-service-ca-bundle\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:07.586252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.586188 5091 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9e4295e8-368d-4799-b1c4-70355610378c-archives-path\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:07.586252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.586204 5091 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/fed516ca-1ba6-4094-8bb4-342bdb41638e-archives-path\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:07.586252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.586197 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-trusted-ca-bundle" pod "9e4295e8-368d-4799-b1c4-70355610378c" (UID: "9e4295e8-368d-4799-b1c4-70355610378c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:07.586455 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.586275 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed516ca-1ba6-4094-8bb4-342bdb41638e-trusted-ca-bundle" pod "fed516ca-1ba6-4094-8bb4-342bdb41638e" (UID: "fed516ca-1ba6-4094-8bb4-342bdb41638e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:07.587785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.587765 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed516ca-1ba6-4094-8bb4-342bdb41638e-kube-api-access-wn2cz" pod "fed516ca-1ba6-4094-8bb4-342bdb41638e" (UID: "fed516ca-1ba6-4094-8bb4-342bdb41638e"). InnerVolumeSpecName "kube-api-access-wn2cz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:07.587869 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.587796 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e4295e8-368d-4799-b1c4-70355610378c-kube-api-access-phptb" pod "9e4295e8-368d-4799-b1c4-70355610378c" (UID: "9e4295e8-368d-4799-b1c4-70355610378c"). InnerVolumeSpecName "kube-api-access-phptb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:07.686642 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.686602 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn2cz\" (UniqueName: \"kubernetes.io/projected/fed516ca-1ba6-4094-8bb4-342bdb41638e-kube-api-access-wn2cz\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:07.686642 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.686636 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-trusted-ca-bundle\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:07.686642 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.686646 5091 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9e4295e8-368d-4799-b1c4-70355610378c-service-ca-bundle\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:07.686866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.686655 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fed516ca-1ba6-4094-8bb4-342bdb41638e-trusted-ca-bundle\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:07.686866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.686666 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phptb\" (UniqueName: \"kubernetes.io/projected/9e4295e8-368d-4799-b1c4-70355610378c-kube-api-access-phptb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:07.801633 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.801597 5091 generic.go:356] "Generic (PLEG): container finished" podID="9e4295e8-368d-4799-b1c4-70355610378c" containerID="72d3814b00adb675b07b69585e936c4b6c61aa78a497d5ddbb3f621d675d026b" exitCode=2 Oct 30 19:32:07.801813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.801639 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rgfw9-smvbs" Oct 30 19:32:07.801813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.801682 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rgfw9-smvbs" event={"ID":"9e4295e8-368d-4799-b1c4-70355610378c","Type":"ContainerDied","Data":"72d3814b00adb675b07b69585e936c4b6c61aa78a497d5ddbb3f621d675d026b"} Oct 30 19:32:07.801813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.801731 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rgfw9-smvbs" event={"ID":"9e4295e8-368d-4799-b1c4-70355610378c","Type":"ContainerDied","Data":"d23691ccb7c2c62b4dfb0b27a3ff430ab94d1b198b3876a7ebd5e846ed247f4c"} Oct 30 19:32:07.801813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.801755 5091 scope.go:122] "RemoveContainer" containerID="72d3814b00adb675b07b69585e936c4b6c61aa78a497d5ddbb3f621d675d026b" Oct 30 19:32:07.803008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.802966 5091 generic.go:356] "Generic (PLEG): container finished" podID="fed516ca-1ba6-4094-8bb4-342bdb41638e" containerID="396634b039b7992a2db13e1b4ad2e2abf371ab0ecd419097f9fc15555e68d36b" exitCode=2 Oct 30 19:32:07.803086 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.803019 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-tllvs-gxphx" Oct 30 19:32:07.803086 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.803033 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tllvs-gxphx" event={"ID":"fed516ca-1ba6-4094-8bb4-342bdb41638e","Type":"ContainerDied","Data":"396634b039b7992a2db13e1b4ad2e2abf371ab0ecd419097f9fc15555e68d36b"} Oct 30 19:32:07.803086 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.803063 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-tllvs-gxphx" event={"ID":"fed516ca-1ba6-4094-8bb4-342bdb41638e","Type":"ContainerDied","Data":"e892061abdc62617106bef4ee735c222ce0ec125268a5139906ec5dd1fbc3699"} Oct 30 19:32:07.813393 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.813374 5091 scope.go:122] "RemoveContainer" containerID="72d3814b00adb675b07b69585e936c4b6c61aa78a497d5ddbb3f621d675d026b" Oct 30 19:32:07.813694 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:32:07.813677 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72d3814b00adb675b07b69585e936c4b6c61aa78a497d5ddbb3f621d675d026b\": container with ID starting with 72d3814b00adb675b07b69585e936c4b6c61aa78a497d5ddbb3f621d675d026b not found: ID does not exist" containerID="72d3814b00adb675b07b69585e936c4b6c61aa78a497d5ddbb3f621d675d026b" Oct 30 19:32:07.813748 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.813702 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72d3814b00adb675b07b69585e936c4b6c61aa78a497d5ddbb3f621d675d026b"} err="failed to get container status \"72d3814b00adb675b07b69585e936c4b6c61aa78a497d5ddbb3f621d675d026b\": rpc error: code = NotFound desc = could not find container \"72d3814b00adb675b07b69585e936c4b6c61aa78a497d5ddbb3f621d675d026b\": container with ID starting with 72d3814b00adb675b07b69585e936c4b6c61aa78a497d5ddbb3f621d675d026b not found: ID does not exist" Oct 30 19:32:07.813748 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.813719 5091 scope.go:122] "RemoveContainer" containerID="396634b039b7992a2db13e1b4ad2e2abf371ab0ecd419097f9fc15555e68d36b" Oct 30 19:32:07.821463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.821446 5091 scope.go:122] "RemoveContainer" containerID="396634b039b7992a2db13e1b4ad2e2abf371ab0ecd419097f9fc15555e68d36b" Oct 30 19:32:07.821681 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:32:07.821667 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396634b039b7992a2db13e1b4ad2e2abf371ab0ecd419097f9fc15555e68d36b\": container with ID starting with 396634b039b7992a2db13e1b4ad2e2abf371ab0ecd419097f9fc15555e68d36b not found: ID does not exist" containerID="396634b039b7992a2db13e1b4ad2e2abf371ab0ecd419097f9fc15555e68d36b" Oct 30 19:32:07.821726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.821686 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396634b039b7992a2db13e1b4ad2e2abf371ab0ecd419097f9fc15555e68d36b"} err="failed to get container status \"396634b039b7992a2db13e1b4ad2e2abf371ab0ecd419097f9fc15555e68d36b\": rpc error: code = NotFound desc = could not find container \"396634b039b7992a2db13e1b4ad2e2abf371ab0ecd419097f9fc15555e68d36b\": container with ID starting with 396634b039b7992a2db13e1b4ad2e2abf371ab0ecd419097f9fc15555e68d36b not found: ID does not exist" Oct 30 19:32:07.832260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.832239 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-tllvs-gxphx"] Oct 30 19:32:07.836994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.836970 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-tllvs-gxphx"] Oct 30 19:32:07.852335 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.852313 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rgfw9-smvbs"] Oct 30 19:32:07.860035 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:07.860016 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-rgfw9-smvbs"] Oct 30 19:32:08.439263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:08.439233 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e4295e8-368d-4799-b1c4-70355610378c" path="/var/lib/kubelet/pods/9e4295e8-368d-4799-b1c4-70355610378c/volumes" Oct 30 19:32:08.439648 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:08.439606 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fed516ca-1ba6-4094-8bb4-342bdb41638e" path="/var/lib/kubelet/pods/fed516ca-1ba6-4094-8bb4-342bdb41638e/volumes" Oct 30 19:32:09.091980 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:09.091945 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:09.250994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:09.250959 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:10.156665 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:10.156619 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:10.334563 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:10.334519 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:15.402909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:15.402854 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:15.546424 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:15.546387 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:15.974385 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:32:15.974348 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:32:18.776555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:18.776512 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:18.919729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:18.919695 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:19.078962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:19.078853 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:19.660340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:19.660292 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:19.802126 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:19.802091 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:23.719941 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:23.719883 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-549665644b-9xg4z" podUID="8161b0bd-5184-44fc-864e-869dc7737a55" containerName="registry" containerID="cri-o://1c72f807c70252639d4f6fac30fadf7c4bef5f8e143c7519cf4e0c3f09bcf4a9" gracePeriod=30 Oct 30 19:32:23.871713 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:23.871686 5091 generic.go:356] "Generic (PLEG): container finished" podID="8161b0bd-5184-44fc-864e-869dc7737a55" containerID="1c72f807c70252639d4f6fac30fadf7c4bef5f8e143c7519cf4e0c3f09bcf4a9" exitCode=0 Oct 30 19:32:23.871870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:23.871737 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-549665644b-9xg4z" event={"ID":"8161b0bd-5184-44fc-864e-869dc7737a55","Type":"ContainerDied","Data":"1c72f807c70252639d4f6fac30fadf7c4bef5f8e143c7519cf4e0c3f09bcf4a9"} Oct 30 19:32:23.992076 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:23.992017 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:32:24.118873 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.118834 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-registry-certificates\") pod \"8161b0bd-5184-44fc-864e-869dc7737a55\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " Oct 30 19:32:24.119059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.118878 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls\") pod \"8161b0bd-5184-44fc-864e-869dc7737a55\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " Oct 30 19:32:24.119059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.118910 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-bound-sa-token\") pod \"8161b0bd-5184-44fc-864e-869dc7737a55\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " Oct 30 19:32:24.119059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.118939 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-trusted-ca\") pod \"8161b0bd-5184-44fc-864e-869dc7737a55\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " Oct 30 19:32:24.119059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.118999 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-kube-api-access-jvxqn\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-kube-api-access-jvxqn\") pod \"8161b0bd-5184-44fc-864e-869dc7737a55\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " Oct 30 19:32:24.119059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.119028 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8161b0bd-5184-44fc-864e-869dc7737a55-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8161b0bd-5184-44fc-864e-869dc7737a55-ca-trust-extracted\") pod \"8161b0bd-5184-44fc-864e-869dc7737a55\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " Oct 30 19:32:24.119059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.119058 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-installation-pull-secrets\") pod \"8161b0bd-5184-44fc-864e-869dc7737a55\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " Oct 30 19:32:24.119385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.119096 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-image-registry-private-configuration\") pod \"8161b0bd-5184-44fc-864e-869dc7737a55\" (UID: \"8161b0bd-5184-44fc-864e-869dc7737a55\") " Oct 30 19:32:24.119385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.119295 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-registry-certificates" pod "8161b0bd-5184-44fc-864e-869dc7737a55" (UID: "8161b0bd-5184-44fc-864e-869dc7737a55"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:24.119597 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.119572 5091 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-registry-certificates\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:24.119732 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.119578 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-trusted-ca" pod "8161b0bd-5184-44fc-864e-869dc7737a55" (UID: "8161b0bd-5184-44fc-864e-869dc7737a55"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:24.121607 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.121571 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls" pod "8161b0bd-5184-44fc-864e-869dc7737a55" (UID: "8161b0bd-5184-44fc-864e-869dc7737a55"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:24.121996 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.121964 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-image-registry-private-configuration" pod "8161b0bd-5184-44fc-864e-869dc7737a55" (UID: "8161b0bd-5184-44fc-864e-869dc7737a55"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:24.122097 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.121972 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-bound-sa-token" pod "8161b0bd-5184-44fc-864e-869dc7737a55" (UID: "8161b0bd-5184-44fc-864e-869dc7737a55"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:24.122097 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.122034 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-installation-pull-secrets" pod "8161b0bd-5184-44fc-864e-869dc7737a55" (UID: "8161b0bd-5184-44fc-864e-869dc7737a55"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:24.122217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.122116 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-kube-api-access-jvxqn" pod "8161b0bd-5184-44fc-864e-869dc7737a55" (UID: "8161b0bd-5184-44fc-864e-869dc7737a55"). InnerVolumeSpecName "kube-api-access-jvxqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:24.126978 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.126958 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8161b0bd-5184-44fc-864e-869dc7737a55-ca-trust-extracted" pod "8161b0bd-5184-44fc-864e-869dc7737a55" (UID: "8161b0bd-5184-44fc-864e-869dc7737a55"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:24.220111 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.220077 5091 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-bound-sa-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:24.220111 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.220107 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8161b0bd-5184-44fc-864e-869dc7737a55-trusted-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:24.220324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.220123 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvxqn\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-kube-api-access-jvxqn\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:24.220324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.220137 5091 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8161b0bd-5184-44fc-864e-869dc7737a55-ca-trust-extracted\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:24.220324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.220188 5091 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-installation-pull-secrets\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:24.220324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.220203 5091 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8161b0bd-5184-44fc-864e-869dc7737a55-image-registry-private-configuration\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:24.220324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.220216 5091 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8161b0bd-5184-44fc-864e-869dc7737a55-registry-tls\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:24.712399 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.712313 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:24.850885 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.850855 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:24.876371 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.876345 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-549665644b-9xg4z" Oct 30 19:32:24.876485 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.876383 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-549665644b-9xg4z" event={"ID":"8161b0bd-5184-44fc-864e-869dc7737a55","Type":"ContainerDied","Data":"fd91f9ee7b191add71a312b2a519f1da0cfd79b926e296d651b82fa36697b6b3"} Oct 30 19:32:24.876485 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.876434 5091 scope.go:122] "RemoveContainer" containerID="1c72f807c70252639d4f6fac30fadf7c4bef5f8e143c7519cf4e0c3f09bcf4a9" Oct 30 19:32:24.902828 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.902802 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-549665644b-9xg4z"] Oct 30 19:32:24.907742 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:24.907723 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-549665644b-9xg4z"] Oct 30 19:32:26.438730 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:26.438700 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8161b0bd-5184-44fc-864e-869dc7737a55" path="/var/lib/kubelet/pods/8161b0bd-5184-44fc-864e-869dc7737a55/volumes" Oct 30 19:32:30.652409 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:30.652378 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:31.089517 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.089471 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-556ccfcb78-9mh6t" podUID="f936e891-6ea7-4d4c-881f-867120559e66" containerName="console" containerID="cri-o://4f44c1e2e9174e65162b0ee5afd8d88478b99568dfb6a701f9dca12273a02d02" gracePeriod=15 Oct 30 19:32:31.131893 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.131861 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:31.279158 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.279115 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:32:31.371691 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.371667 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:32:31.479753 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.479716 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f936e891-6ea7-4d4c-881f-867120559e66-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f936e891-6ea7-4d4c-881f-867120559e66-console-serving-cert\") pod \"f936e891-6ea7-4d4c-881f-867120559e66\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " Oct 30 19:32:31.479753 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.479753 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-console-config\" (UniqueName: \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-console-config\") pod \"f936e891-6ea7-4d4c-881f-867120559e66\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " Oct 30 19:32:31.479981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.479808 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f936e891-6ea7-4d4c-881f-867120559e66-tmp\" (UniqueName: \"kubernetes.io/empty-dir/f936e891-6ea7-4d4c-881f-867120559e66-tmp\") pod \"f936e891-6ea7-4d4c-881f-867120559e66\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " Oct 30 19:32:31.479981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.479833 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-service-ca\" (UniqueName: \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-service-ca\") pod \"f936e891-6ea7-4d4c-881f-867120559e66\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " Oct 30 19:32:31.479981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.479871 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f936e891-6ea7-4d4c-881f-867120559e66-kube-api-access-69hbt\" (UniqueName: \"kubernetes.io/projected/f936e891-6ea7-4d4c-881f-867120559e66-kube-api-access-69hbt\") pod \"f936e891-6ea7-4d4c-881f-867120559e66\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " Oct 30 19:32:31.479981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.479901 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-oauth-serving-cert\") pod \"f936e891-6ea7-4d4c-881f-867120559e66\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " Oct 30 19:32:31.479981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.479931 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f936e891-6ea7-4d4c-881f-867120559e66-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f936e891-6ea7-4d4c-881f-867120559e66-console-oauth-config\") pod \"f936e891-6ea7-4d4c-881f-867120559e66\" (UID: \"f936e891-6ea7-4d4c-881f-867120559e66\") " Oct 30 19:32:31.480339 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.480297 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f936e891-6ea7-4d4c-881f-867120559e66-tmp" pod "f936e891-6ea7-4d4c-881f-867120559e66" (UID: "f936e891-6ea7-4d4c-881f-867120559e66"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:32:31.480453 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.480427 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-console-config" pod "f936e891-6ea7-4d4c-881f-867120559e66" (UID: "f936e891-6ea7-4d4c-881f-867120559e66"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:31.480517 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.480441 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-service-ca" pod "f936e891-6ea7-4d4c-881f-867120559e66" (UID: "f936e891-6ea7-4d4c-881f-867120559e66"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:31.480585 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.480518 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-oauth-serving-cert" pod "f936e891-6ea7-4d4c-881f-867120559e66" (UID: "f936e891-6ea7-4d4c-881f-867120559e66"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:32:31.482215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.482183 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f936e891-6ea7-4d4c-881f-867120559e66-kube-api-access-69hbt" pod "f936e891-6ea7-4d4c-881f-867120559e66" (UID: "f936e891-6ea7-4d4c-881f-867120559e66"). InnerVolumeSpecName "kube-api-access-69hbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:32:31.482215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.482183 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f936e891-6ea7-4d4c-881f-867120559e66-console-oauth-config" pod "f936e891-6ea7-4d4c-881f-867120559e66" (UID: "f936e891-6ea7-4d4c-881f-867120559e66"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:31.482347 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.482210 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f936e891-6ea7-4d4c-881f-867120559e66-console-serving-cert" pod "f936e891-6ea7-4d4c-881f-867120559e66" (UID: "f936e891-6ea7-4d4c-881f-867120559e66"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:32:31.581607 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.581569 5091 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f936e891-6ea7-4d4c-881f-867120559e66-console-serving-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:31.581607 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.581599 5091 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-console-config\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:31.581607 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.581614 5091 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f936e891-6ea7-4d4c-881f-867120559e66-tmp\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:31.581842 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.581626 5091 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:31.581842 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.581638 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69hbt\" (UniqueName: \"kubernetes.io/projected/f936e891-6ea7-4d4c-881f-867120559e66-kube-api-access-69hbt\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:31.581842 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.581651 5091 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f936e891-6ea7-4d4c-881f-867120559e66-oauth-serving-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:31.581842 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.581663 5091 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f936e891-6ea7-4d4c-881f-867120559e66-console-oauth-config\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:32:31.905528 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.905485 5091 generic.go:356] "Generic (PLEG): container finished" podID="f936e891-6ea7-4d4c-881f-867120559e66" containerID="4f44c1e2e9174e65162b0ee5afd8d88478b99568dfb6a701f9dca12273a02d02" exitCode=0 Oct 30 19:32:31.905913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.905551 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-556ccfcb78-9mh6t" Oct 30 19:32:31.905913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.905566 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556ccfcb78-9mh6t" event={"ID":"f936e891-6ea7-4d4c-881f-867120559e66","Type":"ContainerDied","Data":"4f44c1e2e9174e65162b0ee5afd8d88478b99568dfb6a701f9dca12273a02d02"} Oct 30 19:32:31.905913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.905645 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-556ccfcb78-9mh6t" event={"ID":"f936e891-6ea7-4d4c-881f-867120559e66","Type":"ContainerDied","Data":"3de3fbbf19574d2a90408990d62c81e5eb0c01ad9bb9ca5a5fd7de09870739e5"} Oct 30 19:32:31.905913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.905664 5091 scope.go:122] "RemoveContainer" containerID="4f44c1e2e9174e65162b0ee5afd8d88478b99568dfb6a701f9dca12273a02d02" Oct 30 19:32:31.914633 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.914617 5091 scope.go:122] "RemoveContainer" containerID="4f44c1e2e9174e65162b0ee5afd8d88478b99568dfb6a701f9dca12273a02d02" Oct 30 19:32:31.914900 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:32:31.914878 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f44c1e2e9174e65162b0ee5afd8d88478b99568dfb6a701f9dca12273a02d02\": container with ID starting with 4f44c1e2e9174e65162b0ee5afd8d88478b99568dfb6a701f9dca12273a02d02 not found: ID does not exist" containerID="4f44c1e2e9174e65162b0ee5afd8d88478b99568dfb6a701f9dca12273a02d02" Oct 30 19:32:31.914983 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.914907 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f44c1e2e9174e65162b0ee5afd8d88478b99568dfb6a701f9dca12273a02d02"} err="failed to get container status \"4f44c1e2e9174e65162b0ee5afd8d88478b99568dfb6a701f9dca12273a02d02\": rpc error: code = NotFound desc = could not find container \"4f44c1e2e9174e65162b0ee5afd8d88478b99568dfb6a701f9dca12273a02d02\": container with ID starting with 4f44c1e2e9174e65162b0ee5afd8d88478b99568dfb6a701f9dca12273a02d02 not found: ID does not exist" Oct 30 19:32:31.933910 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.933871 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-556ccfcb78-9mh6t"] Oct 30 19:32:31.936438 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:31.936417 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-556ccfcb78-9mh6t"] Oct 30 19:32:32.438572 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:32.438538 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f936e891-6ea7-4d4c-881f-867120559e66" path="/var/lib/kubelet/pods/f936e891-6ea7-4d4c-881f-867120559e66/volumes" Oct 30 19:32:35.925702 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:35.925622 5091 generic.go:356] "Generic (PLEG): container finished" podID="41c1116a-2830-4ea1-aae5-3a052f073293" containerID="6ad589fb6f687aed8c5d1c29d9454efa30fd4151d787cce480432962d2dfab3c" exitCode=0 Oct 30 19:32:35.925702 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:35.925672 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" event={"ID":"41c1116a-2830-4ea1-aae5-3a052f073293","Type":"ContainerDied","Data":"6ad589fb6f687aed8c5d1c29d9454efa30fd4151d787cce480432962d2dfab3c"} Oct 30 19:32:35.926092 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:35.925955 5091 scope.go:122] "RemoveContainer" containerID="6ad589fb6f687aed8c5d1c29d9454efa30fd4151d787cce480432962d2dfab3c" Oct 30 19:32:36.931160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:32:36.931106 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-6d8kg" event={"ID":"41c1116a-2830-4ea1-aae5-3a052f073293","Type":"ContainerStarted","Data":"c3a12b3a29428c8261722dcae752e1b42cceba0d94b016e103b1eaef49705ab5"} Oct 30 19:32:45.974342 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:32:45.974290 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:33:10.215574 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:33:10.215535 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:15.974716 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:33:15.974677 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:33:35.065922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:33:35.065828 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:35.214531 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:33:35.214492 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:33:45.974680 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:33:45.974639 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:33:53.633460 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:33:53.633420 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:10.297682 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:10.297643 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:15.975136 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:34:15.975092 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:34:40.028712 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:40.028674 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:40.361885 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:40.361805 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:42.011467 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:42.011431 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:42.302356 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:42.302267 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:43.617966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:43.617931 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:45.974663 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:34:45.974621 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:34:46.334410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.334372 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:46.392954 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.392922 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd"] Oct 30 19:34:46.393258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.393245 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fed516ca-1ba6-4094-8bb4-342bdb41638e" containerName="insights-gathering" Oct 30 19:34:46.393303 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.393259 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed516ca-1ba6-4094-8bb4-342bdb41638e" containerName="insights-gathering" Oct 30 19:34:46.393303 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.393273 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f936e891-6ea7-4d4c-881f-867120559e66" containerName="console" Oct 30 19:34:46.393303 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.393278 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f936e891-6ea7-4d4c-881f-867120559e66" containerName="console" Oct 30 19:34:46.393303 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.393288 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8161b0bd-5184-44fc-864e-869dc7737a55" containerName="registry" Oct 30 19:34:46.393303 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.393293 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8161b0bd-5184-44fc-864e-869dc7737a55" containerName="registry" Oct 30 19:34:46.393477 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.393306 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9e4295e8-368d-4799-b1c4-70355610378c" containerName="insights-gathering" Oct 30 19:34:46.393477 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.393312 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e4295e8-368d-4799-b1c4-70355610378c" containerName="insights-gathering" Oct 30 19:34:46.393477 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.393328 5091 cpu_manager.go:414] "containerMap: removing container" podUID="731f78f9-9938-4170-a8ff-10ba7a863759" containerName="registry" Oct 30 19:34:46.393477 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.393332 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="731f78f9-9938-4170-a8ff-10ba7a863759" containerName="registry" Oct 30 19:34:46.393477 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.393384 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="731f78f9-9938-4170-a8ff-10ba7a863759" containerName="registry" Oct 30 19:34:46.393477 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.393392 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed516ca-1ba6-4094-8bb4-342bdb41638e" containerName="insights-gathering" Oct 30 19:34:46.393477 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.393398 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f936e891-6ea7-4d4c-881f-867120559e66" containerName="console" Oct 30 19:34:46.393477 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.393404 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8161b0bd-5184-44fc-864e-869dc7737a55" containerName="registry" Oct 30 19:34:46.393477 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.393411 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e4295e8-368d-4799-b1c4-70355610378c" containerName="insights-gathering" Oct 30 19:34:46.395382 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.395365 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" Oct 30 19:34:46.403018 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.402996 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 19:34:46.403018 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.402997 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 19:34:46.403247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.403231 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-rzcth\"" Oct 30 19:34:46.405879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.405856 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd"] Oct 30 19:34:46.485254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.485217 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:46.529917 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.529882 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfbvr\" (UniqueName: \"kubernetes.io/projected/fdbd5bbf-f308-490c-85f3-439284d354fa-kube-api-access-dfbvr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd\" (UID: \"fdbd5bbf-f308-490c-85f3-439284d354fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" Oct 30 19:34:46.530065 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.529926 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdbd5bbf-f308-490c-85f3-439284d354fa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd\" (UID: \"fdbd5bbf-f308-490c-85f3-439284d354fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" Oct 30 19:34:46.530065 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.529953 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdbd5bbf-f308-490c-85f3-439284d354fa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd\" (UID: \"fdbd5bbf-f308-490c-85f3-439284d354fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" Oct 30 19:34:46.561100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.561068 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc"] Oct 30 19:34:46.563319 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.563294 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.575099 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.575076 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-9slkz\"" Oct 30 19:34:46.578253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.578230 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 19:34:46.579285 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.579262 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc"] Oct 30 19:34:46.631304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.631244 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfbvr\" (UniqueName: \"kubernetes.io/projected/fdbd5bbf-f308-490c-85f3-439284d354fa-kube-api-access-dfbvr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd\" (UID: \"fdbd5bbf-f308-490c-85f3-439284d354fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" Oct 30 19:34:46.631304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.631281 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdbd5bbf-f308-490c-85f3-439284d354fa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd\" (UID: \"fdbd5bbf-f308-490c-85f3-439284d354fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" Oct 30 19:34:46.631435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.631311 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdbd5bbf-f308-490c-85f3-439284d354fa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd\" (UID: \"fdbd5bbf-f308-490c-85f3-439284d354fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" Oct 30 19:34:46.631681 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.631661 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdbd5bbf-f308-490c-85f3-439284d354fa-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd\" (UID: \"fdbd5bbf-f308-490c-85f3-439284d354fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" Oct 30 19:34:46.631681 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.631673 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdbd5bbf-f308-490c-85f3-439284d354fa-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd\" (UID: \"fdbd5bbf-f308-490c-85f3-439284d354fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" Oct 30 19:34:46.646721 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.646696 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfbvr\" (UniqueName: \"kubernetes.io/projected/fdbd5bbf-f308-490c-85f3-439284d354fa-kube-api-access-dfbvr\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd\" (UID: \"fdbd5bbf-f308-490c-85f3-439284d354fa\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" Oct 30 19:34:46.687701 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.687673 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:46.696497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.696473 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s"] Oct 30 19:34:46.698807 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.698789 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.707443 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.707423 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-7ztxm\"" Oct 30 19:34:46.712090 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.712057 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" Oct 30 19:34:46.716450 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.716425 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s"] Oct 30 19:34:46.732836 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.732367 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.732836 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.732414 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.732836 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.732493 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.732836 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.732516 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.732836 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.732695 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.732836 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.732751 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx8ww\" (UniqueName: \"kubernetes.io/projected/253ed971-2d90-4358-86d1-314bdb0fad0f-kube-api-access-xx8ww\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.733167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.732840 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253ed971-2d90-4358-86d1-314bdb0fad0f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.733167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.732890 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.733167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.732919 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.833735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.833700 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.833872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.833741 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba979687-111f-44c6-a659-ad087a5fba4b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.833872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.833765 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.833872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.833789 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx8ww\" (UniqueName: \"kubernetes.io/projected/253ed971-2d90-4358-86d1-314bdb0fad0f-kube-api-access-xx8ww\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.833971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.833865 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba979687-111f-44c6-a659-ad087a5fba4b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.833971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.833933 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253ed971-2d90-4358-86d1-314bdb0fad0f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.833971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.833957 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba979687-111f-44c6-a659-ad087a5fba4b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.834085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.833984 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5hxp\" (UniqueName: \"kubernetes.io/projected/ba979687-111f-44c6-a659-ad087a5fba4b-kube-api-access-k5hxp\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.834085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.834013 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba979687-111f-44c6-a659-ad087a5fba4b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.834085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.834036 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba979687-111f-44c6-a659-ad087a5fba4b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.834271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.834160 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.834271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.834180 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.834271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.834187 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.834271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.834234 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.834271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.834256 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba979687-111f-44c6-a659-ad087a5fba4b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.834522 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.834292 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.834522 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.834353 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.834522 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.834401 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.834522 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.834428 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba979687-111f-44c6-a659-ad087a5fba4b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.834522 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.834458 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba979687-111f-44c6-a659-ad087a5fba4b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.834522 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.834505 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.834776 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.834571 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.834776 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.834621 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253ed971-2d90-4358-86d1-314bdb0fad0f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.836473 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.836449 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.836698 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.836679 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.849188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.849132 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx8ww\" (UniqueName: \"kubernetes.io/projected/253ed971-2d90-4358-86d1-314bdb0fad0f-kube-api-access-xx8ww\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.849706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.849688 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-9lwvc\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.856952 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.856929 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:46.876958 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.876939 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:46.935650 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.935622 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba979687-111f-44c6-a659-ad087a5fba4b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.935766 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.935668 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba979687-111f-44c6-a659-ad087a5fba4b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.935766 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.935687 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5hxp\" (UniqueName: \"kubernetes.io/projected/ba979687-111f-44c6-a659-ad087a5fba4b-kube-api-access-k5hxp\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.935766 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.935707 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba979687-111f-44c6-a659-ad087a5fba4b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.935766 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.935724 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba979687-111f-44c6-a659-ad087a5fba4b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.935766 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.935753 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba979687-111f-44c6-a659-ad087a5fba4b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.935994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.935786 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba979687-111f-44c6-a659-ad087a5fba4b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.935994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.935813 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba979687-111f-44c6-a659-ad087a5fba4b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.935994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.935831 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba979687-111f-44c6-a659-ad087a5fba4b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.936167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.936081 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba979687-111f-44c6-a659-ad087a5fba4b-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.936167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.936129 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba979687-111f-44c6-a659-ad087a5fba4b-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.936283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.936189 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba979687-111f-44c6-a659-ad087a5fba4b-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.936345 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.936296 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba979687-111f-44c6-a659-ad087a5fba4b-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.936659 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.936632 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba979687-111f-44c6-a659-ad087a5fba4b-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.938262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.938242 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba979687-111f-44c6-a659-ad087a5fba4b-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.938262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.938254 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba979687-111f-44c6-a659-ad087a5fba4b-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.965107 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.965084 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba979687-111f-44c6-a659-ad087a5fba4b-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.965354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.965336 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5hxp\" (UniqueName: \"kubernetes.io/projected/ba979687-111f-44c6-a659-ad087a5fba4b-kube-api-access-k5hxp\") pod \"openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s\" (UID: \"ba979687-111f-44c6-a659-ad087a5fba4b\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:46.997578 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:46.997556 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd"] Oct 30 19:34:46.998792 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:34:46.998768 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdbd5bbf_f308_490c_85f3_439284d354fa.slice/crio-c59445903fffe59488fecc811450d1acb7a1cdfd79efb19964e088d71f40367a WatchSource:0}: Error finding container c59445903fffe59488fecc811450d1acb7a1cdfd79efb19964e088d71f40367a: Status 404 returned error can't find the container with id c59445903fffe59488fecc811450d1acb7a1cdfd79efb19964e088d71f40367a Oct 30 19:34:47.013541 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:47.013521 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:47.154624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:47.154585 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc"] Oct 30 19:34:47.326559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:47.326535 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s"] Oct 30 19:34:47.328442 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:34:47.328417 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba979687_111f_44c6_a659_ad087a5fba4b.slice/crio-4aba0efdb92cec7ee11400d4eff3423a70850cc279a1772d6dbd1a15b92c3739 WatchSource:0}: Error finding container 4aba0efdb92cec7ee11400d4eff3423a70850cc279a1772d6dbd1a15b92c3739: Status 404 returned error can't find the container with id 4aba0efdb92cec7ee11400d4eff3423a70850cc279a1772d6dbd1a15b92c3739 Oct 30 19:34:47.440240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:47.440169 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" event={"ID":"fdbd5bbf-f308-490c-85f3-439284d354fa","Type":"ContainerStarted","Data":"c59445903fffe59488fecc811450d1acb7a1cdfd79efb19964e088d71f40367a"} Oct 30 19:34:47.441394 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:47.441369 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" event={"ID":"253ed971-2d90-4358-86d1-314bdb0fad0f","Type":"ContainerStarted","Data":"4aedf90faebe628350b2a9d89adaf009d45845b788e8824ee2ac24b7b1b71a79"} Oct 30 19:34:47.442475 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:47.442452 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" event={"ID":"ba979687-111f-44c6-a659-ad087a5fba4b","Type":"ContainerStarted","Data":"4aba0efdb92cec7ee11400d4eff3423a70850cc279a1772d6dbd1a15b92c3739"} Oct 30 19:34:49.524985 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:49.524947 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:34:50.856271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:50.856229 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:34:50.856630 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:50.856327 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:34:50.856630 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:50.856364 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:34:51.193796 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:51.193759 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:34:51.193900 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:51.193832 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:34:51.193900 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:51.193862 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:34:51.465227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:51.465129 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" event={"ID":"253ed971-2d90-4358-86d1-314bdb0fad0f","Type":"ContainerStarted","Data":"355d7d7b08a7b8dca0bcfa3c481dc37e62a6271f7f73a60fe12e415f7db89f9e"} Oct 30 19:34:51.466920 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:51.466892 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" event={"ID":"ba979687-111f-44c6-a659-ad087a5fba4b","Type":"ContainerStarted","Data":"7c7890407af6988a73e1a286eb789c31e162e3728cee420bd2d37ada301a8b04"} Oct 30 19:34:51.559758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:51.559699 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" podStartSLOduration=1.8645769749999999 podStartE2EDuration="5.559678167s" totalImagesPullingTime="3.695101192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:51.504007084 +0000 UTC m=+211.199215042" watchObservedRunningTime="2026-10-30 19:34:51.559678167 +0000 UTC m=+211.254886123" Oct 30 19:34:51.559966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:51.559854 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" podStartSLOduration=1.6970463649999998 podStartE2EDuration="5.559847181s" totalImagesPullingTime="3.862800816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:34:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:34:51.558845032 +0000 UTC m=+211.254052994" watchObservedRunningTime="2026-10-30 19:34:51.559847181 +0000 UTC m=+211.255055139" Oct 30 19:34:51.877786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:51.877735 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:51.883524 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:51.883496 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:52.014574 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:52.014531 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:52.019292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:52.019263 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:52.472750 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:52.472716 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:52.472946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:52.472770 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:52.473691 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:52.473663 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:34:52.473961 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:52.473934 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s" Oct 30 19:34:53.478211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:53.478110 5091 generic.go:356] "Generic (PLEG): container finished" podID="fdbd5bbf-f308-490c-85f3-439284d354fa" containerID="f68dbf4972b944883deb319f8af0ee1366db915d4c4e61a97880dcda76a8ca11" exitCode=0 Oct 30 19:34:53.478628 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:53.478201 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" event={"ID":"fdbd5bbf-f308-490c-85f3-439284d354fa","Type":"ContainerDied","Data":"f68dbf4972b944883deb319f8af0ee1366db915d4c4e61a97880dcda76a8ca11"} Oct 30 19:34:54.483692 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:54.483608 5091 generic.go:356] "Generic (PLEG): container finished" podID="fdbd5bbf-f308-490c-85f3-439284d354fa" containerID="9f82dfd612bb6aec730fbfed5645d3be103e818acadbbb2dc928fa079feb5bea" exitCode=0 Oct 30 19:34:54.484065 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:34:54.483696 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" event={"ID":"fdbd5bbf-f308-490c-85f3-439284d354fa","Type":"ContainerDied","Data":"9f82dfd612bb6aec730fbfed5645d3be103e818acadbbb2dc928fa079feb5bea"} Oct 30 19:35:00.512755 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:00.512720 5091 generic.go:356] "Generic (PLEG): container finished" podID="fdbd5bbf-f308-490c-85f3-439284d354fa" containerID="f891bad4b9f1cf10542797e8b9c4a0b6276450c0afd95a6f59340bf8fe55af42" exitCode=0 Oct 30 19:35:00.513044 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:00.512793 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" event={"ID":"fdbd5bbf-f308-490c-85f3-439284d354fa","Type":"ContainerDied","Data":"f891bad4b9f1cf10542797e8b9c4a0b6276450c0afd95a6f59340bf8fe55af42"} Oct 30 19:35:01.309434 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:01.309351 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:01.454281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:01.454238 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:01.641825 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:01.641803 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" Oct 30 19:35:01.674199 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:01.674175 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbd5bbf-f308-490c-85f3-439284d354fa-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdbd5bbf-f308-490c-85f3-439284d354fa-bundle\") pod \"fdbd5bbf-f308-490c-85f3-439284d354fa\" (UID: \"fdbd5bbf-f308-490c-85f3-439284d354fa\") " Oct 30 19:35:01.674315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:01.674301 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fdbd5bbf-f308-490c-85f3-439284d354fa-util\" (UniqueName: \"kubernetes.io/empty-dir/fdbd5bbf-f308-490c-85f3-439284d354fa-util\") pod \"fdbd5bbf-f308-490c-85f3-439284d354fa\" (UID: \"fdbd5bbf-f308-490c-85f3-439284d354fa\") " Oct 30 19:35:01.674366 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:01.674333 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdbd5bbf-f308-490c-85f3-439284d354fa-kube-api-access-dfbvr\" (UniqueName: \"kubernetes.io/projected/fdbd5bbf-f308-490c-85f3-439284d354fa-kube-api-access-dfbvr\") pod \"fdbd5bbf-f308-490c-85f3-439284d354fa\" (UID: \"fdbd5bbf-f308-490c-85f3-439284d354fa\") " Oct 30 19:35:01.674756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:01.674725 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbd5bbf-f308-490c-85f3-439284d354fa-bundle" pod "fdbd5bbf-f308-490c-85f3-439284d354fa" (UID: "fdbd5bbf-f308-490c-85f3-439284d354fa"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:01.676558 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:01.676527 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdbd5bbf-f308-490c-85f3-439284d354fa-kube-api-access-dfbvr" pod "fdbd5bbf-f308-490c-85f3-439284d354fa" (UID: "fdbd5bbf-f308-490c-85f3-439284d354fa"). InnerVolumeSpecName "kube-api-access-dfbvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:35:01.679065 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:01.679037 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fdbd5bbf-f308-490c-85f3-439284d354fa-util" pod "fdbd5bbf-f308-490c-85f3-439284d354fa" (UID: "fdbd5bbf-f308-490c-85f3-439284d354fa"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:35:01.774892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:01.774857 5091 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fdbd5bbf-f308-490c-85f3-439284d354fa-util\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:01.774892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:01.774889 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfbvr\" (UniqueName: \"kubernetes.io/projected/fdbd5bbf-f308-490c-85f3-439284d354fa-kube-api-access-dfbvr\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:01.775082 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:01.774905 5091 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fdbd5bbf-f308-490c-85f3-439284d354fa-bundle\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:35:02.522706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:02.522674 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" event={"ID":"fdbd5bbf-f308-490c-85f3-439284d354fa","Type":"ContainerDied","Data":"c59445903fffe59488fecc811450d1acb7a1cdfd79efb19964e088d71f40367a"} Oct 30 19:35:02.522838 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:02.522724 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c59445903fffe59488fecc811450d1acb7a1cdfd79efb19964e088d71f40367a" Oct 30 19:35:02.522838 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:02.522684 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6x7twd" Oct 30 19:35:10.353369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:10.353337 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:15.120981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:15.120941 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:15.286500 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:15.286462 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:15.974316 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:35:15.974278 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:35:23.791538 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:23.791499 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:24.827927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:24.827892 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:25.009123 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.009086 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:25.334413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.334381 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qxd58"] Oct 30 19:35:25.334703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.334691 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fdbd5bbf-f308-490c-85f3-439284d354fa" containerName="util" Oct 30 19:35:25.334756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.334705 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdbd5bbf-f308-490c-85f3-439284d354fa" containerName="util" Oct 30 19:35:25.334756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.334720 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fdbd5bbf-f308-490c-85f3-439284d354fa" containerName="pull" Oct 30 19:35:25.334756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.334725 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdbd5bbf-f308-490c-85f3-439284d354fa" containerName="pull" Oct 30 19:35:25.334756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.334731 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fdbd5bbf-f308-490c-85f3-439284d354fa" containerName="extract" Oct 30 19:35:25.334756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.334736 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdbd5bbf-f308-490c-85f3-439284d354fa" containerName="extract" Oct 30 19:35:25.334897 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.334793 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdbd5bbf-f308-490c-85f3-439284d354fa" containerName="extract" Oct 30 19:35:25.339420 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.339401 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qxd58" Oct 30 19:35:25.349478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.349451 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jvbww\"" Oct 30 19:35:25.350781 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.350763 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:35:25.351798 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.351783 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 19:35:25.365655 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.365632 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qxd58"] Oct 30 19:35:25.473836 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.473806 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8e2111f-2842-48a7-b8c0-2ed073fc2d6b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qxd58\" (UID: \"e8e2111f-2842-48a7-b8c0-2ed073fc2d6b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qxd58" Oct 30 19:35:25.474000 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.473869 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssf4l\" (UniqueName: \"kubernetes.io/projected/e8e2111f-2842-48a7-b8c0-2ed073fc2d6b-kube-api-access-ssf4l\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qxd58\" (UID: \"e8e2111f-2842-48a7-b8c0-2ed073fc2d6b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qxd58" Oct 30 19:35:25.574964 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.574934 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssf4l\" (UniqueName: \"kubernetes.io/projected/e8e2111f-2842-48a7-b8c0-2ed073fc2d6b-kube-api-access-ssf4l\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qxd58\" (UID: \"e8e2111f-2842-48a7-b8c0-2ed073fc2d6b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qxd58" Oct 30 19:35:25.575110 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.574989 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8e2111f-2842-48a7-b8c0-2ed073fc2d6b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qxd58\" (UID: \"e8e2111f-2842-48a7-b8c0-2ed073fc2d6b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qxd58" Oct 30 19:35:25.575335 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.575319 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8e2111f-2842-48a7-b8c0-2ed073fc2d6b-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qxd58\" (UID: \"e8e2111f-2842-48a7-b8c0-2ed073fc2d6b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qxd58" Oct 30 19:35:25.591221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.591160 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssf4l\" (UniqueName: \"kubernetes.io/projected/e8e2111f-2842-48a7-b8c0-2ed073fc2d6b-kube-api-access-ssf4l\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-qxd58\" (UID: \"e8e2111f-2842-48a7-b8c0-2ed073fc2d6b\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qxd58" Oct 30 19:35:25.651827 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.651803 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qxd58" Oct 30 19:35:25.927398 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:25.927371 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qxd58"] Oct 30 19:35:25.929267 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:35:25.929227 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e2111f_2842_48a7_b8c0_2ed073fc2d6b.slice/crio-686d4ebe38329118836b072aa715294fb933c778dee7e1012bd97c69af7da159 WatchSource:0}: Error finding container 686d4ebe38329118836b072aa715294fb933c778dee7e1012bd97c69af7da159: Status 404 returned error can't find the container with id 686d4ebe38329118836b072aa715294fb933c778dee7e1012bd97c69af7da159 Oct 30 19:35:26.616542 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:26.616505 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qxd58" event={"ID":"e8e2111f-2842-48a7-b8c0-2ed073fc2d6b","Type":"ContainerStarted","Data":"686d4ebe38329118836b072aa715294fb933c778dee7e1012bd97c69af7da159"} Oct 30 19:35:28.237276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:28.237237 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:28.394595 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:28.394557 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:28.627811 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:28.627771 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qxd58" event={"ID":"e8e2111f-2842-48a7-b8c0-2ed073fc2d6b","Type":"ContainerStarted","Data":"e7605c565241fa6df54d301b56ee1846462c79ef3e3897193991f08a1ebffb88"} Oct 30 19:35:28.664171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:28.664106 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-qxd58" podStartSLOduration=1.162147127 podStartE2EDuration="3.664094435s" totalImagesPullingTime="2.501947308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:28.661076836 +0000 UTC m=+248.356284794" watchObservedRunningTime="2026-10-30 19:35:28.664094435 +0000 UTC m=+248.359302391" Oct 30 19:35:29.409318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.409286 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:29.707215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.707133 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5"] Oct 30 19:35:29.709351 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.709316 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:29.709592 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.709573 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.731644 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.731616 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 19:35:29.732058 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.732038 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 19:35:29.732186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.732081 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 19:35:29.732186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.732107 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 19:35:29.732186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.732125 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-98qhj\"" Oct 30 19:35:29.732186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.732171 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 19:35:29.780032 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.780002 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5"] Oct 30 19:35:29.810821 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.810794 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed8f905d-46f1-4300-854f-242d71f9ede6-webhook-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.811000 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.810973 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ed8f905d-46f1-4300-854f-242d71f9ede6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.811068 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.811050 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ed8f905d-46f1-4300-854f-242d71f9ede6-manifests\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.811109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.811085 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed8f905d-46f1-4300-854f-242d71f9ede6-tmp\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.811164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.811130 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed8f905d-46f1-4300-854f-242d71f9ede6-metrics-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.811237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.811218 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9mcn\" (UniqueName: \"kubernetes.io/projected/ed8f905d-46f1-4300-854f-242d71f9ede6-kube-api-access-c9mcn\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.912163 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.912114 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ed8f905d-46f1-4300-854f-242d71f9ede6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.912325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.912170 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ed8f905d-46f1-4300-854f-242d71f9ede6-manifests\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.912325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.912188 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed8f905d-46f1-4300-854f-242d71f9ede6-tmp\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.912325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.912214 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed8f905d-46f1-4300-854f-242d71f9ede6-metrics-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.912325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.912234 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9mcn\" (UniqueName: \"kubernetes.io/projected/ed8f905d-46f1-4300-854f-242d71f9ede6-kube-api-access-c9mcn\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.912325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.912260 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed8f905d-46f1-4300-854f-242d71f9ede6-webhook-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.912626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.912608 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ed8f905d-46f1-4300-854f-242d71f9ede6-manifests\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.914677 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.914654 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed8f905d-46f1-4300-854f-242d71f9ede6-tmp\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.914788 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.914716 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ed8f905d-46f1-4300-854f-242d71f9ede6-metrics-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.914889 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.914869 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ed8f905d-46f1-4300-854f-242d71f9ede6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.914927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.914870 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ed8f905d-46f1-4300-854f-242d71f9ede6-webhook-cert\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.930406 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.930386 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9mcn\" (UniqueName: \"kubernetes.io/projected/ed8f905d-46f1-4300-854f-242d71f9ede6-kube-api-access-c9mcn\") pod \"opendatahub-operator-controller-manager-78f5b8588d-9gfr5\" (UID: \"ed8f905d-46f1-4300-854f-242d71f9ede6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:29.975105 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:29.975051 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:30.021469 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:30.021443 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:30.137523 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:30.137475 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:30.309738 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:30.309701 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:30.359536 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:35:30.359507 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8f905d_46f1_4300_854f_242d71f9ede6.slice/crio-9236ad71f9e24ed65c18622e0e473b847533a048a2eda4da060de85f4d1fd97d WatchSource:0}: Error finding container 9236ad71f9e24ed65c18622e0e473b847533a048a2eda4da060de85f4d1fd97d: Status 404 returned error can't find the container with id 9236ad71f9e24ed65c18622e0e473b847533a048a2eda4da060de85f4d1fd97d Oct 30 19:35:30.360821 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:30.360789 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5"] Oct 30 19:35:30.637516 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:30.637429 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" event={"ID":"ed8f905d-46f1-4300-854f-242d71f9ede6","Type":"ContainerStarted","Data":"9236ad71f9e24ed65c18622e0e473b847533a048a2eda4da060de85f4d1fd97d"} Oct 30 19:35:31.215128 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:31.215042 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:31.802033 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:31.801999 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:32.079634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:32.079556 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:32.387137 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:32.387047 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:32.851450 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:32.851406 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:33.656124 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:33.656090 5091 generic.go:356] "Generic (PLEG): container finished" podID="ed8f905d-46f1-4300-854f-242d71f9ede6" containerID="231ddb53a2362dd7fd47f426d3691f9bc80c4bcfe2af376ade88cb1695d96968" exitCode=0 Oct 30 19:35:33.656257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:33.656124 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" event={"ID":"ed8f905d-46f1-4300-854f-242d71f9ede6","Type":"ContainerDied","Data":"231ddb53a2362dd7fd47f426d3691f9bc80c4bcfe2af376ade88cb1695d96968"} Oct 30 19:35:34.664367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:34.664328 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" event={"ID":"ed8f905d-46f1-4300-854f-242d71f9ede6","Type":"ContainerStarted","Data":"6af2373e92eaa4e29d512e8060b33679876958ba199e820d0f10d9a34cf2114d"} Oct 30 19:35:34.664889 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:34.664641 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:34.728702 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:34.728653 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" podStartSLOduration=2.1309535840000002 podStartE2EDuration="5.728640383s" totalImagesPullingTime="3.18918091s" totalInitContainerRuntime="408.505889ms" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:29 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:34.725344158 +0000 UTC m=+254.420552116" watchObservedRunningTime="2026-10-30 19:35:34.728640383 +0000 UTC m=+254.423848333" Oct 30 19:35:35.687769 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:35.687738 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:35.879542 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:35.879506 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:36.747251 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:36.747219 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gbld7"] Oct 30 19:35:36.749699 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:36.749675 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbld7" Oct 30 19:35:36.761720 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:36.761699 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gbld7"] Oct 30 19:35:36.761951 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:36.761934 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 19:35:36.762327 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:36.762307 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-bnhvx\"" Oct 30 19:35:36.771708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:36.771678 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klvv2\" (UniqueName: \"kubernetes.io/projected/dc0d07b1-78e3-4608-9c78-a33564585b2f-kube-api-access-klvv2\") pod \"authorino-operator-7b7bb4bc77-gbld7\" (UID: \"dc0d07b1-78e3-4608-9c78-a33564585b2f\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbld7" Oct 30 19:35:36.861113 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:36.861085 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 19:35:36.872186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:36.872139 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klvv2\" (UniqueName: \"kubernetes.io/projected/dc0d07b1-78e3-4608-9c78-a33564585b2f-kube-api-access-klvv2\") pod \"authorino-operator-7b7bb4bc77-gbld7\" (UID: \"dc0d07b1-78e3-4608-9c78-a33564585b2f\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbld7" Oct 30 19:35:36.886807 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:36.886779 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klvv2\" (UniqueName: \"kubernetes.io/projected/dc0d07b1-78e3-4608-9c78-a33564585b2f-kube-api-access-klvv2\") pod \"authorino-operator-7b7bb4bc77-gbld7\" (UID: \"dc0d07b1-78e3-4608-9c78-a33564585b2f\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbld7" Oct 30 19:35:37.061531 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:37.061495 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbld7" Oct 30 19:35:37.346974 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:37.346941 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-gbld7"] Oct 30 19:35:37.348458 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:35:37.348429 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc0d07b1_78e3_4608_9c78_a33564585b2f.slice/crio-2ef83a2b04e7245cd8113f8f14d6d95ec13b7c190ecfb2cad8bb66c00cc1e93d WatchSource:0}: Error finding container 2ef83a2b04e7245cd8113f8f14d6d95ec13b7c190ecfb2cad8bb66c00cc1e93d: Status 404 returned error can't find the container with id 2ef83a2b04e7245cd8113f8f14d6d95ec13b7c190ecfb2cad8bb66c00cc1e93d Oct 30 19:35:37.368276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:37.368244 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:37.609078 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:37.608975 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:37.679253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:37.679215 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbld7" event={"ID":"dc0d07b1-78e3-4608-9c78-a33564585b2f","Type":"ContainerStarted","Data":"2ef83a2b04e7245cd8113f8f14d6d95ec13b7c190ecfb2cad8bb66c00cc1e93d"} Oct 30 19:35:38.236487 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:38.236446 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:38.913409 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:38.913258 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:39.119200 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:39.119139 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:39.294284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:39.294250 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:40.693970 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:40.693934 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbld7" event={"ID":"dc0d07b1-78e3-4608-9c78-a33564585b2f","Type":"ContainerStarted","Data":"d880b26809944f8702a275d3d0d570007f93ad89ead5bad28819f9db4af31a13"} Oct 30 19:35:40.694376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:40.694073 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbld7" Oct 30 19:35:40.720595 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:40.720526 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbld7" podStartSLOduration=2.17045459 podStartE2EDuration="4.720514555s" totalImagesPullingTime="2.550059965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:40.720137883 +0000 UTC m=+260.415345839" watchObservedRunningTime="2026-10-30 19:35:40.720514555 +0000 UTC m=+260.415722506" Oct 30 19:35:42.023140 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:42.023066 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:42.371091 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:42.371012 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:44.096618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:44.096585 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv"] Oct 30 19:35:44.100187 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:44.100169 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv" Oct 30 19:35:44.109328 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:44.109305 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hmjfz\"" Oct 30 19:35:44.122374 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:44.122353 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv"] Oct 30 19:35:44.131326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:44.131297 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9632993c-8352-4aee-ada2-c5e5fdc74cd0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4ttjv\" (UID: \"9632993c-8352-4aee-ada2-c5e5fdc74cd0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv" Oct 30 19:35:44.131432 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:44.131369 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pm5s\" (UniqueName: \"kubernetes.io/projected/9632993c-8352-4aee-ada2-c5e5fdc74cd0-kube-api-access-7pm5s\") pod \"kuadrant-operator-controller-manager-678d87766f-4ttjv\" (UID: \"9632993c-8352-4aee-ada2-c5e5fdc74cd0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv" Oct 30 19:35:44.232504 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:44.232481 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9632993c-8352-4aee-ada2-c5e5fdc74cd0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4ttjv\" (UID: \"9632993c-8352-4aee-ada2-c5e5fdc74cd0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv" Oct 30 19:35:44.232632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:44.232535 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pm5s\" (UniqueName: \"kubernetes.io/projected/9632993c-8352-4aee-ada2-c5e5fdc74cd0-kube-api-access-7pm5s\") pod \"kuadrant-operator-controller-manager-678d87766f-4ttjv\" (UID: \"9632993c-8352-4aee-ada2-c5e5fdc74cd0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv" Oct 30 19:35:44.232886 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:44.232866 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9632993c-8352-4aee-ada2-c5e5fdc74cd0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-4ttjv\" (UID: \"9632993c-8352-4aee-ada2-c5e5fdc74cd0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv" Oct 30 19:35:44.247788 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:44.247762 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pm5s\" (UniqueName: \"kubernetes.io/projected/9632993c-8352-4aee-ada2-c5e5fdc74cd0-kube-api-access-7pm5s\") pod \"kuadrant-operator-controller-manager-678d87766f-4ttjv\" (UID: \"9632993c-8352-4aee-ada2-c5e5fdc74cd0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv" Oct 30 19:35:44.412243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:44.412173 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv" Oct 30 19:35:44.423024 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:44.422998 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:44.720404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:44.720376 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv"] Oct 30 19:35:44.723219 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:35:44.723185 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9632993c_8352_4aee_ada2_c5e5fdc74cd0.slice/crio-8cb78926a709d30f279d8f4d4d8e0ed6be386dc88a1b518a6ea32e50660675e6 WatchSource:0}: Error finding container 8cb78926a709d30f279d8f4d4d8e0ed6be386dc88a1b518a6ea32e50660675e6: Status 404 returned error can't find the container with id 8cb78926a709d30f279d8f4d4d8e0ed6be386dc88a1b518a6ea32e50660675e6 Oct 30 19:35:44.944899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:44.944866 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:45.670435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:45.670406 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-78f5b8588d-9gfr5" Oct 30 19:35:45.717530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:45.717499 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv" event={"ID":"9632993c-8352-4aee-ada2-c5e5fdc74cd0","Type":"ContainerStarted","Data":"8cb78926a709d30f279d8f4d4d8e0ed6be386dc88a1b518a6ea32e50660675e6"} Oct 30 19:35:45.878412 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:45.878373 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:45.976728 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:35:45.976655 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:35:47.985633 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:47.985592 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:48.132165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:48.132063 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:49.738308 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:49.738269 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv" event={"ID":"9632993c-8352-4aee-ada2-c5e5fdc74cd0","Type":"ContainerStarted","Data":"7ce910b249470035082d534b6972a07694f0bfc3767f950836a7ce6e26a5f0ee"} Oct 30 19:35:49.738767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:49.738401 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv" Oct 30 19:35:49.778293 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:49.778242 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv" podStartSLOduration=1.658160669 podStartE2EDuration="5.778229209s" totalImagesPullingTime="4.12006854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:35:49.775025507 +0000 UTC m=+269.470233480" watchObservedRunningTime="2026-10-30 19:35:49.778229209 +0000 UTC m=+269.473437165" Oct 30 19:35:50.112232 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:50.112191 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:50.278876 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:50.278844 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:51.500099 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:51.500069 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:51.666131 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:51.666100 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:51.700673 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:51.700642 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-gbld7" Oct 30 19:35:57.238280 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:57.238238 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:57.375831 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:57.375791 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:57.605910 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:57.605875 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:57.749916 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:57.749876 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:58.047898 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:58.047873 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:58.222330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:58.222290 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:58.700498 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:58.700466 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:59.018793 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.018756 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:59.240058 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.240022 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:59.295364 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.295276 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86"] Oct 30 19:35:59.300376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.300339 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.315177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.315129 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-kgxrx\"" Oct 30 19:35:59.319365 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.319343 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86"] Oct 30 19:35:59.364597 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.364560 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/617067b0-e672-454d-a74c-f80936e85ac3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.364779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.364592 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/617067b0-e672-454d-a74c-f80936e85ac3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.364779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.364695 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/617067b0-e672-454d-a74c-f80936e85ac3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.364779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.364737 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/617067b0-e672-454d-a74c-f80936e85ac3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.364779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.364767 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/617067b0-e672-454d-a74c-f80936e85ac3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.364926 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.364830 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/617067b0-e672-454d-a74c-f80936e85ac3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.364962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.364921 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/617067b0-e672-454d-a74c-f80936e85ac3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.364995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.364962 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/617067b0-e672-454d-a74c-f80936e85ac3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.365050 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.365030 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86lfc\" (UniqueName: \"kubernetes.io/projected/617067b0-e672-454d-a74c-f80936e85ac3-kube-api-access-86lfc\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.420191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.420122 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:59.466368 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.466337 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/617067b0-e672-454d-a74c-f80936e85ac3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.466546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.466374 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/617067b0-e672-454d-a74c-f80936e85ac3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.466546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.466403 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/617067b0-e672-454d-a74c-f80936e85ac3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.466546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.466422 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/617067b0-e672-454d-a74c-f80936e85ac3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.466546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.466447 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/617067b0-e672-454d-a74c-f80936e85ac3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.466546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.466473 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/617067b0-e672-454d-a74c-f80936e85ac3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.466546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.466513 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/617067b0-e672-454d-a74c-f80936e85ac3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.466546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.466543 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/617067b0-e672-454d-a74c-f80936e85ac3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.466888 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.466589 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86lfc\" (UniqueName: \"kubernetes.io/projected/617067b0-e672-454d-a74c-f80936e85ac3-kube-api-access-86lfc\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.466888 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.466868 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/617067b0-e672-454d-a74c-f80936e85ac3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.466989 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.466911 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/617067b0-e672-454d-a74c-f80936e85ac3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.467192 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.467166 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/617067b0-e672-454d-a74c-f80936e85ac3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.467292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.467248 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/617067b0-e672-454d-a74c-f80936e85ac3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.467612 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.467579 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/617067b0-e672-454d-a74c-f80936e85ac3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.469516 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.469490 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/617067b0-e672-454d-a74c-f80936e85ac3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.469885 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.469862 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/617067b0-e672-454d-a74c-f80936e85ac3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.487116 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.487088 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86lfc\" (UniqueName: \"kubernetes.io/projected/617067b0-e672-454d-a74c-f80936e85ac3-kube-api-access-86lfc\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.487473 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.487452 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/617067b0-e672-454d-a74c-f80936e85ac3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6c786f9b57kmt86\" (UID: \"617067b0-e672-454d-a74c-f80936e85ac3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.602051 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.601951 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:35:59.613959 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.613935 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:35:59.929659 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.929635 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86"] Oct 30 19:35:59.931899 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:35:59.931873 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617067b0_e672_454d_a74c_f80936e85ac3.slice/crio-23521cdd8969134803796cb78cd6f8e92550a0dbfee9c8106af23fa1a59efb44 WatchSource:0}: Error finding container 23521cdd8969134803796cb78cd6f8e92550a0dbfee9c8106af23fa1a59efb44: Status 404 returned error can't find the container with id 23521cdd8969134803796cb78cd6f8e92550a0dbfee9c8106af23fa1a59efb44 Oct 30 19:35:59.934530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.934501 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:35:59.934618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.934560 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:35:59.934618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.934587 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:35:59.941115 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.941094 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jz2dn"] Oct 30 19:35:59.945762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.945744 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" Oct 30 19:35:59.955945 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.955923 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jz2dn"] Oct 30 19:35:59.958565 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.958545 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 19:35:59.992946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:35:59.992919 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:00.037831 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.037799 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kcgcv\"" Oct 30 19:36:00.051621 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.051590 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jz2dn"] Oct 30 19:36:00.072530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.072491 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/46213dd7-ecc4-4474-8b7c-2f416e2dd56c-config-file\") pod \"limitador-limitador-74dfcfb894-jz2dn\" (UID: \"46213dd7-ecc4-4474-8b7c-2f416e2dd56c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" Oct 30 19:36:00.072708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.072562 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fcmm\" (UniqueName: \"kubernetes.io/projected/46213dd7-ecc4-4474-8b7c-2f416e2dd56c-kube-api-access-9fcmm\") pod \"limitador-limitador-74dfcfb894-jz2dn\" (UID: \"46213dd7-ecc4-4474-8b7c-2f416e2dd56c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" Oct 30 19:36:00.149427 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.149393 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:00.174276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.174243 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/46213dd7-ecc4-4474-8b7c-2f416e2dd56c-config-file\") pod \"limitador-limitador-74dfcfb894-jz2dn\" (UID: \"46213dd7-ecc4-4474-8b7c-2f416e2dd56c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" Oct 30 19:36:00.174512 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.174494 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fcmm\" (UniqueName: \"kubernetes.io/projected/46213dd7-ecc4-4474-8b7c-2f416e2dd56c-kube-api-access-9fcmm\") pod \"limitador-limitador-74dfcfb894-jz2dn\" (UID: \"46213dd7-ecc4-4474-8b7c-2f416e2dd56c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" Oct 30 19:36:00.175027 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.175000 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/46213dd7-ecc4-4474-8b7c-2f416e2dd56c-config-file\") pod \"limitador-limitador-74dfcfb894-jz2dn\" (UID: \"46213dd7-ecc4-4474-8b7c-2f416e2dd56c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" Oct 30 19:36:00.190990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.190922 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fcmm\" (UniqueName: \"kubernetes.io/projected/46213dd7-ecc4-4474-8b7c-2f416e2dd56c-kube-api-access-9fcmm\") pod \"limitador-limitador-74dfcfb894-jz2dn\" (UID: \"46213dd7-ecc4-4474-8b7c-2f416e2dd56c\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" Oct 30 19:36:00.270874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.270838 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" Oct 30 19:36:00.551106 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.551080 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jz2dn"] Oct 30 19:36:00.553406 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:36:00.553373 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46213dd7_ecc4_4474_8b7c_2f416e2dd56c.slice/crio-5df7eb55e68d1c8dc0869868d1c1195e03d08fd3dc0164a88055dbb12c113677 WatchSource:0}: Error finding container 5df7eb55e68d1c8dc0869868d1c1195e03d08fd3dc0164a88055dbb12c113677: Status 404 returned error can't find the container with id 5df7eb55e68d1c8dc0869868d1c1195e03d08fd3dc0164a88055dbb12c113677 Oct 30 19:36:00.745159 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.745126 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-4ttjv" Oct 30 19:36:00.789294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.789249 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" event={"ID":"46213dd7-ecc4-4474-8b7c-2f416e2dd56c","Type":"ContainerStarted","Data":"5df7eb55e68d1c8dc0869868d1c1195e03d08fd3dc0164a88055dbb12c113677"} Oct 30 19:36:00.791245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.791217 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" event={"ID":"617067b0-e672-454d-a74c-f80936e85ac3","Type":"ContainerStarted","Data":"7e891ea21c7b62f44b112d762a8096c5464e81d53b8b29205e01146a3c5a1ac5"} Oct 30 19:36:00.791400 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.791264 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" event={"ID":"617067b0-e672-454d-a74c-f80936e85ac3","Type":"ContainerStarted","Data":"23521cdd8969134803796cb78cd6f8e92550a0dbfee9c8106af23fa1a59efb44"} Oct 30 19:36:00.873617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:00.873506 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" podStartSLOduration=1.873484184 podStartE2EDuration="1.873484184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:00.872206854 +0000 UTC m=+280.567414810" watchObservedRunningTime="2026-10-30 19:36:00.873484184 +0000 UTC m=+280.568692143" Oct 30 19:36:01.292840 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.292809 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:01.425438 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.425411 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89"] Oct 30 19:36:01.428838 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.428822 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:01.438276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.438227 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 19:36:01.438504 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.438478 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 19:36:01.438613 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.438513 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-qnxkv\"" Oct 30 19:36:01.445990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.445971 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 19:36:01.447548 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.447529 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89"] Oct 30 19:36:01.486133 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.486104 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjv84\" (UniqueName: \"kubernetes.io/projected/62578a06-c01e-4e9b-8580-2c82df02f25e-kube-api-access-gjv84\") pod \"kserve-module-controller-manager-7566cf5fc8-wjk89\" (UID: \"62578a06-c01e-4e9b-8580-2c82df02f25e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:01.486279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.486206 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62578a06-c01e-4e9b-8580-2c82df02f25e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-wjk89\" (UID: \"62578a06-c01e-4e9b-8580-2c82df02f25e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:01.486279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.486267 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62578a06-c01e-4e9b-8580-2c82df02f25e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-wjk89\" (UID: \"62578a06-c01e-4e9b-8580-2c82df02f25e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:01.486399 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.486355 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62578a06-c01e-4e9b-8580-2c82df02f25e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-wjk89\" (UID: \"62578a06-c01e-4e9b-8580-2c82df02f25e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:01.487136 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.487090 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:01.587846 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.587764 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62578a06-c01e-4e9b-8580-2c82df02f25e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-wjk89\" (UID: \"62578a06-c01e-4e9b-8580-2c82df02f25e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:01.588013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.587885 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62578a06-c01e-4e9b-8580-2c82df02f25e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-wjk89\" (UID: \"62578a06-c01e-4e9b-8580-2c82df02f25e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:01.588013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.587957 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62578a06-c01e-4e9b-8580-2c82df02f25e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-wjk89\" (UID: \"62578a06-c01e-4e9b-8580-2c82df02f25e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:01.588013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.587994 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjv84\" (UniqueName: \"kubernetes.io/projected/62578a06-c01e-4e9b-8580-2c82df02f25e-kube-api-access-gjv84\") pod \"kserve-module-controller-manager-7566cf5fc8-wjk89\" (UID: \"62578a06-c01e-4e9b-8580-2c82df02f25e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:01.588312 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.588290 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/62578a06-c01e-4e9b-8580-2c82df02f25e-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-wjk89\" (UID: \"62578a06-c01e-4e9b-8580-2c82df02f25e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:01.588606 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.588585 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/62578a06-c01e-4e9b-8580-2c82df02f25e-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-wjk89\" (UID: \"62578a06-c01e-4e9b-8580-2c82df02f25e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:01.589434 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.589409 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/62578a06-c01e-4e9b-8580-2c82df02f25e-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-wjk89\" (UID: \"62578a06-c01e-4e9b-8580-2c82df02f25e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:01.614642 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.614610 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:36:01.620403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.620382 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:36:01.628046 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.628020 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjv84\" (UniqueName: \"kubernetes.io/projected/62578a06-c01e-4e9b-8580-2c82df02f25e-kube-api-access-gjv84\") pod \"kserve-module-controller-manager-7566cf5fc8-wjk89\" (UID: \"62578a06-c01e-4e9b-8580-2c82df02f25e\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:01.742407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.742360 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:01.805397 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.805045 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:36:01.827034 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:01.827000 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6c786f9b57kmt86" Oct 30 19:36:02.046508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:02.046482 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89"] Oct 30 19:36:02.048229 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:36:02.048204 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62578a06_c01e_4e9b_8580_2c82df02f25e.slice/crio-b4cce3080fd4d2560b2ac777f220be748538cf42adffe6f7bf752bf4dd15055a WatchSource:0}: Error finding container b4cce3080fd4d2560b2ac777f220be748538cf42adffe6f7bf752bf4dd15055a: Status 404 returned error can't find the container with id b4cce3080fd4d2560b2ac777f220be748538cf42adffe6f7bf752bf4dd15055a Oct 30 19:36:02.812578 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:02.812473 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" event={"ID":"62578a06-c01e-4e9b-8580-2c82df02f25e","Type":"ContainerStarted","Data":"b4cce3080fd4d2560b2ac777f220be748538cf42adffe6f7bf752bf4dd15055a"} Oct 30 19:36:02.814700 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:02.814670 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" event={"ID":"46213dd7-ecc4-4474-8b7c-2f416e2dd56c","Type":"ContainerStarted","Data":"1b66368e9355f1f9f2c836383e2631ef21f0164a3162b339186019eed6bc8c14"} Oct 30 19:36:02.814943 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:02.814926 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" Oct 30 19:36:02.849845 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:02.849787 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" podStartSLOduration=2.493633479 podStartE2EDuration="3.849768119s" totalImagesPullingTime="1.35613464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:35:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:02.846350168 +0000 UTC m=+282.541558125" watchObservedRunningTime="2026-10-30 19:36:02.849768119 +0000 UTC m=+282.544976076" Oct 30 19:36:04.172347 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:04.172314 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:04.412860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:04.412822 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:04.825260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:04.825174 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" event={"ID":"62578a06-c01e-4e9b-8580-2c82df02f25e","Type":"ContainerStarted","Data":"2789d75d87f3104696cea20b09b74bb4c97705b087e5de419e005df306899d14"} Oct 30 19:36:04.825399 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:04.825281 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:04.872263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:04.872215 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" podStartSLOduration=1.353683454 podStartE2EDuration="3.872202013s" totalImagesPullingTime="2.518518559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:04.870389562 +0000 UTC m=+284.565597518" watchObservedRunningTime="2026-10-30 19:36:04.872202013 +0000 UTC m=+284.567409969" Oct 30 19:36:05.879412 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:05.879380 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:06.089505 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:06.089473 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:08.749066 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:08.749029 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:08.934382 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:08.934351 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:10.426472 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:10.426438 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:11.189249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:11.189216 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:11.333962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:11.333929 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:11.736840 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:11.736803 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:11.907030 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:11.906998 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:13.820288 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:13.820264 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" Oct 30 19:36:15.974088 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:36:15.974050 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:36:16.149359 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.149325 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jz2dn"] Oct 30 19:36:16.149535 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.149511 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" podUID="46213dd7-ecc4-4474-8b7c-2f416e2dd56c" containerName="limitador" containerID="cri-o://1b66368e9355f1f9f2c836383e2631ef21f0164a3162b339186019eed6bc8c14" gracePeriod=30 Oct 30 19:36:16.691523 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.691500 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" Oct 30 19:36:16.827292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.827262 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46213dd7-ecc4-4474-8b7c-2f416e2dd56c-config-file\" (UniqueName: \"kubernetes.io/configmap/46213dd7-ecc4-4474-8b7c-2f416e2dd56c-config-file\") pod \"46213dd7-ecc4-4474-8b7c-2f416e2dd56c\" (UID: \"46213dd7-ecc4-4474-8b7c-2f416e2dd56c\") " Oct 30 19:36:16.827480 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.827314 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46213dd7-ecc4-4474-8b7c-2f416e2dd56c-kube-api-access-9fcmm\" (UniqueName: \"kubernetes.io/projected/46213dd7-ecc4-4474-8b7c-2f416e2dd56c-kube-api-access-9fcmm\") pod \"46213dd7-ecc4-4474-8b7c-2f416e2dd56c\" (UID: \"46213dd7-ecc4-4474-8b7c-2f416e2dd56c\") " Oct 30 19:36:16.827672 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.827648 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46213dd7-ecc4-4474-8b7c-2f416e2dd56c-config-file" pod "46213dd7-ecc4-4474-8b7c-2f416e2dd56c" (UID: "46213dd7-ecc4-4474-8b7c-2f416e2dd56c"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:36:16.829565 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.829542 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46213dd7-ecc4-4474-8b7c-2f416e2dd56c-kube-api-access-9fcmm" pod "46213dd7-ecc4-4474-8b7c-2f416e2dd56c" (UID: "46213dd7-ecc4-4474-8b7c-2f416e2dd56c"). InnerVolumeSpecName "kube-api-access-9fcmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:36:16.881295 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.881269 5091 generic.go:356] "Generic (PLEG): container finished" podID="46213dd7-ecc4-4474-8b7c-2f416e2dd56c" containerID="1b66368e9355f1f9f2c836383e2631ef21f0164a3162b339186019eed6bc8c14" exitCode=0 Oct 30 19:36:16.881393 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.881320 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" Oct 30 19:36:16.881393 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.881353 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" event={"ID":"46213dd7-ecc4-4474-8b7c-2f416e2dd56c","Type":"ContainerDied","Data":"1b66368e9355f1f9f2c836383e2631ef21f0164a3162b339186019eed6bc8c14"} Oct 30 19:36:16.881514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.881408 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-jz2dn" event={"ID":"46213dd7-ecc4-4474-8b7c-2f416e2dd56c","Type":"ContainerDied","Data":"5df7eb55e68d1c8dc0869868d1c1195e03d08fd3dc0164a88055dbb12c113677"} Oct 30 19:36:16.881514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.881435 5091 scope.go:122] "RemoveContainer" containerID="1b66368e9355f1f9f2c836383e2631ef21f0164a3162b339186019eed6bc8c14" Oct 30 19:36:16.890444 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.890430 5091 scope.go:122] "RemoveContainer" containerID="1b66368e9355f1f9f2c836383e2631ef21f0164a3162b339186019eed6bc8c14" Oct 30 19:36:16.890715 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:36:16.890693 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b66368e9355f1f9f2c836383e2631ef21f0164a3162b339186019eed6bc8c14\": container with ID starting with 1b66368e9355f1f9f2c836383e2631ef21f0164a3162b339186019eed6bc8c14 not found: ID does not exist" containerID="1b66368e9355f1f9f2c836383e2631ef21f0164a3162b339186019eed6bc8c14" Oct 30 19:36:16.890758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.890723 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b66368e9355f1f9f2c836383e2631ef21f0164a3162b339186019eed6bc8c14"} err="failed to get container status \"1b66368e9355f1f9f2c836383e2631ef21f0164a3162b339186019eed6bc8c14\": rpc error: code = NotFound desc = could not find container \"1b66368e9355f1f9f2c836383e2631ef21f0164a3162b339186019eed6bc8c14\": container with ID starting with 1b66368e9355f1f9f2c836383e2631ef21f0164a3162b339186019eed6bc8c14 not found: ID does not exist" Oct 30 19:36:16.909278 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.909255 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jz2dn"] Oct 30 19:36:16.919556 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.919536 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-jz2dn"] Oct 30 19:36:16.928725 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.928707 5091 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/46213dd7-ecc4-4474-8b7c-2f416e2dd56c-config-file\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:16.928789 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:16.928728 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fcmm\" (UniqueName: \"kubernetes.io/projected/46213dd7-ecc4-4474-8b7c-2f416e2dd56c-kube-api-access-9fcmm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:36:17.309939 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:17.309902 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:17.463281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:17.463249 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:17.690681 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:17.690589 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:17.836450 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:17.836413 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:18.073372 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:18.073341 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:18.237913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:18.237883 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:18.439018 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:18.438942 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46213dd7-ecc4-4474-8b7c-2f416e2dd56c" path="/var/lib/kubelet/pods/46213dd7-ecc4-4474-8b7c-2f416e2dd56c/volumes" Oct 30 19:36:20.462846 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:20.462726 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:36:20.463333 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:20.463308 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:36:21.401990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:21.401948 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:21.558778 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:21.558745 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:22.102484 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.102448 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx"] Oct 30 19:36:22.102809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.102797 5091 cpu_manager.go:414] "containerMap: removing container" podUID="46213dd7-ecc4-4474-8b7c-2f416e2dd56c" containerName="limitador" Oct 30 19:36:22.102851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.102810 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46213dd7-ecc4-4474-8b7c-2f416e2dd56c" containerName="limitador" Oct 30 19:36:22.102900 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.102892 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="46213dd7-ecc4-4474-8b7c-2f416e2dd56c" containerName="limitador" Oct 30 19:36:22.105659 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.105638 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx" Oct 30 19:36:22.113102 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.113081 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-k24s7\"" Oct 30 19:36:22.115723 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.115483 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 19:36:22.119342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.119310 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx"] Oct 30 19:36:22.122934 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.122915 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 19:36:22.123046 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.123027 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 19:36:22.247230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.247197 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:22.275514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.275481 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjr2w\" (UniqueName: \"kubernetes.io/projected/8b14f008-1553-4840-9071-b5899529835d-kube-api-access-bjr2w\") pod \"postgres-7cdbf6dc44-2fgzx\" (UID: \"8b14f008-1553-4840-9071-b5899529835d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx" Oct 30 19:36:22.275627 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.275607 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8b14f008-1553-4840-9071-b5899529835d-data\") pod \"postgres-7cdbf6dc44-2fgzx\" (UID: \"8b14f008-1553-4840-9071-b5899529835d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx" Oct 30 19:36:22.376381 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.376279 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8b14f008-1553-4840-9071-b5899529835d-data\") pod \"postgres-7cdbf6dc44-2fgzx\" (UID: \"8b14f008-1553-4840-9071-b5899529835d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx" Oct 30 19:36:22.376529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.376403 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjr2w\" (UniqueName: \"kubernetes.io/projected/8b14f008-1553-4840-9071-b5899529835d-kube-api-access-bjr2w\") pod \"postgres-7cdbf6dc44-2fgzx\" (UID: \"8b14f008-1553-4840-9071-b5899529835d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx" Oct 30 19:36:22.376720 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.376699 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/8b14f008-1553-4840-9071-b5899529835d-data\") pod \"postgres-7cdbf6dc44-2fgzx\" (UID: \"8b14f008-1553-4840-9071-b5899529835d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx" Oct 30 19:36:22.400589 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.400539 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjr2w\" (UniqueName: \"kubernetes.io/projected/8b14f008-1553-4840-9071-b5899529835d-kube-api-access-bjr2w\") pod \"postgres-7cdbf6dc44-2fgzx\" (UID: \"8b14f008-1553-4840-9071-b5899529835d\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx" Oct 30 19:36:22.417215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.417190 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx" Oct 30 19:36:22.697611 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.697584 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx"] Oct 30 19:36:22.699121 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:36:22.699096 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b14f008_1553_4840_9071_b5899529835d.slice/crio-c8dc2fd71458b167b3685085be8d45fea4764d304897be94e5d7605ebb901c82 WatchSource:0}: Error finding container c8dc2fd71458b167b3685085be8d45fea4764d304897be94e5d7605ebb901c82: Status 404 returned error can't find the container with id c8dc2fd71458b167b3685085be8d45fea4764d304897be94e5d7605ebb901c82 Oct 30 19:36:22.700668 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.700647 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:36:22.910679 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:22.910641 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx" event={"ID":"8b14f008-1553-4840-9071-b5899529835d","Type":"ContainerStarted","Data":"c8dc2fd71458b167b3685085be8d45fea4764d304897be94e5d7605ebb901c82"} Oct 30 19:36:23.601531 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:23.601500 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:23.769543 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:23.769513 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:25.831731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:25.831702 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-wjk89" Oct 30 19:36:28.297219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:28.297181 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:28.444251 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:28.444217 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:28.948465 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:28.948427 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx" event={"ID":"8b14f008-1553-4840-9071-b5899529835d","Type":"ContainerStarted","Data":"ad8dbf504ce3b69066844196700e249146c774e87a31d6cd2e5a1dfa3708ab69"} Oct 30 19:36:28.948646 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:28.948492 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx" Oct 30 19:36:28.971064 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:28.971011 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx" podStartSLOduration=1.183929725 podStartE2EDuration="6.970993469s" totalImagesPullingTime="5.787063744s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:28.969373481 +0000 UTC m=+308.664581439" watchObservedRunningTime="2026-10-30 19:36:28.970993469 +0000 UTC m=+308.666201427" Oct 30 19:36:34.983850 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:34.983782 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-2fgzx" Oct 30 19:36:41.979523 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:41.979475 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:42.040251 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.040219 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5f44bf7884-smq9v"] Oct 30 19:36:42.049906 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.049883 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f44bf7884-smq9v" Oct 30 19:36:42.054325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.054296 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f44bf7884-smq9v"] Oct 30 19:36:42.061363 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.061340 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 19:36:42.063314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.063292 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 19:36:42.063407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.063296 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2w2d4\"" Oct 30 19:36:42.063407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.063321 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 19:36:42.124701 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.124671 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-metrics-certs\") pod \"maas-controller-5f44bf7884-smq9v\" (UID: \"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6\") " pod="opendatahub/maas-controller-5f44bf7884-smq9v" Oct 30 19:36:42.124855 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.124707 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-webhook-cert\") pod \"maas-controller-5f44bf7884-smq9v\" (UID: \"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6\") " pod="opendatahub/maas-controller-5f44bf7884-smq9v" Oct 30 19:36:42.124855 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.124832 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kszn5\" (UniqueName: \"kubernetes.io/projected/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-kube-api-access-kszn5\") pod \"maas-controller-5f44bf7884-smq9v\" (UID: \"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6\") " pod="opendatahub/maas-controller-5f44bf7884-smq9v" Oct 30 19:36:42.203460 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.203420 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:42.225539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.225513 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-webhook-cert\") pod \"maas-controller-5f44bf7884-smq9v\" (UID: \"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6\") " pod="opendatahub/maas-controller-5f44bf7884-smq9v" Oct 30 19:36:42.225634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.225571 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kszn5\" (UniqueName: \"kubernetes.io/projected/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-kube-api-access-kszn5\") pod \"maas-controller-5f44bf7884-smq9v\" (UID: \"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6\") " pod="opendatahub/maas-controller-5f44bf7884-smq9v" Oct 30 19:36:42.225703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.225633 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-metrics-certs\") pod \"maas-controller-5f44bf7884-smq9v\" (UID: \"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6\") " pod="opendatahub/maas-controller-5f44bf7884-smq9v" Oct 30 19:36:42.228204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.228178 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-metrics-certs\") pod \"maas-controller-5f44bf7884-smq9v\" (UID: \"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6\") " pod="opendatahub/maas-controller-5f44bf7884-smq9v" Oct 30 19:36:42.228307 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.228185 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-webhook-cert\") pod \"maas-controller-5f44bf7884-smq9v\" (UID: \"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6\") " pod="opendatahub/maas-controller-5f44bf7884-smq9v" Oct 30 19:36:42.239954 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.239901 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kszn5\" (UniqueName: \"kubernetes.io/projected/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-kube-api-access-kszn5\") pod \"maas-controller-5f44bf7884-smq9v\" (UID: \"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6\") " pod="opendatahub/maas-controller-5f44bf7884-smq9v" Oct 30 19:36:42.348857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.348807 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:42.365503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.365475 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f44bf7884-smq9v" Oct 30 19:36:42.639896 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:42.639859 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5f44bf7884-smq9v"] Oct 30 19:36:42.641298 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:36:42.641270 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode431e1ef_fa48_4b7b_9df0_1257a6bd33b6.slice/crio-be3cca1583f8193bfabd558870edd77b6f26b29d5cb249aafcd3fedc7b9e03be WatchSource:0}: Error finding container be3cca1583f8193bfabd558870edd77b6f26b29d5cb249aafcd3fedc7b9e03be: Status 404 returned error can't find the container with id be3cca1583f8193bfabd558870edd77b6f26b29d5cb249aafcd3fedc7b9e03be Oct 30 19:36:43.012294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:43.012261 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f44bf7884-smq9v" event={"ID":"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6","Type":"ContainerStarted","Data":"be3cca1583f8193bfabd558870edd77b6f26b29d5cb249aafcd3fedc7b9e03be"} Oct 30 19:36:45.979209 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:36:45.979165 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:36:47.706045 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:47.705973 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:47.912124 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:47.912083 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:48.037444 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:48.037407 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f44bf7884-smq9v" event={"ID":"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6","Type":"ContainerStarted","Data":"620dfd8221f7c95f8be5a0aa72c666e932f8196e2195f1dd328b7123d36733b9"} Oct 30 19:36:48.037603 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:48.037587 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5f44bf7884-smq9v" Oct 30 19:36:48.101970 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:48.101925 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5f44bf7884-smq9v" podStartSLOduration=1.3263453969999999 podStartE2EDuration="6.101914567s" totalImagesPullingTime="4.77556917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:48.101217108 +0000 UTC m=+327.796425066" watchObservedRunningTime="2026-10-30 19:36:48.101914567 +0000 UTC m=+327.797122524" Oct 30 19:36:49.038584 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:49.038545 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:49.196661 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:49.196618 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:55.751109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:55.751072 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:55.889542 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:55.885188 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n"] Oct 30 19:36:55.892778 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:55.892753 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" Oct 30 19:36:55.903376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:55.903349 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n"] Oct 30 19:36:55.907905 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:55.907880 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 19:36:55.908045 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:55.907983 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-ctgsz\"" Oct 30 19:36:55.927105 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:55.927067 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:55.961027 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:55.960993 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnm5c\" (UniqueName: \"kubernetes.io/projected/87e17fd4-767b-422a-80f0-984deabea5b6-kube-api-access-fnm5c\") pod \"llmisvc-controller-manager-869d5dcc5d-ks55n\" (UID: \"87e17fd4-767b-422a-80f0-984deabea5b6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" Oct 30 19:36:55.961216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:55.961186 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87e17fd4-767b-422a-80f0-984deabea5b6-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ks55n\" (UID: \"87e17fd4-767b-422a-80f0-984deabea5b6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" Oct 30 19:36:56.061890 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:56.061824 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87e17fd4-767b-422a-80f0-984deabea5b6-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ks55n\" (UID: \"87e17fd4-767b-422a-80f0-984deabea5b6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" Oct 30 19:36:56.061890 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:56.061874 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnm5c\" (UniqueName: \"kubernetes.io/projected/87e17fd4-767b-422a-80f0-984deabea5b6-kube-api-access-fnm5c\") pod \"llmisvc-controller-manager-869d5dcc5d-ks55n\" (UID: \"87e17fd4-767b-422a-80f0-984deabea5b6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" Oct 30 19:36:56.064169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:56.064133 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87e17fd4-767b-422a-80f0-984deabea5b6-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-ks55n\" (UID: \"87e17fd4-767b-422a-80f0-984deabea5b6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" Oct 30 19:36:56.075543 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:56.075516 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnm5c\" (UniqueName: \"kubernetes.io/projected/87e17fd4-767b-422a-80f0-984deabea5b6-kube-api-access-fnm5c\") pod \"llmisvc-controller-manager-869d5dcc5d-ks55n\" (UID: \"87e17fd4-767b-422a-80f0-984deabea5b6\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" Oct 30 19:36:56.110564 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:56.110536 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:56.208385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:56.208356 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" Oct 30 19:36:56.276041 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:56.275630 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:56.517844 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:36:56.517815 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod87e17fd4_767b_422a_80f0_984deabea5b6.slice/crio-776ba513df83cc158cae6f02bbe6b4f1a0fa8885caa310abbe49293abd40baa5 WatchSource:0}: Error finding container 776ba513df83cc158cae6f02bbe6b4f1a0fa8885caa310abbe49293abd40baa5: Status 404 returned error can't find the container with id 776ba513df83cc158cae6f02bbe6b4f1a0fa8885caa310abbe49293abd40baa5 Oct 30 19:36:56.518377 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:56.518357 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n"] Oct 30 19:36:57.081444 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:57.081402 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" event={"ID":"87e17fd4-767b-422a-80f0-984deabea5b6","Type":"ContainerStarted","Data":"776ba513df83cc158cae6f02bbe6b4f1a0fa8885caa310abbe49293abd40baa5"} Oct 30 19:36:57.346216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:57.346098 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:57.623225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:57.622828 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:57.779994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:57.779953 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:36:59.044636 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:59.044606 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5f44bf7884-smq9v" Oct 30 19:36:59.093375 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:59.093343 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" event={"ID":"87e17fd4-767b-422a-80f0-984deabea5b6","Type":"ContainerStarted","Data":"8f34f0943369e3649b4a82e1026abb9572d945bf5e85c0223afc51c468320ac6"} Oct 30 19:36:59.093516 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:59.093486 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" Oct 30 19:36:59.367702 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:59.367612 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" podStartSLOduration=2.688548406 podStartE2EDuration="4.367596927s" totalImagesPullingTime="1.679048521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:36:59.168481029 +0000 UTC m=+338.863688987" watchObservedRunningTime="2026-10-30 19:36:59.367596927 +0000 UTC m=+339.062804884" Oct 30 19:36:59.368333 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:59.368304 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f44bf7884-smq9v"] Oct 30 19:36:59.368525 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:36:59.368498 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5f44bf7884-smq9v" podUID="e431e1ef-fa48-4b7b-9df0-1257a6bd33b6" containerName="manager" containerID="cri-o://620dfd8221f7c95f8be5a0aa72c666e932f8196e2195f1dd328b7123d36733b9" gracePeriod=30 Oct 30 19:37:00.101720 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:00.101670 5091 generic.go:356] "Generic (PLEG): container finished" podID="e431e1ef-fa48-4b7b-9df0-1257a6bd33b6" containerID="620dfd8221f7c95f8be5a0aa72c666e932f8196e2195f1dd328b7123d36733b9" exitCode=0 Oct 30 19:37:00.102178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:00.101725 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f44bf7884-smq9v" event={"ID":"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6","Type":"ContainerDied","Data":"620dfd8221f7c95f8be5a0aa72c666e932f8196e2195f1dd328b7123d36733b9"} Oct 30 19:37:00.163545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:00.163521 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f44bf7884-smq9v" Oct 30 19:37:00.200439 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:00.200412 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-kube-api-access-kszn5\" (UniqueName: \"kubernetes.io/projected/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-kube-api-access-kszn5\") pod \"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6\" (UID: \"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6\") " Oct 30 19:37:00.200596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:00.200461 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-metrics-certs\") pod \"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6\" (UID: \"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6\") " Oct 30 19:37:00.200596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:00.200516 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-webhook-cert\" (UniqueName: \"kubernetes.io/secret/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-webhook-cert\") pod \"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6\" (UID: \"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6\") " Oct 30 19:37:00.202736 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:00.202705 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-webhook-cert" pod "e431e1ef-fa48-4b7b-9df0-1257a6bd33b6" (UID: "e431e1ef-fa48-4b7b-9df0-1257a6bd33b6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:00.203013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:00.202991 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-metrics-certs" pod "e431e1ef-fa48-4b7b-9df0-1257a6bd33b6" (UID: "e431e1ef-fa48-4b7b-9df0-1257a6bd33b6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:37:00.205244 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:00.205227 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-kube-api-access-kszn5" pod "e431e1ef-fa48-4b7b-9df0-1257a6bd33b6" (UID: "e431e1ef-fa48-4b7b-9df0-1257a6bd33b6"). InnerVolumeSpecName "kube-api-access-kszn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:37:00.302189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:00.302094 5091 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-metrics-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:00.302189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:00.302123 5091 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-webhook-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:00.302189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:00.302133 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kszn5\" (UniqueName: \"kubernetes.io/projected/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6-kube-api-access-kszn5\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:37:01.107294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:01.107260 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5f44bf7884-smq9v" Oct 30 19:37:01.107719 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:01.107295 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5f44bf7884-smq9v" event={"ID":"e431e1ef-fa48-4b7b-9df0-1257a6bd33b6","Type":"ContainerDied","Data":"be3cca1583f8193bfabd558870edd77b6f26b29d5cb249aafcd3fedc7b9e03be"} Oct 30 19:37:01.107719 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:01.107357 5091 scope.go:122] "RemoveContainer" containerID="620dfd8221f7c95f8be5a0aa72c666e932f8196e2195f1dd328b7123d36733b9" Oct 30 19:37:01.130469 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:01.130443 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5f44bf7884-smq9v"] Oct 30 19:37:01.135439 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:01.135418 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5f44bf7884-smq9v"] Oct 30 19:37:02.439084 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:02.439051 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e431e1ef-fa48-4b7b-9df0-1257a6bd33b6" path="/var/lib/kubelet/pods/e431e1ef-fa48-4b7b-9df0-1257a6bd33b6/volumes" Oct 30 19:37:07.325607 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.325577 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:07.529213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.529179 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:07.729744 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.729669 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29"] Oct 30 19:37:07.730069 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.730057 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e431e1ef-fa48-4b7b-9df0-1257a6bd33b6" containerName="manager" Oct 30 19:37:07.730115 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.730071 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e431e1ef-fa48-4b7b-9df0-1257a6bd33b6" containerName="manager" Oct 30 19:37:07.730165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.730134 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e431e1ef-fa48-4b7b-9df0-1257a6bd33b6" containerName="manager" Oct 30 19:37:07.734625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.734593 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:07.743802 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.743775 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 19:37:07.746352 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.746330 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 19:37:07.746478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.746393 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 19:37:07.746478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.746446 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 19:37:07.746478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.746458 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-prwwx\"" Oct 30 19:37:07.746631 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.746500 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 19:37:07.748702 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.748678 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29"] Oct 30 19:37:07.871660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.871622 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cb0d04a-afe3-43ed-918a-448bb6b0a196-cert\") pod \"jobset-controller-manager-fd8cbc9d9-m5t29\" (UID: \"0cb0d04a-afe3-43ed-918a-448bb6b0a196\") " pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:07.871800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.871764 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55wcv\" (UniqueName: \"kubernetes.io/projected/0cb0d04a-afe3-43ed-918a-448bb6b0a196-kube-api-access-55wcv\") pod \"jobset-controller-manager-fd8cbc9d9-m5t29\" (UID: \"0cb0d04a-afe3-43ed-918a-448bb6b0a196\") " pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:07.871899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.871879 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb0d04a-afe3-43ed-918a-448bb6b0a196-metrics-certs\") pod \"jobset-controller-manager-fd8cbc9d9-m5t29\" (UID: \"0cb0d04a-afe3-43ed-918a-448bb6b0a196\") " pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:07.871968 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.871910 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0cb0d04a-afe3-43ed-918a-448bb6b0a196-manager-config\") pod \"jobset-controller-manager-fd8cbc9d9-m5t29\" (UID: \"0cb0d04a-afe3-43ed-918a-448bb6b0a196\") " pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:07.972355 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.972317 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb0d04a-afe3-43ed-918a-448bb6b0a196-metrics-certs\") pod \"jobset-controller-manager-fd8cbc9d9-m5t29\" (UID: \"0cb0d04a-afe3-43ed-918a-448bb6b0a196\") " pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:07.972355 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.972352 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0cb0d04a-afe3-43ed-918a-448bb6b0a196-manager-config\") pod \"jobset-controller-manager-fd8cbc9d9-m5t29\" (UID: \"0cb0d04a-afe3-43ed-918a-448bb6b0a196\") " pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:07.972608 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.972420 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cb0d04a-afe3-43ed-918a-448bb6b0a196-cert\") pod \"jobset-controller-manager-fd8cbc9d9-m5t29\" (UID: \"0cb0d04a-afe3-43ed-918a-448bb6b0a196\") " pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:07.972608 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.972454 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55wcv\" (UniqueName: \"kubernetes.io/projected/0cb0d04a-afe3-43ed-918a-448bb6b0a196-kube-api-access-55wcv\") pod \"jobset-controller-manager-fd8cbc9d9-m5t29\" (UID: \"0cb0d04a-afe3-43ed-918a-448bb6b0a196\") " pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:07.973161 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.973124 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0cb0d04a-afe3-43ed-918a-448bb6b0a196-manager-config\") pod \"jobset-controller-manager-fd8cbc9d9-m5t29\" (UID: \"0cb0d04a-afe3-43ed-918a-448bb6b0a196\") " pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:07.974801 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.974777 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cb0d04a-afe3-43ed-918a-448bb6b0a196-metrics-certs\") pod \"jobset-controller-manager-fd8cbc9d9-m5t29\" (UID: \"0cb0d04a-afe3-43ed-918a-448bb6b0a196\") " pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:07.974898 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.974828 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cb0d04a-afe3-43ed-918a-448bb6b0a196-cert\") pod \"jobset-controller-manager-fd8cbc9d9-m5t29\" (UID: \"0cb0d04a-afe3-43ed-918a-448bb6b0a196\") " pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:07.990190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:07.990118 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55wcv\" (UniqueName: \"kubernetes.io/projected/0cb0d04a-afe3-43ed-918a-448bb6b0a196-kube-api-access-55wcv\") pod \"jobset-controller-manager-fd8cbc9d9-m5t29\" (UID: \"0cb0d04a-afe3-43ed-918a-448bb6b0a196\") " pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:08.049225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:08.049201 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:08.324622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:08.324596 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29"] Oct 30 19:37:08.326339 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:37:08.326313 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cb0d04a_afe3_43ed_918a_448bb6b0a196.slice/crio-6fda7427b14d77f182585abf9e946076d30bb5a4401692a0c9b1610b4875586c WatchSource:0}: Error finding container 6fda7427b14d77f182585abf9e946076d30bb5a4401692a0c9b1610b4875586c: Status 404 returned error can't find the container with id 6fda7427b14d77f182585abf9e946076d30bb5a4401692a0c9b1610b4875586c Oct 30 19:37:09.146883 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:09.146849 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" event={"ID":"0cb0d04a-afe3-43ed-918a-448bb6b0a196","Type":"ContainerStarted","Data":"6fda7427b14d77f182585abf9e946076d30bb5a4401692a0c9b1610b4875586c"} Oct 30 19:37:10.449195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:10.449141 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:12.164376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:12.164341 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" event={"ID":"0cb0d04a-afe3-43ed-918a-448bb6b0a196","Type":"ContainerStarted","Data":"1ced2c6be7a741ed9d70107717aaa9c3a5aa15d0241614e643db4e807b81c610"} Oct 30 19:37:12.164866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:12.164440 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:12.188029 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:12.187985 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" podStartSLOduration=1.976386465 podStartE2EDuration="5.187973615s" totalImagesPullingTime="3.21158715s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:37:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:37:12.186784487 +0000 UTC m=+351.881992444" watchObservedRunningTime="2026-10-30 19:37:12.187973615 +0000 UTC m=+351.883181571" Oct 30 19:37:15.975105 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:37:15.975065 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:37:19.043823 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:19.043785 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:19.206238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:19.206198 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:20.403061 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:20.403027 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:20.560012 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:20.559967 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:23.174195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:23.174158 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-fd8cbc9d9-m5t29" Oct 30 19:37:27.501884 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:27.501846 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:27.656438 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:27.656397 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:30.104743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:30.104711 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" Oct 30 19:37:30.251313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:30.251283 5091 generic.go:356] "Generic (PLEG): container finished" podID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerID="355d7d7b08a7b8dca0bcfa3c481dc37e62a6271f7f73a60fe12e415f7db89f9e" exitCode=0 Oct 30 19:37:30.251524 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:30.251341 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" event={"ID":"253ed971-2d90-4358-86d1-314bdb0fad0f","Type":"ContainerDied","Data":"355d7d7b08a7b8dca0bcfa3c481dc37e62a6271f7f73a60fe12e415f7db89f9e"} Oct 30 19:37:30.251738 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:30.251717 5091 scope.go:122] "RemoveContainer" containerID="355d7d7b08a7b8dca0bcfa3c481dc37e62a6271f7f73a60fe12e415f7db89f9e" Oct 30 19:37:30.252814 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:30.252784 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:37:30.252893 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:30.252856 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:37:30.252928 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:30.252893 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:37:31.275858 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:31.275825 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" event={"ID":"253ed971-2d90-4358-86d1-314bdb0fad0f","Type":"ContainerStarted","Data":"057048b586b7d00f31eb1fab4ac107bd7c936d13502ad3955fea4424fa94805e"} Oct 30 19:37:31.877155 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:31.877112 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:37:31.877356 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:31.877319 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 19:37:31.877432 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:31.877363 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 19:37:32.877710 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:32.877672 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 19:37:32.878136 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:32.877744 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 19:37:32.966010 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:32.965973 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:33.112652 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:33.112615 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:37:33.877661 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:33.877625 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 19:37:33.877932 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:33.877693 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 19:37:34.877719 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:34.877684 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 19:37:34.877910 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:34.877738 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 19:37:35.877956 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:35.877926 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body= Oct 30 19:37:35.878426 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:35.877983 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" Oct 30 19:37:36.881269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:36.881238 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:37:36.881644 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:36.881485 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:37:36.882295 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:37:36.882275 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:37:45.974806 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:37:45.974770 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:38:10.512602 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:10.512517 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:15.974329 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:38:15.974287 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:38:30.661623 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:30.661590 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n"] Oct 30 19:38:30.662108 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:30.661786 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" podUID="87e17fd4-767b-422a-80f0-984deabea5b6" containerName="manager" containerID="cri-o://8f34f0943369e3649b4a82e1026abb9572d945bf5e85c0223afc51c468320ac6" gracePeriod=30 Oct 30 19:38:30.914306 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:30.914248 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" Oct 30 19:38:31.065685 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:31.065651 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87e17fd4-767b-422a-80f0-984deabea5b6-cert\" (UniqueName: \"kubernetes.io/secret/87e17fd4-767b-422a-80f0-984deabea5b6-cert\") pod \"87e17fd4-767b-422a-80f0-984deabea5b6\" (UID: \"87e17fd4-767b-422a-80f0-984deabea5b6\") " Oct 30 19:38:31.065865 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:31.065750 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e17fd4-767b-422a-80f0-984deabea5b6-kube-api-access-fnm5c\" (UniqueName: \"kubernetes.io/projected/87e17fd4-767b-422a-80f0-984deabea5b6-kube-api-access-fnm5c\") pod \"87e17fd4-767b-422a-80f0-984deabea5b6\" (UID: \"87e17fd4-767b-422a-80f0-984deabea5b6\") " Oct 30 19:38:31.067749 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:31.067717 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87e17fd4-767b-422a-80f0-984deabea5b6-cert" pod "87e17fd4-767b-422a-80f0-984deabea5b6" (UID: "87e17fd4-767b-422a-80f0-984deabea5b6"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:38:31.067854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:31.067804 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e17fd4-767b-422a-80f0-984deabea5b6-kube-api-access-fnm5c" pod "87e17fd4-767b-422a-80f0-984deabea5b6" (UID: "87e17fd4-767b-422a-80f0-984deabea5b6"). InnerVolumeSpecName "kube-api-access-fnm5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:38:31.166410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:31.166328 5091 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87e17fd4-767b-422a-80f0-984deabea5b6-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:31.166410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:31.166361 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnm5c\" (UniqueName: \"kubernetes.io/projected/87e17fd4-767b-422a-80f0-984deabea5b6-kube-api-access-fnm5c\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:38:31.552817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:31.552780 5091 generic.go:356] "Generic (PLEG): container finished" podID="87e17fd4-767b-422a-80f0-984deabea5b6" containerID="8f34f0943369e3649b4a82e1026abb9572d945bf5e85c0223afc51c468320ac6" exitCode=0 Oct 30 19:38:31.553024 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:31.552826 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" Oct 30 19:38:31.553024 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:31.552873 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" event={"ID":"87e17fd4-767b-422a-80f0-984deabea5b6","Type":"ContainerDied","Data":"8f34f0943369e3649b4a82e1026abb9572d945bf5e85c0223afc51c468320ac6"} Oct 30 19:38:31.553024 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:31.552934 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n" event={"ID":"87e17fd4-767b-422a-80f0-984deabea5b6","Type":"ContainerDied","Data":"776ba513df83cc158cae6f02bbe6b4f1a0fa8885caa310abbe49293abd40baa5"} Oct 30 19:38:31.553024 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:31.552965 5091 scope.go:122] "RemoveContainer" containerID="8f34f0943369e3649b4a82e1026abb9572d945bf5e85c0223afc51c468320ac6" Oct 30 19:38:31.561968 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:31.561952 5091 scope.go:122] "RemoveContainer" containerID="8f34f0943369e3649b4a82e1026abb9572d945bf5e85c0223afc51c468320ac6" Oct 30 19:38:31.562265 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:38:31.562243 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f34f0943369e3649b4a82e1026abb9572d945bf5e85c0223afc51c468320ac6\": container with ID starting with 8f34f0943369e3649b4a82e1026abb9572d945bf5e85c0223afc51c468320ac6 not found: ID does not exist" containerID="8f34f0943369e3649b4a82e1026abb9572d945bf5e85c0223afc51c468320ac6" Oct 30 19:38:31.562342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:31.562272 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f34f0943369e3649b4a82e1026abb9572d945bf5e85c0223afc51c468320ac6"} err="failed to get container status \"8f34f0943369e3649b4a82e1026abb9572d945bf5e85c0223afc51c468320ac6\": rpc error: code = NotFound desc = could not find container \"8f34f0943369e3649b4a82e1026abb9572d945bf5e85c0223afc51c468320ac6\": container with ID starting with 8f34f0943369e3649b4a82e1026abb9572d945bf5e85c0223afc51c468320ac6 not found: ID does not exist" Oct 30 19:38:31.581589 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:31.581566 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n"] Oct 30 19:38:31.592510 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:31.592485 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-ks55n"] Oct 30 19:38:32.439921 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:32.439889 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e17fd4-767b-422a-80f0-984deabea5b6" path="/var/lib/kubelet/pods/87e17fd4-767b-422a-80f0-984deabea5b6/volumes" Oct 30 19:38:32.923069 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:32.923033 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:33.070650 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:33.070613 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:37.782620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:37.782584 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:38.060831 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:38:38.060732 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:38:45.975135 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:38:45.975091 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:39:02.840310 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:02.840267 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:03.236786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:03.236702 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:04.542364 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:04.542324 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:04.711940 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:04.711907 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:08.682134 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:08.682101 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:10.571469 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:10.571428 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:11.327008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.326973 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw"] Oct 30 19:39:11.327404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.327386 5091 cpu_manager.go:414] "containerMap: removing container" podUID="87e17fd4-767b-422a-80f0-984deabea5b6" containerName="manager" Oct 30 19:39:11.327404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.327402 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e17fd4-767b-422a-80f0-984deabea5b6" containerName="manager" Oct 30 19:39:11.327565 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.327474 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e17fd4-767b-422a-80f0-984deabea5b6" containerName="manager" Oct 30 19:39:11.330595 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.330570 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.340433 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.340407 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 19:39:11.340579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.340456 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 19:39:11.340675 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.340646 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 19:39:11.343708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.343686 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw"] Oct 30 19:39:11.421140 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.421094 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/540a152c-2be5-4253-b272-586b34835043-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.421341 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.421139 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/540a152c-2be5-4253-b272-586b34835043-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.421341 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.421278 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/540a152c-2be5-4253-b272-586b34835043-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.421341 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.421314 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/540a152c-2be5-4253-b272-586b34835043-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.421503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.421376 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/540a152c-2be5-4253-b272-586b34835043-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.421503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.421396 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-224gh\" (UniqueName: \"kubernetes.io/projected/540a152c-2be5-4253-b272-586b34835043-kube-api-access-224gh\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.430804 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.430772 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:11.441404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.441375 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-qx8jh\"" Oct 30 19:39:11.522234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.522193 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/540a152c-2be5-4253-b272-586b34835043-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.522425 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.522257 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/540a152c-2be5-4253-b272-586b34835043-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.522425 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.522307 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/540a152c-2be5-4253-b272-586b34835043-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.522425 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.522329 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-224gh\" (UniqueName: \"kubernetes.io/projected/540a152c-2be5-4253-b272-586b34835043-kube-api-access-224gh\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.522586 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.522504 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/540a152c-2be5-4253-b272-586b34835043-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.522586 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.522532 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/540a152c-2be5-4253-b272-586b34835043-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.522703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.522675 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/540a152c-2be5-4253-b272-586b34835043-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.522766 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.522698 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/540a152c-2be5-4253-b272-586b34835043-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.523017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.522986 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/540a152c-2be5-4253-b272-586b34835043-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.525396 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.525368 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/540a152c-2be5-4253-b272-586b34835043-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.525558 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.525538 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/540a152c-2be5-4253-b272-586b34835043-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.536243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.536221 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-224gh\" (UniqueName: \"kubernetes.io/projected/540a152c-2be5-4253-b272-586b34835043-kube-api-access-224gh\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-kkzgw\" (UID: \"540a152c-2be5-4253-b272-586b34835043\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.644215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.644100 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:11.929404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:11.929378 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw"] Oct 30 19:39:11.930620 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:39:11.930592 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540a152c_2be5_4253_b272_586b34835043.slice/crio-9b87c5d68917ecc0e9ab13f050d1ae765caecc117ef5ef886a4afc11d7787431 WatchSource:0}: Error finding container 9b87c5d68917ecc0e9ab13f050d1ae765caecc117ef5ef886a4afc11d7787431: Status 404 returned error can't find the container with id 9b87c5d68917ecc0e9ab13f050d1ae765caecc117ef5ef886a4afc11d7787431 Oct 30 19:39:12.762779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:12.762743 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" event={"ID":"540a152c-2be5-4253-b272-586b34835043","Type":"ContainerStarted","Data":"9b87c5d68917ecc0e9ab13f050d1ae765caecc117ef5ef886a4afc11d7787431"} Oct 30 19:39:13.039806 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:13.039724 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:13.778615 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:13.778571 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" event={"ID":"540a152c-2be5-4253-b272-586b34835043","Type":"ContainerStarted","Data":"0766f972206927348c0bc8a8c08752464c797201a178bab49fe6dfe11cec47d0"} Oct 30 19:39:13.806919 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:13.806865 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" podStartSLOduration=1.43858847 podStartE2EDuration="2.806848866s" totalImagesPullingTime="1.368260396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:13.804870166 +0000 UTC m=+473.500078126" watchObservedRunningTime="2026-10-30 19:39:13.806848866 +0000 UTC m=+473.502056823" Oct 30 19:39:14.645057 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:14.645026 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:15.975787 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:39:15.975747 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:39:16.361334 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:16.361286 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:17.835945 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:17.835908 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:19.269367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.269250 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-zp9tr"] Oct 30 19:39:19.274295 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.274270 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.284475 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.284446 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 30 19:39:19.287273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.287252 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-zp9tr"] Oct 30 19:39:19.359537 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.359500 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:19.404517 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.404481 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/634bf037-b673-4967-b0d9-9d6ef06eb1ca-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.404804 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.404522 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/634bf037-b673-4967-b0d9-9d6ef06eb1ca-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.404804 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.404560 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/634bf037-b673-4967-b0d9-9d6ef06eb1ca-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.404804 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.404695 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/634bf037-b673-4967-b0d9-9d6ef06eb1ca-home\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.404980 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.404813 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llwmd\" (UniqueName: \"kubernetes.io/projected/634bf037-b673-4967-b0d9-9d6ef06eb1ca-kube-api-access-llwmd\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.404980 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.404935 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/634bf037-b673-4967-b0d9-9d6ef06eb1ca-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.506071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.506035 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/634bf037-b673-4967-b0d9-9d6ef06eb1ca-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.506071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.506075 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/634bf037-b673-4967-b0d9-9d6ef06eb1ca-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.506341 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.506100 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/634bf037-b673-4967-b0d9-9d6ef06eb1ca-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.506341 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.506128 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/634bf037-b673-4967-b0d9-9d6ef06eb1ca-home\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.506341 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.506236 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llwmd\" (UniqueName: \"kubernetes.io/projected/634bf037-b673-4967-b0d9-9d6ef06eb1ca-kube-api-access-llwmd\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.506341 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.506277 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/634bf037-b673-4967-b0d9-9d6ef06eb1ca-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.506586 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.506555 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/634bf037-b673-4967-b0d9-9d6ef06eb1ca-home\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.506657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.506571 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/634bf037-b673-4967-b0d9-9d6ef06eb1ca-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.506720 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.506701 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/634bf037-b673-4967-b0d9-9d6ef06eb1ca-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.508512 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.508481 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/634bf037-b673-4967-b0d9-9d6ef06eb1ca-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.508731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.508715 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/634bf037-b673-4967-b0d9-9d6ef06eb1ca-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.520194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.520113 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llwmd\" (UniqueName: \"kubernetes.io/projected/634bf037-b673-4967-b0d9-9d6ef06eb1ca-kube-api-access-llwmd\") pod \"echo-simulator-kserve-64b6f8f96c-zp9tr\" (UID: \"634bf037-b673-4967-b0d9-9d6ef06eb1ca\") " pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.593884 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.593849 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:19.882535 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:19.882499 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-zp9tr"] Oct 30 19:39:19.882773 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:39:19.882740 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod634bf037_b673_4967_b0d9_9d6ef06eb1ca.slice/crio-1c405432f560ea6e55209b8d008e30bd8e283500b1c1839f1285806c1222e634 WatchSource:0}: Error finding container 1c405432f560ea6e55209b8d008e30bd8e283500b1c1839f1285806c1222e634: Status 404 returned error can't find the container with id 1c405432f560ea6e55209b8d008e30bd8e283500b1c1839f1285806c1222e634 Oct 30 19:39:20.045857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.045821 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc"] Oct 30 19:39:20.052746 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.051069 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.059830 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.059787 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 19:39:20.063306 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.062972 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc"] Oct 30 19:39:20.128866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.128773 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:20.214068 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.214036 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fc93c8-2259-4138-9691-4c9f4e758a01-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.214246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.214158 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fc93c8-2259-4138-9691-4c9f4e758a01-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.214246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.214203 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-844ms\" (UniqueName: \"kubernetes.io/projected/f1fc93c8-2259-4138-9691-4c9f4e758a01-kube-api-access-844ms\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.214324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.214266 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fc93c8-2259-4138-9691-4c9f4e758a01-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.214324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.214305 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fc93c8-2259-4138-9691-4c9f4e758a01-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.214386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.214333 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fc93c8-2259-4138-9691-4c9f4e758a01-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.315599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.314875 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fc93c8-2259-4138-9691-4c9f4e758a01-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.315599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.314945 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fc93c8-2259-4138-9691-4c9f4e758a01-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.315599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.315050 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fc93c8-2259-4138-9691-4c9f4e758a01-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.315599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.315104 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fc93c8-2259-4138-9691-4c9f4e758a01-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.315599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.315182 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-844ms\" (UniqueName: \"kubernetes.io/projected/f1fc93c8-2259-4138-9691-4c9f4e758a01-kube-api-access-844ms\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.315599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.315228 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fc93c8-2259-4138-9691-4c9f4e758a01-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.315599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.315375 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1fc93c8-2259-4138-9691-4c9f4e758a01-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.315599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.315373 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1fc93c8-2259-4138-9691-4c9f4e758a01-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.316514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.315615 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1fc93c8-2259-4138-9691-4c9f4e758a01-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.317553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.317528 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1fc93c8-2259-4138-9691-4c9f4e758a01-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.317795 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.317775 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1fc93c8-2259-4138-9691-4c9f4e758a01-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.330487 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.330457 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-844ms\" (UniqueName: \"kubernetes.io/projected/f1fc93c8-2259-4138-9691-4c9f4e758a01-kube-api-access-844ms\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc\" (UID: \"f1fc93c8-2259-4138-9691-4c9f4e758a01\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.375979 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.375869 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:20.710374 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.709424 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc"] Oct 30 19:39:20.821112 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.821074 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" event={"ID":"634bf037-b673-4967-b0d9-9d6ef06eb1ca","Type":"ContainerStarted","Data":"1c405432f560ea6e55209b8d008e30bd8e283500b1c1839f1285806c1222e634"} Oct 30 19:39:20.822439 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:20.822376 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" event={"ID":"f1fc93c8-2259-4138-9691-4c9f4e758a01","Type":"ContainerStarted","Data":"56e18a168b91c85a041af50ee60d9c3e3bfda1a0c7667eb60da986ccc0a70869"} Oct 30 19:39:21.212744 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.212711 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:21.530096 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.530064 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7"] Oct 30 19:39:21.534612 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.534590 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.540654 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.540595 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7"] Oct 30 19:39:21.544901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.544879 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 19:39:21.632733 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.632694 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/542be5bf-ced0-491a-aade-a84c72a92e5c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.632944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.632750 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/542be5bf-ced0-491a-aade-a84c72a92e5c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.632944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.632825 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/542be5bf-ced0-491a-aade-a84c72a92e5c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.632944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.632878 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/542be5bf-ced0-491a-aade-a84c72a92e5c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.632944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.632911 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw59x\" (UniqueName: \"kubernetes.io/projected/542be5bf-ced0-491a-aade-a84c72a92e5c-kube-api-access-sw59x\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.633222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.632960 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/542be5bf-ced0-491a-aade-a84c72a92e5c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.634475 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.634443 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:21.644223 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.644200 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:21.671169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.670422 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:21.734475 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.734446 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/542be5bf-ced0-491a-aade-a84c72a92e5c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.734564 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.734497 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/542be5bf-ced0-491a-aade-a84c72a92e5c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.734622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.734566 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/542be5bf-ced0-491a-aade-a84c72a92e5c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.734678 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.734642 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/542be5bf-ced0-491a-aade-a84c72a92e5c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.734733 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.734687 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw59x\" (UniqueName: \"kubernetes.io/projected/542be5bf-ced0-491a-aade-a84c72a92e5c-kube-api-access-sw59x\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.734788 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.734735 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/542be5bf-ced0-491a-aade-a84c72a92e5c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.734856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.734837 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/542be5bf-ced0-491a-aade-a84c72a92e5c-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.735105 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.735078 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/542be5bf-ced0-491a-aade-a84c72a92e5c-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.735483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.735462 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/542be5bf-ced0-491a-aade-a84c72a92e5c-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.737275 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.737252 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/542be5bf-ced0-491a-aade-a84c72a92e5c-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.737551 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.737526 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/542be5bf-ced0-491a-aade-a84c72a92e5c-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.750459 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.750426 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw59x\" (UniqueName: \"kubernetes.io/projected/542be5bf-ced0-491a-aade-a84c72a92e5c-kube-api-access-sw59x\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7\" (UID: \"542be5bf-ced0-491a-aade-a84c72a92e5c\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.828290 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.828246 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" event={"ID":"f1fc93c8-2259-4138-9691-4c9f4e758a01","Type":"ContainerStarted","Data":"4ec868e6a8c5c02ff7cb8618406f149c990298d60d4b89285580320b30c71144"} Oct 30 19:39:21.829909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.829881 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" event={"ID":"634bf037-b673-4967-b0d9-9d6ef06eb1ca","Type":"ContainerStarted","Data":"16282159c75c137ca161bc5940cd43993a315dcd54492c3bebc0d8621c3cddd4"} Oct 30 19:39:21.841538 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.841509 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" Oct 30 19:39:21.857446 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.857280 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:21.859912 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.859866 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" podStartSLOduration=1.5145785219999999 podStartE2EDuration="1.859852812s" totalImagesPullingTime="345.27429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:21.857956467 +0000 UTC m=+481.553164429" watchObservedRunningTime="2026-10-30 19:39:21.859852812 +0000 UTC m=+481.555060769" Oct 30 19:39:21.988294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:21.988231 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" podStartSLOduration=1.151772583 podStartE2EDuration="2.988217872s" totalImagesPullingTime="1.836445289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:21.9580253 +0000 UTC m=+481.653233269" watchObservedRunningTime="2026-10-30 19:39:21.988217872 +0000 UTC m=+481.683425830" Oct 30 19:39:22.152177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:22.152137 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7"] Oct 30 19:39:22.155029 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:39:22.154985 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod542be5bf_ced0_491a_aade_a84c72a92e5c.slice/crio-29306445d07c63ea8faea8c28fb0fdca2006ea7525d0d42d3bc196c4cb30fe7b WatchSource:0}: Error finding container 29306445d07c63ea8faea8c28fb0fdca2006ea7525d0d42d3bc196c4cb30fe7b: Status 404 returned error can't find the container with id 29306445d07c63ea8faea8c28fb0fdca2006ea7525d0d42d3bc196c4cb30fe7b Oct 30 19:39:22.376916 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:22.376877 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:22.595353 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:22.595272 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:22.836814 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:22.836774 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" event={"ID":"542be5bf-ced0-491a-aade-a84c72a92e5c","Type":"ContainerStarted","Data":"1f3c53cfbdd5455ad3914d0b537fcd9367091172be7754f61229c10737356b95"} Oct 30 19:39:22.837055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:22.837024 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" event={"ID":"542be5bf-ced0-491a-aade-a84c72a92e5c","Type":"ContainerStarted","Data":"29306445d07c63ea8faea8c28fb0fdca2006ea7525d0d42d3bc196c4cb30fe7b"} Oct 30 19:39:22.858242 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:22.858208 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:22.862367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:22.862319 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podStartSLOduration=1.527671239 podStartE2EDuration="1.86230566s" totalImagesPullingTime="334.634421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:22.859510763 +0000 UTC m=+482.554718726" watchObservedRunningTime="2026-10-30 19:39:22.86230566 +0000 UTC m=+482.557513627" Oct 30 19:39:29.594677 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:29.594637 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:29.607126 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:29.607102 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:29.882587 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:29.882508 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-zp9tr" Oct 30 19:39:30.376646 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:30.376602 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:30.388909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:30.388882 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:30.888225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:30.888140 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" Oct 30 19:39:31.858059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:31.858027 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:31.870334 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:31.870310 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:31.894608 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:31.894586 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:39:41.301556 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:41.301514 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:41.459374 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:41.459342 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:42.857700 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:42.857646 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:43.009442 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:43.009388 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:39:45.976419 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:39:45.976370 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:39:47.009545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.009513 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z"] Oct 30 19:39:47.017765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.017738 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.029463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.029432 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z"] Oct 30 19:39:47.088222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.088173 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.088222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.088228 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.088440 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.088355 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.088440 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.088387 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.088440 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.088411 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5shq8\" (UniqueName: \"kubernetes.io/projected/892edc88-bb54-46e1-acd1-701313b9ae0a-kube-api-access-5shq8\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.088562 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.088501 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.088562 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.088519 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.088632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.088594 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892edc88-bb54-46e1-acd1-701313b9ae0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.088735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.088716 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.190264 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.190222 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.190264 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.190271 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.190503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.190297 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5shq8\" (UniqueName: \"kubernetes.io/projected/892edc88-bb54-46e1-acd1-701313b9ae0a-kube-api-access-5shq8\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.190503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.190342 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.190503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.190369 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.190503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.190414 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892edc88-bb54-46e1-acd1-701313b9ae0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.190503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.190490 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.190787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.190580 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.190787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.190606 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.190787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.190739 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.190787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.190774 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.190994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.190807 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.191086 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.191043 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.191579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.191556 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892edc88-bb54-46e1-acd1-701313b9ae0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.193598 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.193569 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.193710 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.193691 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.204185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.204139 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5shq8\" (UniqueName: \"kubernetes.io/projected/892edc88-bb54-46e1-acd1-701313b9ae0a-kube-api-access-5shq8\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.204385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.204369 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-hc22z\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.337643 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.337553 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:47.622731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.622699 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z"] Oct 30 19:39:47.623837 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:39:47.623811 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod892edc88_bb54_46e1_acd1_701313b9ae0a.slice/crio-417efd96100f34b231adb1433aabb138108472373065ea6ede7a091dd1ca9d7e WatchSource:0}: Error finding container 417efd96100f34b231adb1433aabb138108472373065ea6ede7a091dd1ca9d7e: Status 404 returned error can't find the container with id 417efd96100f34b231adb1433aabb138108472373065ea6ede7a091dd1ca9d7e Oct 30 19:39:47.626134 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.626099 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:39:47.626303 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.626202 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:39:47.626303 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.626249 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:39:47.963874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.963776 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" event={"ID":"892edc88-bb54-46e1-acd1-701313b9ae0a","Type":"ContainerStarted","Data":"2a197d207774acfbab864b95083dd2da8774ad0887239e372d2438fdbdb68b77"} Oct 30 19:39:47.963874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.963830 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" event={"ID":"892edc88-bb54-46e1-acd1-701313b9ae0a","Type":"ContainerStarted","Data":"417efd96100f34b231adb1433aabb138108472373065ea6ede7a091dd1ca9d7e"} Oct 30 19:39:47.990892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:47.990840 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" podStartSLOduration=1.990825145 podStartE2EDuration="1.990825145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:39:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:39:47.990066782 +0000 UTC m=+507.685274753" watchObservedRunningTime="2026-10-30 19:39:47.990825145 +0000 UTC m=+507.686033103" Oct 30 19:39:48.338081 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:48.338045 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:48.339667 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:48.339640 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hc22z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 19:39:48.339778 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:48.339689 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" podUID="892edc88-bb54-46e1-acd1-701313b9ae0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 19:39:49.338687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:49.338651 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hc22z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 19:39:49.339208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:49.338702 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" podUID="892edc88-bb54-46e1-acd1-701313b9ae0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 19:39:50.338124 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:50.338090 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hc22z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 19:39:50.338389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:50.338164 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" podUID="892edc88-bb54-46e1-acd1-701313b9ae0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 19:39:51.337947 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:51.337902 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hc22z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 19:39:51.338487 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:51.337966 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" podUID="892edc88-bb54-46e1-acd1-701313b9ae0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 19:39:52.338027 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:52.337987 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hc22z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 19:39:52.338534 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:52.338048 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" podUID="892edc88-bb54-46e1-acd1-701313b9ae0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 19:39:53.338242 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:53.338204 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hc22z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 19:39:53.338741 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:53.338298 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" podUID="892edc88-bb54-46e1-acd1-701313b9ae0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 19:39:54.338387 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:54.338348 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hc22z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 19:39:54.338865 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:54.338406 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" podUID="892edc88-bb54-46e1-acd1-701313b9ae0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 19:39:55.765523 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:55.765488 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:55.765995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:55.765818 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:39:55.766800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:39:55.766779 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 19:40:02.122784 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.122739 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt"] Oct 30 19:40:02.130208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.130191 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.172112 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.172072 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt"] Oct 30 19:40:02.260798 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.260764 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c63bc1ab-ca04-4273-ae53-2d0923e4900b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.260965 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.260816 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63bc1ab-ca04-4273-ae53-2d0923e4900b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.260965 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.260840 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63bc1ab-ca04-4273-ae53-2d0923e4900b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.261037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.260956 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63bc1ab-ca04-4273-ae53-2d0923e4900b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.261037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.260994 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nlfh\" (UniqueName: \"kubernetes.io/projected/c63bc1ab-ca04-4273-ae53-2d0923e4900b-kube-api-access-7nlfh\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.261100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.261084 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63bc1ab-ca04-4273-ae53-2d0923e4900b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.261177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.261132 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63bc1ab-ca04-4273-ae53-2d0923e4900b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.261223 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.261201 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63bc1ab-ca04-4273-ae53-2d0923e4900b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.261263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.261248 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63bc1ab-ca04-4273-ae53-2d0923e4900b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.362798 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.362764 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63bc1ab-ca04-4273-ae53-2d0923e4900b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.362997 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.362817 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63bc1ab-ca04-4273-ae53-2d0923e4900b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.362997 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.362837 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63bc1ab-ca04-4273-ae53-2d0923e4900b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.362997 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.362955 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63bc1ab-ca04-4273-ae53-2d0923e4900b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.363204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.363003 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c63bc1ab-ca04-4273-ae53-2d0923e4900b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.363204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.363055 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63bc1ab-ca04-4273-ae53-2d0923e4900b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.363204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.363098 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63bc1ab-ca04-4273-ae53-2d0923e4900b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.363204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.363175 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63bc1ab-ca04-4273-ae53-2d0923e4900b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.363429 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.363205 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c63bc1ab-ca04-4273-ae53-2d0923e4900b-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.363429 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.363216 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nlfh\" (UniqueName: \"kubernetes.io/projected/c63bc1ab-ca04-4273-ae53-2d0923e4900b-kube-api-access-7nlfh\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.363545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.363425 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63bc1ab-ca04-4273-ae53-2d0923e4900b-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.363545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.363498 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c63bc1ab-ca04-4273-ae53-2d0923e4900b-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.363653 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.363636 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c63bc1ab-ca04-4273-ae53-2d0923e4900b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.363698 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.363683 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c63bc1ab-ca04-4273-ae53-2d0923e4900b-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.365287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.365262 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c63bc1ab-ca04-4273-ae53-2d0923e4900b-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.365552 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.365531 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c63bc1ab-ca04-4273-ae53-2d0923e4900b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.377646 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.377591 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c63bc1ab-ca04-4273-ae53-2d0923e4900b-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.377863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.377837 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nlfh\" (UniqueName: \"kubernetes.io/projected/c63bc1ab-ca04-4273-ae53-2d0923e4900b-kube-api-access-7nlfh\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-zzkkt\" (UID: \"c63bc1ab-ca04-4273-ae53-2d0923e4900b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.444001 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.443981 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:02.728976 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.728953 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt"] Oct 30 19:40:02.729485 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:02.729461 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63bc1ab_ca04_4273_ae53_2d0923e4900b.slice/crio-c33153cd9f857e7227a1998fee9fbda6045c09bc0a05d3cd26cbc92ff4aa798d WatchSource:0}: Error finding container c33153cd9f857e7227a1998fee9fbda6045c09bc0a05d3cd26cbc92ff4aa798d: Status 404 returned error can't find the container with id c33153cd9f857e7227a1998fee9fbda6045c09bc0a05d3cd26cbc92ff4aa798d Oct 30 19:40:02.731558 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.731523 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:02.731664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.731602 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:02.731664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:02.731646 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:03.045227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:03.045187 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" event={"ID":"c63bc1ab-ca04-4273-ae53-2d0923e4900b","Type":"ContainerStarted","Data":"c3002e1312bc66c4c873e9a03bd791f4a21764c8dd36af6b9e0f4dd6194385f3"} Oct 30 19:40:03.045423 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:03.045252 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" event={"ID":"c63bc1ab-ca04-4273-ae53-2d0923e4900b","Type":"ContainerStarted","Data":"c33153cd9f857e7227a1998fee9fbda6045c09bc0a05d3cd26cbc92ff4aa798d"} Oct 30 19:40:03.075764 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:03.075711 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" podStartSLOduration=1.07569801 podStartE2EDuration="1.07569801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:03.074752263 +0000 UTC m=+522.769960224" watchObservedRunningTime="2026-10-30 19:40:03.07569801 +0000 UTC m=+522.770905968" Oct 30 19:40:03.444914 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:03.444829 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:03.466169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:03.456304 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:40:03.466169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:03.456382 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" podUID="c63bc1ab-ca04-4273-ae53-2d0923e4900b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 19:40:04.444800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:04.444767 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:40:04.444967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:04.444818 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" podUID="c63bc1ab-ca04-4273-ae53-2d0923e4900b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 19:40:05.444683 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:05.444646 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:40:05.444860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:05.444696 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" podUID="c63bc1ab-ca04-4273-ae53-2d0923e4900b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 19:40:06.444839 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:06.444797 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:40:06.445266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:06.444865 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" podUID="c63bc1ab-ca04-4273-ae53-2d0923e4900b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 19:40:07.444496 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:07.444458 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:40:07.444678 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:07.444556 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" podUID="c63bc1ab-ca04-4273-ae53-2d0923e4900b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 19:40:08.444914 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:08.444882 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:40:08.445385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:08.444941 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" podUID="c63bc1ab-ca04-4273-ae53-2d0923e4900b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 19:40:09.444679 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:09.444646 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body= Oct 30 19:40:09.444848 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:09.444694 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" podUID="c63bc1ab-ca04-4273-ae53-2d0923e4900b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" Oct 30 19:40:10.621485 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:10.621445 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:10.872075 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:10.871993 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:10.872442 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:10.872419 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:10.873049 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:10.873028 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" Oct 30 19:40:13.651626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:13.651588 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:14.373716 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.373685 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7"] Oct 30 19:40:14.378247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.378220 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.388688 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.388660 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3a55af38-openshift-default-dockercfg-27ttt\"" Oct 30 19:40:14.392975 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.392949 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7"] Oct 30 19:40:14.474628 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.474596 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62bpj\" (UniqueName: \"kubernetes.io/projected/8253e5fe-3ea9-4367-b43d-274dc85176f8-kube-api-access-62bpj\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.474800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.474657 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-envoy\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.474800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.474736 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-workload-certs\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.474800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.474765 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-data\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.474960 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.474899 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-workload-socket\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.475000 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.474969 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-credential-socket\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.475039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.475012 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8253e5fe-3ea9-4367-b43d-274dc85176f8-istiod-ca-cert\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.475039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.475034 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-token\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.475118 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.475073 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-podinfo\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.581177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.575714 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62bpj\" (UniqueName: \"kubernetes.io/projected/8253e5fe-3ea9-4367-b43d-274dc85176f8-kube-api-access-62bpj\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.581177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.575787 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-envoy\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.581177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.575823 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-workload-certs\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.581177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.575848 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-data\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.581177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.575915 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-workload-socket\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.581177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.575952 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-credential-socket\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.581177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.575976 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8253e5fe-3ea9-4367-b43d-274dc85176f8-istiod-ca-cert\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.581177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.576005 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-token\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.581177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.576171 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-podinfo\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.581177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.577059 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-workload-socket\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.581177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.579470 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-data\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.581177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.579718 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-credential-socket\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.581177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.580364 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8253e5fe-3ea9-4367-b43d-274dc85176f8-istiod-ca-cert\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.597169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.593441 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-envoy\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.597169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.596440 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-workload-certs\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.610803 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.610772 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62bpj\" (UniqueName: \"kubernetes.io/projected/8253e5fe-3ea9-4367-b43d-274dc85176f8-kube-api-access-62bpj\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.616193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.611318 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-token\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.616455 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.612173 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-podinfo\") pod \"e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.693745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.693647 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:14.864910 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.864861 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:14.977724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.977693 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7"] Oct 30 19:40:14.979945 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:14.979915 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8253e5fe_3ea9_4367_b43d_274dc85176f8.slice/crio-b10319e26f314b824cd76dd9aef19801cb38c8ea9bbce194dee728a06348bb75 WatchSource:0}: Error finding container b10319e26f314b824cd76dd9aef19801cb38c8ea9bbce194dee728a06348bb75: Status 404 returned error can't find the container with id b10319e26f314b824cd76dd9aef19801cb38c8ea9bbce194dee728a06348bb75 Oct 30 19:40:14.982360 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.982333 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:14.982457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.982402 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:14.982457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:14.982437 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:15.131874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.131841 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" event={"ID":"8253e5fe-3ea9-4367-b43d-274dc85176f8","Type":"ContainerStarted","Data":"ca9f3c187630b8610dfd3333be4cd7315d86c363552b1a47e09f5bc6a311353f"} Oct 30 19:40:15.132039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.131892 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" event={"ID":"8253e5fe-3ea9-4367-b43d-274dc85176f8","Type":"ContainerStarted","Data":"b10319e26f314b824cd76dd9aef19801cb38c8ea9bbce194dee728a06348bb75"} Oct 30 19:40:15.164099 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.164038 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" podStartSLOduration=1.164019171 podStartE2EDuration="1.164019171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:15.163416846 +0000 UTC m=+534.858624832" watchObservedRunningTime="2026-10-30 19:40:15.164019171 +0000 UTC m=+534.859227124" Oct 30 19:40:15.604766 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.604729 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:15.693765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.693734 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:15.952208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.949733 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc"] Oct 30 19:40:15.958894 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.957232 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:15.984165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.982225 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc"] Oct 30 19:40:15.984165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.982548 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0897eecc-openshift-default-dockercfg-qgg7g\"" Oct 30 19:40:15.997620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.997588 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-envoy\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:15.997789 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.997652 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-data\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:15.997789 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.997699 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-credential-socket\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:15.997789 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.997734 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-workload-certs\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:15.997789 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.997775 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-token\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:15.998020 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.997817 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-workload-socket\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:15.998020 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.997923 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9cb9939-3547-4232-b6c0-d2a40803a666-istiod-ca-cert\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:15.998020 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.997957 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p867c\" (UniqueName: \"kubernetes.io/projected/d9cb9939-3547-4232-b6c0-d2a40803a666-kube-api-access-p867c\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:15.998020 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:15.997986 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-podinfo\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:15.999642 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:40:15.999587 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:40:16.065930 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.065890 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:16.099191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.099130 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-envoy\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.099370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.099206 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-data\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.099370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.099236 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-credential-socket\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.099370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.099256 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-workload-certs\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.099370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.099279 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-token\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.099370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.099302 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-workload-socket\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.099634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.099380 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9cb9939-3547-4232-b6c0-d2a40803a666-istiod-ca-cert\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.099634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.099406 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p867c\" (UniqueName: \"kubernetes.io/projected/d9cb9939-3547-4232-b6c0-d2a40803a666-kube-api-access-p867c\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.099634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.099437 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-podinfo\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.099786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.099696 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-data\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.099823 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.099785 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-credential-socket\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.099867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.099820 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-workload-socket\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.100103 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.100082 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9cb9939-3547-4232-b6c0-d2a40803a666-istiod-ca-cert\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.100211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.100186 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-workload-certs\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.101546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.101526 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-envoy\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.101959 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.101938 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-podinfo\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.113642 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.113616 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p867c\" (UniqueName: \"kubernetes.io/projected/d9cb9939-3547-4232-b6c0-d2a40803a666-kube-api-access-p867c\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.114320 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.114303 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-token\") pod \"e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.245237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.245122 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:16.309315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.309275 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:16.600889 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.600860 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc"] Oct 30 19:40:16.603141 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:16.603101 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9cb9939_3547_4232_b6c0_d2a40803a666.slice/crio-aed4529108ddb937b08a60f08ff6ab442ed75de62722178d00321df4044435f4 WatchSource:0}: Error finding container aed4529108ddb937b08a60f08ff6ab442ed75de62722178d00321df4044435f4: Status 404 returned error can't find the container with id aed4529108ddb937b08a60f08ff6ab442ed75de62722178d00321df4044435f4 Oct 30 19:40:16.605659 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.605623 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:16.605779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.605693 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:16.605779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.605726 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:16.684978 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.684950 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:16.700583 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.700557 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:16.838856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:16.838820 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:17.145348 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:17.145308 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" event={"ID":"d9cb9939-3547-4232-b6c0-d2a40803a666","Type":"ContainerStarted","Data":"fefd57fd8b93e0f45f5d08c0c0e949e413a155fdede32c576f4403595fafd81d"} Oct 30 19:40:17.145530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:17.145373 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" event={"ID":"d9cb9939-3547-4232-b6c0-d2a40803a666","Type":"ContainerStarted","Data":"aed4529108ddb937b08a60f08ff6ab442ed75de62722178d00321df4044435f4"} Oct 30 19:40:17.145743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:17.145714 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:17.146828 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:17.146807 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:17.177341 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:17.177294 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" podStartSLOduration=2.1772811770000002 podStartE2EDuration="2.177281177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:17.176272002 +0000 UTC m=+536.871479961" watchObservedRunningTime="2026-10-30 19:40:17.177281177 +0000 UTC m=+536.872489134" Oct 30 19:40:17.292663 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:17.292628 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:17.310477 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:17.310449 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:17.317517 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:17.317475 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:17.466792 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:17.466700 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:17.615530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:17.615493 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:18.017934 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.017894 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:18.151527 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.151501 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:18.152421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.152401 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:18.266041 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.266003 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:18.429170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.429063 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:18.793217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.793168 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:18.839700 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.839659 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz"] Oct 30 19:40:18.843630 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.843602 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" Oct 30 19:40:18.856191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.855715 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3a55af38-dockercfg-tfwqd\"" Oct 30 19:40:18.856191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.855715 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3a55af38\"" Oct 30 19:40:18.858193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.858135 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz"] Oct 30 19:40:18.868902 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.868870 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm"] Oct 30 19:40:18.873617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.873586 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" Oct 30 19:40:18.888896 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.883594 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm"] Oct 30 19:40:18.953170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.939534 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:18.953170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.941435 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc79v\" (UniqueName: \"kubernetes.io/projected/91802908-1257-4dc4-8e5b-baaeb4dbdfbb-kube-api-access-gc79v\") pod \"payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz\" (UID: \"91802908-1257-4dc4-8e5b-baaeb4dbdfbb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" Oct 30 19:40:18.953170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.941538 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91802908-1257-4dc4-8e5b-baaeb4dbdfbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz\" (UID: \"91802908-1257-4dc4-8e5b-baaeb4dbdfbb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" Oct 30 19:40:18.953170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.941592 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pcxk\" (UniqueName: \"kubernetes.io/projected/550f7a23-825c-45dc-a2bf-ee67264eed56-kube-api-access-5pcxk\") pod \"payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm\" (UID: \"550f7a23-825c-45dc-a2bf-ee67264eed56\") " pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" Oct 30 19:40:18.953170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:18.941648 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550f7a23-825c-45dc-a2bf-ee67264eed56-plugins-config-volume\") pod \"payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm\" (UID: \"550f7a23-825c-45dc-a2bf-ee67264eed56\") " pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" Oct 30 19:40:19.042856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.042813 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc79v\" (UniqueName: \"kubernetes.io/projected/91802908-1257-4dc4-8e5b-baaeb4dbdfbb-kube-api-access-gc79v\") pod \"payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz\" (UID: \"91802908-1257-4dc4-8e5b-baaeb4dbdfbb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" Oct 30 19:40:19.043372 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.042898 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91802908-1257-4dc4-8e5b-baaeb4dbdfbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz\" (UID: \"91802908-1257-4dc4-8e5b-baaeb4dbdfbb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" Oct 30 19:40:19.043372 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.042943 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pcxk\" (UniqueName: \"kubernetes.io/projected/550f7a23-825c-45dc-a2bf-ee67264eed56-kube-api-access-5pcxk\") pod \"payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm\" (UID: \"550f7a23-825c-45dc-a2bf-ee67264eed56\") " pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" Oct 30 19:40:19.043372 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.042988 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550f7a23-825c-45dc-a2bf-ee67264eed56-plugins-config-volume\") pod \"payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm\" (UID: \"550f7a23-825c-45dc-a2bf-ee67264eed56\") " pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" Oct 30 19:40:19.043950 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.043916 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550f7a23-825c-45dc-a2bf-ee67264eed56-plugins-config-volume\") pod \"payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm\" (UID: \"550f7a23-825c-45dc-a2bf-ee67264eed56\") " pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" Oct 30 19:40:19.043950 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.043936 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91802908-1257-4dc4-8e5b-baaeb4dbdfbb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz\" (UID: \"91802908-1257-4dc4-8e5b-baaeb4dbdfbb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" Oct 30 19:40:19.057073 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.057043 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc79v\" (UniqueName: \"kubernetes.io/projected/91802908-1257-4dc4-8e5b-baaeb4dbdfbb-kube-api-access-gc79v\") pod \"payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz\" (UID: \"91802908-1257-4dc4-8e5b-baaeb4dbdfbb\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" Oct 30 19:40:19.057073 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.057055 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pcxk\" (UniqueName: \"kubernetes.io/projected/550f7a23-825c-45dc-a2bf-ee67264eed56-kube-api-access-5pcxk\") pod \"payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm\" (UID: \"550f7a23-825c-45dc-a2bf-ee67264eed56\") " pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" Oct 30 19:40:19.158839 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.158809 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" Oct 30 19:40:19.204171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.203624 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" Oct 30 19:40:19.440868 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.440836 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz"] Oct 30 19:40:19.442011 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:19.441905 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91802908_1257_4dc4_8e5b_baaeb4dbdfbb.slice/crio-7e67d0450b0ab138a73045e788af6f7f6a6d6a1bc922f681d0a86d04efe8ad2d WatchSource:0}: Error finding container 7e67d0450b0ab138a73045e788af6f7f6a6d6a1bc922f681d0a86d04efe8ad2d: Status 404 returned error can't find the container with id 7e67d0450b0ab138a73045e788af6f7f6a6d6a1bc922f681d0a86d04efe8ad2d Oct 30 19:40:19.455031 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.455004 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:19.520890 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.520661 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm"] Oct 30 19:40:19.521305 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:19.521200 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod550f7a23_825c_45dc_a2bf_ee67264eed56.slice/crio-4f1c9d4d771a0cdc4ef065f93e6bb660f358d4c8994f117e403fe3b4f5efb1de WatchSource:0}: Error finding container 4f1c9d4d771a0cdc4ef065f93e6bb660f358d4c8994f117e403fe3b4f5efb1de: Status 404 returned error can't find the container with id 4f1c9d4d771a0cdc4ef065f93e6bb660f358d4c8994f117e403fe3b4f5efb1de Oct 30 19:40:19.632399 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.632365 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:19.662448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.662417 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln"] Oct 30 19:40:19.667363 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.667341 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" Oct 30 19:40:19.677043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.677020 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b33c04be\"" Oct 30 19:40:19.680026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.677120 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b33c04be-dockercfg-qq9ft\"" Oct 30 19:40:19.683159 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.683120 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln"] Oct 30 19:40:19.713202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.713093 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz"] Oct 30 19:40:19.717048 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.717021 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" Oct 30 19:40:19.725979 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.725949 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz"] Oct 30 19:40:19.752581 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.752541 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6733505d-3dd3-462d-a96a-ee60c3764e64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln\" (UID: \"6733505d-3dd3-462d-a96a-ee60c3764e64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" Oct 30 19:40:19.752754 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.752612 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfm9h\" (UniqueName: \"kubernetes.io/projected/6733505d-3dd3-462d-a96a-ee60c3764e64-kube-api-access-qfm9h\") pod \"payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln\" (UID: \"6733505d-3dd3-462d-a96a-ee60c3764e64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" Oct 30 19:40:19.782609 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.782562 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:19.854481 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.854440 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6733505d-3dd3-462d-a96a-ee60c3764e64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln\" (UID: \"6733505d-3dd3-462d-a96a-ee60c3764e64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" Oct 30 19:40:19.854662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.854496 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n77s\" (UniqueName: \"kubernetes.io/projected/34ec41bf-040e-46ff-82df-7a3f73929cc8-kube-api-access-9n77s\") pod \"payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz\" (UID: \"34ec41bf-040e-46ff-82df-7a3f73929cc8\") " pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" Oct 30 19:40:19.854662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.854545 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfm9h\" (UniqueName: \"kubernetes.io/projected/6733505d-3dd3-462d-a96a-ee60c3764e64-kube-api-access-qfm9h\") pod \"payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln\" (UID: \"6733505d-3dd3-462d-a96a-ee60c3764e64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" Oct 30 19:40:19.854789 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.854764 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ec41bf-040e-46ff-82df-7a3f73929cc8-plugins-config-volume\") pod \"payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz\" (UID: \"34ec41bf-040e-46ff-82df-7a3f73929cc8\") " pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" Oct 30 19:40:19.855308 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.855287 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6733505d-3dd3-462d-a96a-ee60c3764e64-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln\" (UID: \"6733505d-3dd3-462d-a96a-ee60c3764e64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" Oct 30 19:40:19.870277 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.870254 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfm9h\" (UniqueName: \"kubernetes.io/projected/6733505d-3dd3-462d-a96a-ee60c3764e64-kube-api-access-qfm9h\") pod \"payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln\" (UID: \"6733505d-3dd3-462d-a96a-ee60c3764e64\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" Oct 30 19:40:19.955613 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.955562 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ec41bf-040e-46ff-82df-7a3f73929cc8-plugins-config-volume\") pod \"payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz\" (UID: \"34ec41bf-040e-46ff-82df-7a3f73929cc8\") " pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" Oct 30 19:40:19.955825 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.955778 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n77s\" (UniqueName: \"kubernetes.io/projected/34ec41bf-040e-46ff-82df-7a3f73929cc8-kube-api-access-9n77s\") pod \"payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz\" (UID: \"34ec41bf-040e-46ff-82df-7a3f73929cc8\") " pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" Oct 30 19:40:19.956548 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.956517 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ec41bf-040e-46ff-82df-7a3f73929cc8-plugins-config-volume\") pod \"payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz\" (UID: \"34ec41bf-040e-46ff-82df-7a3f73929cc8\") " pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" Oct 30 19:40:19.970287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.970218 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n77s\" (UniqueName: \"kubernetes.io/projected/34ec41bf-040e-46ff-82df-7a3f73929cc8-kube-api-access-9n77s\") pod \"payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz\" (UID: \"34ec41bf-040e-46ff-82df-7a3f73929cc8\") " pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" Oct 30 19:40:19.985084 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:19.985048 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" Oct 30 19:40:20.036847 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.036577 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" Oct 30 19:40:20.170467 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.170354 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" event={"ID":"550f7a23-825c-45dc-a2bf-ee67264eed56","Type":"ContainerStarted","Data":"4f1c9d4d771a0cdc4ef065f93e6bb660f358d4c8994f117e403fe3b4f5efb1de"} Oct 30 19:40:20.174933 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.174843 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" event={"ID":"91802908-1257-4dc4-8e5b-baaeb4dbdfbb","Type":"ContainerStarted","Data":"7e67d0450b0ab138a73045e788af6f7f6a6d6a1bc922f681d0a86d04efe8ad2d"} Oct 30 19:40:20.277110 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.277072 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln"] Oct 30 19:40:20.278206 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:20.278170 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6733505d_3dd3_462d_a96a_ee60c3764e64.slice/crio-6f0169a4a8c3a104fe69873c9a0239cf1edb0cd8b4ebdeac0ace98c87a5be872 WatchSource:0}: Error finding container 6f0169a4a8c3a104fe69873c9a0239cf1edb0cd8b4ebdeac0ace98c87a5be872: Status 404 returned error can't find the container with id 6f0169a4a8c3a104fe69873c9a0239cf1edb0cd8b4ebdeac0ace98c87a5be872 Oct 30 19:40:20.365080 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.365052 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz"] Oct 30 19:40:20.369032 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:20.368988 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34ec41bf_040e_46ff_82df_7a3f73929cc8.slice/crio-1f16240daa735a87a0bde27416aa1c36669e98190c936aa9f1cedfc63d888f2a WatchSource:0}: Error finding container 1f16240daa735a87a0bde27416aa1c36669e98190c936aa9f1cedfc63d888f2a: Status 404 returned error can't find the container with id 1f16240daa735a87a0bde27416aa1c36669e98190c936aa9f1cedfc63d888f2a Oct 30 19:40:20.623612 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.623526 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:20.805333 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.805295 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:20.807839 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.807809 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28"] Oct 30 19:40:20.811667 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.811641 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" Oct 30 19:40:20.822417 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.822390 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28"] Oct 30 19:40:20.823109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.823070 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0897eecc\"" Oct 30 19:40:20.829326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.829234 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h"] Oct 30 19:40:20.840263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.840242 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" Oct 30 19:40:20.849318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.849267 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0897eecc-dockercfg-lr5hb\"" Oct 30 19:40:20.850009 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.849983 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h"] Oct 30 19:40:20.866871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.866842 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec2b758-fec8-49ce-a483-e3f91af11c33-plugins-config-volume\") pod \"payload-processing-e2e-ait-0897eecc-75774c584b-cqx28\" (UID: \"8ec2b758-fec8-49ce-a483-e3f91af11c33\") " pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" Oct 30 19:40:20.867000 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.866887 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvgwh\" (UniqueName: \"kubernetes.io/projected/8ec2b758-fec8-49ce-a483-e3f91af11c33-kube-api-access-vvgwh\") pod \"payload-processing-e2e-ait-0897eecc-75774c584b-cqx28\" (UID: \"8ec2b758-fec8-49ce-a483-e3f91af11c33\") " pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" Oct 30 19:40:20.966383 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.962196 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:20.967755 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.967721 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f46b66e-1ca4-41ca-9be7-040cb8dec18a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h\" (UID: \"8f46b66e-1ca4-41ca-9be7-040cb8dec18a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" Oct 30 19:40:20.967898 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.967770 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76zj6\" (UniqueName: \"kubernetes.io/projected/8f46b66e-1ca4-41ca-9be7-040cb8dec18a-kube-api-access-76zj6\") pod \"payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h\" (UID: \"8f46b66e-1ca4-41ca-9be7-040cb8dec18a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" Oct 30 19:40:20.967898 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.967821 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec2b758-fec8-49ce-a483-e3f91af11c33-plugins-config-volume\") pod \"payload-processing-e2e-ait-0897eecc-75774c584b-cqx28\" (UID: \"8ec2b758-fec8-49ce-a483-e3f91af11c33\") " pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" Oct 30 19:40:20.967898 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.967860 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvgwh\" (UniqueName: \"kubernetes.io/projected/8ec2b758-fec8-49ce-a483-e3f91af11c33-kube-api-access-vvgwh\") pod \"payload-processing-e2e-ait-0897eecc-75774c584b-cqx28\" (UID: \"8ec2b758-fec8-49ce-a483-e3f91af11c33\") " pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" Oct 30 19:40:20.970977 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.970926 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec2b758-fec8-49ce-a483-e3f91af11c33-plugins-config-volume\") pod \"payload-processing-e2e-ait-0897eecc-75774c584b-cqx28\" (UID: \"8ec2b758-fec8-49ce-a483-e3f91af11c33\") " pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" Oct 30 19:40:20.989918 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:20.989851 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvgwh\" (UniqueName: \"kubernetes.io/projected/8ec2b758-fec8-49ce-a483-e3f91af11c33-kube-api-access-vvgwh\") pod \"payload-processing-e2e-ait-0897eecc-75774c584b-cqx28\" (UID: \"8ec2b758-fec8-49ce-a483-e3f91af11c33\") " pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" Oct 30 19:40:21.070434 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:21.070395 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f46b66e-1ca4-41ca-9be7-040cb8dec18a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h\" (UID: \"8f46b66e-1ca4-41ca-9be7-040cb8dec18a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" Oct 30 19:40:21.070434 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:21.070446 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76zj6\" (UniqueName: \"kubernetes.io/projected/8f46b66e-1ca4-41ca-9be7-040cb8dec18a-kube-api-access-76zj6\") pod \"payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h\" (UID: \"8f46b66e-1ca4-41ca-9be7-040cb8dec18a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" Oct 30 19:40:21.073102 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:21.071731 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f46b66e-1ca4-41ca-9be7-040cb8dec18a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h\" (UID: \"8f46b66e-1ca4-41ca-9be7-040cb8dec18a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" Oct 30 19:40:21.089859 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:21.089796 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76zj6\" (UniqueName: \"kubernetes.io/projected/8f46b66e-1ca4-41ca-9be7-040cb8dec18a-kube-api-access-76zj6\") pod \"payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h\" (UID: \"8f46b66e-1ca4-41ca-9be7-040cb8dec18a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" Oct 30 19:40:21.139294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:21.139262 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" Oct 30 19:40:21.163101 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:21.163072 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" Oct 30 19:40:21.193181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:21.185825 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" event={"ID":"6733505d-3dd3-462d-a96a-ee60c3764e64","Type":"ContainerStarted","Data":"6f0169a4a8c3a104fe69873c9a0239cf1edb0cd8b4ebdeac0ace98c87a5be872"} Oct 30 19:40:21.193181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:21.191049 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" event={"ID":"34ec41bf-040e-46ff-82df-7a3f73929cc8","Type":"ContainerStarted","Data":"1f16240daa735a87a0bde27416aa1c36669e98190c936aa9f1cedfc63d888f2a"} Oct 30 19:40:21.517410 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:21.515705 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec2b758_fec8_49ce_a483_e3f91af11c33.slice/crio-50d5bd57fa26687db9cb633e1a9f0e600840f9ee7383939fd93493ed03edd1f1 WatchSource:0}: Error finding container 50d5bd57fa26687db9cb633e1a9f0e600840f9ee7383939fd93493ed03edd1f1: Status 404 returned error can't find the container with id 50d5bd57fa26687db9cb633e1a9f0e600840f9ee7383939fd93493ed03edd1f1 Oct 30 19:40:21.517410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:21.516352 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28"] Oct 30 19:40:21.608134 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:21.608032 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h"] Oct 30 19:40:21.615957 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:21.615918 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f46b66e_1ca4_41ca_9be7_040cb8dec18a.slice/crio-6f6a40ab13a93d727e5e6fb34e30dd553214b55b683154d0fc859a37622c2b91 WatchSource:0}: Error finding container 6f6a40ab13a93d727e5e6fb34e30dd553214b55b683154d0fc859a37622c2b91: Status 404 returned error can't find the container with id 6f6a40ab13a93d727e5e6fb34e30dd553214b55b683154d0fc859a37622c2b91 Oct 30 19:40:21.679775 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:21.679716 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:21.847207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:21.847120 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:21.995264 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:21.995225 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:22.204585 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:22.204496 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" event={"ID":"8ec2b758-fec8-49ce-a483-e3f91af11c33","Type":"ContainerStarted","Data":"50d5bd57fa26687db9cb633e1a9f0e600840f9ee7383939fd93493ed03edd1f1"} Oct 30 19:40:22.209584 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:22.209549 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" event={"ID":"8f46b66e-1ca4-41ca-9be7-040cb8dec18a","Type":"ContainerStarted","Data":"6f6a40ab13a93d727e5e6fb34e30dd553214b55b683154d0fc859a37622c2b91"} Oct 30 19:40:22.711574 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:22.711481 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:22.862830 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:22.862786 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:23.015527 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.015492 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:23.156000 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.155963 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:23.381444 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.379491 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7"] Oct 30 19:40:23.399234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.394291 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.408396 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.404709 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7"] Oct 30 19:40:23.408396 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.405952 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-d05eeacd-openshift-default-dockercfg-5rh7n\"" Oct 30 19:40:23.480405 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.480361 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:23.526779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.526734 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-data\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.526962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.526798 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-workload-socket\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.526962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.526831 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-envoy\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.526962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.526906 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-credential-socket\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.527343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.526978 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czctk\" (UniqueName: \"kubernetes.io/projected/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-kube-api-access-czctk\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.527343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.527062 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istiod-ca-cert\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.527343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.527092 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-token\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.527343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.527122 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-workload-certs\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.527343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.527172 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-podinfo\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.628268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.628230 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-credential-socket\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.628441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.628297 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czctk\" (UniqueName: \"kubernetes.io/projected/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-kube-api-access-czctk\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.628441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.628352 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istiod-ca-cert\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.628441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.628377 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-token\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.628441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.628407 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-workload-certs\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.628441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.628429 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-podinfo\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.628746 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.628478 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-data\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.628746 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.628501 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-workload-socket\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.628746 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.628518 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-envoy\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.630282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.629811 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istiod-ca-cert\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.630282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.629848 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-credential-socket\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.630282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.629967 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-workload-certs\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.630282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.630106 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-workload-socket\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.630282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.630188 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-data\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.638539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.638445 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:23.639490 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.639469 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-envoy\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.644268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.644225 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czctk\" (UniqueName: \"kubernetes.io/projected/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-kube-api-access-czctk\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.644735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.644686 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-podinfo\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.645292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.645270 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-token\") pod \"e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.720707 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.720467 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:23.977343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:23.977245 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:24.033177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:24.033091 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7"] Oct 30 19:40:24.039922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:24.039498 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:24.039922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:24.039586 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:24.039922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:24.039633 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:24.125516 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:24.125267 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:24.235509 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:24.235427 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" event={"ID":"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8","Type":"ContainerStarted","Data":"0e89c6f15fc7793aea59c2f8c71a6bd14f211ecdb5cde1b721757cb51982f197"} Oct 30 19:40:24.235509 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:24.235490 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" event={"ID":"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8","Type":"ContainerStarted","Data":"1ca4ce270011c4b7bbb800188f1dff2418c43364205807cbd391870a053bea90"} Oct 30 19:40:24.263592 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:24.263539 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" podStartSLOduration=1.263521665 podStartE2EDuration="1.263521665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:24.262625765 +0000 UTC m=+543.957833719" watchObservedRunningTime="2026-10-30 19:40:24.263521665 +0000 UTC m=+543.958729625" Oct 30 19:40:24.721442 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:24.721406 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:25.224288 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.224254 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:25.242011 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.241974 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" event={"ID":"34ec41bf-040e-46ff-82df-7a3f73929cc8","Type":"ContainerStarted","Data":"330b5530283ec52dbca78f56ff5180df87fb583b96680f101050db58829f0065"} Oct 30 19:40:25.242190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.242057 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" Oct 30 19:40:25.243525 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.243501 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" event={"ID":"550f7a23-825c-45dc-a2bf-ee67264eed56","Type":"ContainerStarted","Data":"114b0b4e66909979829e312a1d32dc13b65850c5d82c2c353d7db0f745737e4c"} Oct 30 19:40:25.243654 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.243548 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" Oct 30 19:40:25.245036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.245017 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" event={"ID":"8ec2b758-fec8-49ce-a483-e3f91af11c33","Type":"ContainerStarted","Data":"9fb60a995645282e04d820d3f827bd708a35e3565fa6d00ad14abbb4a0cae8d6"} Oct 30 19:40:25.245121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.245060 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" Oct 30 19:40:25.255120 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.255091 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" event={"ID":"91802908-1257-4dc4-8e5b-baaeb4dbdfbb","Type":"ContainerStarted","Data":"f91b86465c1afc447ffd5907302fef0a12b3f1ea329c04363ab3dfa0dafa8d65"} Oct 30 19:40:25.255265 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.255190 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" Oct 30 19:40:25.257856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.257829 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" event={"ID":"6733505d-3dd3-462d-a96a-ee60c3764e64","Type":"ContainerStarted","Data":"a413811902b0492fb459760fac0dd802ef2b8d2b9258c3c700fad3ca0fa4892c"} Oct 30 19:40:25.258199 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.258181 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" Oct 30 19:40:25.259587 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.259566 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" event={"ID":"8f46b66e-1ca4-41ca-9be7-040cb8dec18a","Type":"ContainerStarted","Data":"92c85cbef00fddb14f9abe1d8451dd0ba232b9a2a40d2967fa85dafbb0452f6f"} Oct 30 19:40:25.259693 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.259665 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" Oct 30 19:40:25.277479 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.277379 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" podStartSLOduration=1.549792522 podStartE2EDuration="6.277360923s" totalImagesPullingTime="4.727568401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:25.275292924 +0000 UTC m=+544.970500883" watchObservedRunningTime="2026-10-30 19:40:25.277360923 +0000 UTC m=+544.972568880" Oct 30 19:40:25.344771 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.344705 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" podStartSLOduration=2.658920066 podStartE2EDuration="7.344686744s" totalImagesPullingTime="4.685766678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:25.337699484 +0000 UTC m=+545.032907443" watchObservedRunningTime="2026-10-30 19:40:25.344686744 +0000 UTC m=+545.039894705" Oct 30 19:40:25.345900 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.345848 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" podStartSLOduration=2.341844499 podStartE2EDuration="7.345836884s" totalImagesPullingTime="5.003992385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:25.314531496 +0000 UTC m=+545.009739455" watchObservedRunningTime="2026-10-30 19:40:25.345836884 +0000 UTC m=+545.041044842" Oct 30 19:40:25.362089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.362039 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" podStartSLOduration=1.62726355 podStartE2EDuration="6.362031757s" totalImagesPullingTime="4.734768207s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:25.359864529 +0000 UTC m=+545.055072511" watchObservedRunningTime="2026-10-30 19:40:25.362031757 +0000 UTC m=+545.057239715" Oct 30 19:40:25.385230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.385180 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" podStartSLOduration=2.4979480499999998 podStartE2EDuration="5.385165831s" totalImagesPullingTime="2.887217781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:25.38265534 +0000 UTC m=+545.077863298" watchObservedRunningTime="2026-10-30 19:40:25.385165831 +0000 UTC m=+545.080373789" Oct 30 19:40:25.411124 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.411059 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" podStartSLOduration=2.531028031 podStartE2EDuration="5.411040455s" totalImagesPullingTime="2.880012424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:25.40982292 +0000 UTC m=+545.105030878" watchObservedRunningTime="2026-10-30 19:40:25.411040455 +0000 UTC m=+545.106248416" Oct 30 19:40:25.726037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.725960 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:25.871607 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:25.871561 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:26.032722 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:26.032691 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:26.266766 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:26.266730 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:26.267030 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:26.267006 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:26.385829 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:26.385753 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:27.203729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:27.203691 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:27.417211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:27.414678 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:28.140203 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:28.140164 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:28.985063 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:28.985024 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:29.877318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:29.877129 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:30.111528 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:30.111486 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:30.256930 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:30.256900 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:30.792036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:30.791999 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:31.810172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:31.809861 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:32.234273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.234198 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf"] Oct 30 19:40:32.239337 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.239319 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" Oct 30 19:40:32.255195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.254316 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:40:32.262171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.259311 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:40:32.282181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.266095 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf"] Oct 30 19:40:32.392711 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.392671 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1d482894-grcnf\" (UID: \"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" Oct 30 19:40:32.392711 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.392716 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnt7k\" (UniqueName: \"kubernetes.io/projected/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-kube-api-access-bnt7k\") pod \"maas-api-revoke-keys-e2e-migrate-1d482894-grcnf\" (UID: \"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" Oct 30 19:40:32.392964 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.392823 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1d482894-grcnf\" (UID: \"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" Oct 30 19:40:32.493588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.493503 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1d482894-grcnf\" (UID: \"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" Oct 30 19:40:32.493588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.493555 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnt7k\" (UniqueName: \"kubernetes.io/projected/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-kube-api-access-bnt7k\") pod \"maas-api-revoke-keys-e2e-migrate-1d482894-grcnf\" (UID: \"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" Oct 30 19:40:32.493806 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.493640 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1d482894-grcnf\" (UID: \"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" Oct 30 19:40:32.494340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.494312 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1d482894-grcnf\" (UID: \"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" Oct 30 19:40:32.495884 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.495862 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-1d482894-grcnf\" (UID: \"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" Oct 30 19:40:32.507787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.507764 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnt7k\" (UniqueName: \"kubernetes.io/projected/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-kube-api-access-bnt7k\") pod \"maas-api-revoke-keys-e2e-migrate-1d482894-grcnf\" (UID: \"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" Oct 30 19:40:32.590959 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.590928 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" Oct 30 19:40:32.868925 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.868828 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:32.918020 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:32.917971 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf"] Oct 30 19:40:32.922069 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:32.922023 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce0f96ef_166c_4b4c_97fb_d08beb3b36aa.slice/crio-1f9fb3b9287b9c0e7adc1b770f02fe113e49bfb976bc4cbb146ec7a0cdc2aa95 WatchSource:0}: Error finding container 1f9fb3b9287b9c0e7adc1b770f02fe113e49bfb976bc4cbb146ec7a0cdc2aa95: Status 404 returned error can't find the container with id 1f9fb3b9287b9c0e7adc1b770f02fe113e49bfb976bc4cbb146ec7a0cdc2aa95 Oct 30 19:40:33.206359 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:33.206276 5091 patch_prober.go:29] interesting pod/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:40:33.206359 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:33.206329 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" podUID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:33.332439 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:33.332401 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" event={"ID":"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa","Type":"ContainerStarted","Data":"1f9fb3b9287b9c0e7adc1b770f02fe113e49bfb976bc4cbb146ec7a0cdc2aa95"} Oct 30 19:40:33.921883 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:33.921847 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:34.946624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:34.943692 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:36.006558 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:36.006489 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:36.285004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:36.284911 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" Oct 30 19:40:36.285337 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:36.285315 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" Oct 30 19:40:36.286126 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:36.286106 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" Oct 30 19:40:36.288348 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:36.288327 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" Oct 30 19:40:36.289408 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:36.289003 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" Oct 30 19:40:36.292382 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:36.292361 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" Oct 30 19:40:36.978202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:36.969754 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:37.414428 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.414390 5091 generic.go:356] "Generic (PLEG): container finished" podID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerID="ca9f3c187630b8610dfd3333be4cd7315d86c363552b1a47e09f5bc6a311353f" exitCode=0 Oct 30 19:40:37.414837 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.414466 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" event={"ID":"8253e5fe-3ea9-4367-b43d-274dc85176f8","Type":"ContainerDied","Data":"ca9f3c187630b8610dfd3333be4cd7315d86c363552b1a47e09f5bc6a311353f"} Oct 30 19:40:37.414982 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.414963 5091 scope.go:122] "RemoveContainer" containerID="ca9f3c187630b8610dfd3333be4cd7315d86c363552b1a47e09f5bc6a311353f" Oct 30 19:40:37.416706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.416671 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:37.416848 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.416762 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:37.416848 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.416813 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:37.505643 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.503190 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2"] Oct 30 19:40:37.509778 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.509190 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" Oct 30 19:40:37.515109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.515078 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2"] Oct 30 19:40:37.627208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.627179 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7022c039-2eb1-4cb5-87b1-71d26169c4fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3a55af38-k76v2\" (UID: \"7022c039-2eb1-4cb5-87b1-71d26169c4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" Oct 30 19:40:37.627387 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.627268 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r96g5\" (UniqueName: \"kubernetes.io/projected/7022c039-2eb1-4cb5-87b1-71d26169c4fe-kube-api-access-r96g5\") pod \"maas-api-revoke-keys-e2e-ait-3a55af38-k76v2\" (UID: \"7022c039-2eb1-4cb5-87b1-71d26169c4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" Oct 30 19:40:37.627456 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.627435 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7022c039-2eb1-4cb5-87b1-71d26169c4fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3a55af38-k76v2\" (UID: \"7022c039-2eb1-4cb5-87b1-71d26169c4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" Oct 30 19:40:37.728898 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.728865 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7022c039-2eb1-4cb5-87b1-71d26169c4fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3a55af38-k76v2\" (UID: \"7022c039-2eb1-4cb5-87b1-71d26169c4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" Oct 30 19:40:37.729100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.728962 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7022c039-2eb1-4cb5-87b1-71d26169c4fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3a55af38-k76v2\" (UID: \"7022c039-2eb1-4cb5-87b1-71d26169c4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" Oct 30 19:40:37.729100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.729023 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r96g5\" (UniqueName: \"kubernetes.io/projected/7022c039-2eb1-4cb5-87b1-71d26169c4fe-kube-api-access-r96g5\") pod \"maas-api-revoke-keys-e2e-ait-3a55af38-k76v2\" (UID: \"7022c039-2eb1-4cb5-87b1-71d26169c4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" Oct 30 19:40:37.730233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.730202 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7022c039-2eb1-4cb5-87b1-71d26169c4fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-3a55af38-k76v2\" (UID: \"7022c039-2eb1-4cb5-87b1-71d26169c4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" Oct 30 19:40:37.731916 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.731881 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7022c039-2eb1-4cb5-87b1-71d26169c4fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-3a55af38-k76v2\" (UID: \"7022c039-2eb1-4cb5-87b1-71d26169c4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" Oct 30 19:40:37.760179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.743726 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r96g5\" (UniqueName: \"kubernetes.io/projected/7022c039-2eb1-4cb5-87b1-71d26169c4fe-kube-api-access-r96g5\") pod \"maas-api-revoke-keys-e2e-ait-3a55af38-k76v2\" (UID: \"7022c039-2eb1-4cb5-87b1-71d26169c4fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" Oct 30 19:40:37.830735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:37.830702 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" Oct 30 19:40:38.091957 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.091764 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:38.186898 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.180599 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2"] Oct 30 19:40:38.189098 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:38.189061 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7022c039_2eb1_4cb5_87b1_71d26169c4fe.slice/crio-f1d77c0a878a23f64aaa36b17550f93cfd83620df2fea562b11eeb6d894ecc97 WatchSource:0}: Error finding container f1d77c0a878a23f64aaa36b17550f93cfd83620df2fea562b11eeb6d894ecc97: Status 404 returned error can't find the container with id f1d77c0a878a23f64aaa36b17550f93cfd83620df2fea562b11eeb6d894ecc97 Oct 30 19:40:38.397409 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.397370 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf"] Oct 30 19:40:38.402772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.402752 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" Oct 30 19:40:38.411665 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.411340 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf"] Oct 30 19:40:38.421601 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.421223 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" event={"ID":"7022c039-2eb1-4cb5-87b1-71d26169c4fe","Type":"ContainerStarted","Data":"f1d77c0a878a23f64aaa36b17550f93cfd83620df2fea562b11eeb6d894ecc97"} Oct 30 19:40:38.423168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.423130 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" event={"ID":"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa","Type":"ContainerStarted","Data":"b4936a27ae9c0f07e3ade20449226484147072d19395053391297f1bd5c22007"} Oct 30 19:40:38.427400 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.427379 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" event={"ID":"8253e5fe-3ea9-4367-b43d-274dc85176f8","Type":"ContainerStarted","Data":"b582102bd1cdaa6e45a54f89d214bc3a335fb312298d96f6b2e2f4e805735cc6"} Oct 30 19:40:38.474531 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.447037 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" podStartSLOduration=2.031840044 podStartE2EDuration="6.447018841s" totalImagesPullingTime="4.415178797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:38.444117908 +0000 UTC m=+558.139325878" watchObservedRunningTime="2026-10-30 19:40:38.447018841 +0000 UTC m=+558.142226798" Oct 30 19:40:38.544172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.540643 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jbqn\" (UniqueName: \"kubernetes.io/projected/a00fed1d-f03c-4656-aad2-1f07ac428e68-kube-api-access-7jbqn\") pod \"maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf\" (UID: \"a00fed1d-f03c-4656-aad2-1f07ac428e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" Oct 30 19:40:38.544172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.540830 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00fed1d-f03c-4656-aad2-1f07ac428e68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf\" (UID: \"a00fed1d-f03c-4656-aad2-1f07ac428e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" Oct 30 19:40:38.544172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.540978 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00fed1d-f03c-4656-aad2-1f07ac428e68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf\" (UID: \"a00fed1d-f03c-4656-aad2-1f07ac428e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" Oct 30 19:40:38.642080 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.642044 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jbqn\" (UniqueName: \"kubernetes.io/projected/a00fed1d-f03c-4656-aad2-1f07ac428e68-kube-api-access-7jbqn\") pod \"maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf\" (UID: \"a00fed1d-f03c-4656-aad2-1f07ac428e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" Oct 30 19:40:38.642248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.642111 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00fed1d-f03c-4656-aad2-1f07ac428e68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf\" (UID: \"a00fed1d-f03c-4656-aad2-1f07ac428e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" Oct 30 19:40:38.642248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.642168 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00fed1d-f03c-4656-aad2-1f07ac428e68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf\" (UID: \"a00fed1d-f03c-4656-aad2-1f07ac428e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" Oct 30 19:40:38.642747 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.642725 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00fed1d-f03c-4656-aad2-1f07ac428e68-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf\" (UID: \"a00fed1d-f03c-4656-aad2-1f07ac428e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" Oct 30 19:40:38.645122 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.645099 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00fed1d-f03c-4656-aad2-1f07ac428e68-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf\" (UID: \"a00fed1d-f03c-4656-aad2-1f07ac428e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" Oct 30 19:40:38.668547 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.659946 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jbqn\" (UniqueName: \"kubernetes.io/projected/a00fed1d-f03c-4656-aad2-1f07ac428e68-kube-api-access-7jbqn\") pod \"maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf\" (UID: \"a00fed1d-f03c-4656-aad2-1f07ac428e68\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" Oct 30 19:40:38.695164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.694992 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:38.696178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.696041 5091 patch_prober.go:29] interesting pod/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:40:38.696178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.696093 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" podUID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 19:40:38.721685 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:38.721432 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" Oct 30 19:40:39.017325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:39.017228 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:39.101766 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:39.101733 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf"] Oct 30 19:40:39.104211 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:39.104176 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda00fed1d_f03c_4656_aad2_1f07ac428e68.slice/crio-1eecf0593f962a436bc5d2dc395206f8bba0fd87edd79e8093676020a0c6c4ae WatchSource:0}: Error finding container 1eecf0593f962a436bc5d2dc395206f8bba0fd87edd79e8093676020a0c6c4ae: Status 404 returned error can't find the container with id 1eecf0593f962a436bc5d2dc395206f8bba0fd87edd79e8093676020a0c6c4ae Oct 30 19:40:39.434564 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:39.434494 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" event={"ID":"a00fed1d-f03c-4656-aad2-1f07ac428e68","Type":"ContainerStarted","Data":"1eecf0593f962a436bc5d2dc395206f8bba0fd87edd79e8093676020a0c6c4ae"} Oct 30 19:40:39.436381 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:39.435993 5091 generic.go:356] "Generic (PLEG): container finished" podID="7022c039-2eb1-4cb5-87b1-71d26169c4fe" containerID="4454abf83f27e674e129398ce97db53c9bd0ebc7eee229b0636b68550ec1ee39" exitCode=0 Oct 30 19:40:39.436381 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:39.436322 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" event={"ID":"7022c039-2eb1-4cb5-87b1-71d26169c4fe","Type":"ContainerDied","Data":"4454abf83f27e674e129398ce97db53c9bd0ebc7eee229b0636b68550ec1ee39"} Oct 30 19:40:39.438286 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:39.437569 5091 generic.go:356] "Generic (PLEG): container finished" podID="ce0f96ef-166c-4b4c-97fb-d08beb3b36aa" containerID="b4936a27ae9c0f07e3ade20449226484147072d19395053391297f1bd5c22007" exitCode=0 Oct 30 19:40:39.438286 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:39.438241 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" event={"ID":"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa","Type":"ContainerDied","Data":"b4936a27ae9c0f07e3ade20449226484147072d19395053391297f1bd5c22007"} Oct 30 19:40:39.701269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:39.694537 5091 patch_prober.go:29] interesting pod/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:40:39.701269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:39.694603 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" podUID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 19:40:39.736176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:39.736130 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr"] Oct 30 19:40:39.740550 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:39.740518 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" Oct 30 19:40:39.789168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:39.769451 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr"] Oct 30 19:40:39.915189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:39.909745 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53cf7ed8-9b51-491c-802f-9c1a0da29378-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr\" (UID: \"53cf7ed8-9b51-491c-802f-9c1a0da29378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" Oct 30 19:40:39.915635 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:39.915609 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2t66\" (UniqueName: \"kubernetes.io/projected/53cf7ed8-9b51-491c-802f-9c1a0da29378-kube-api-access-r2t66\") pod \"maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr\" (UID: \"53cf7ed8-9b51-491c-802f-9c1a0da29378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" Oct 30 19:40:39.916690 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:39.916506 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53cf7ed8-9b51-491c-802f-9c1a0da29378-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr\" (UID: \"53cf7ed8-9b51-491c-802f-9c1a0da29378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" Oct 30 19:40:40.018066 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.018020 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53cf7ed8-9b51-491c-802f-9c1a0da29378-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr\" (UID: \"53cf7ed8-9b51-491c-802f-9c1a0da29378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" Oct 30 19:40:40.018433 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.018405 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53cf7ed8-9b51-491c-802f-9c1a0da29378-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr\" (UID: \"53cf7ed8-9b51-491c-802f-9c1a0da29378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" Oct 30 19:40:40.019506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.019126 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2t66\" (UniqueName: \"kubernetes.io/projected/53cf7ed8-9b51-491c-802f-9c1a0da29378-kube-api-access-r2t66\") pod \"maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr\" (UID: \"53cf7ed8-9b51-491c-802f-9c1a0da29378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" Oct 30 19:40:40.019690 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.019272 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53cf7ed8-9b51-491c-802f-9c1a0da29378-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr\" (UID: \"53cf7ed8-9b51-491c-802f-9c1a0da29378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" Oct 30 19:40:40.022886 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.022862 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53cf7ed8-9b51-491c-802f-9c1a0da29378-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr\" (UID: \"53cf7ed8-9b51-491c-802f-9c1a0da29378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" Oct 30 19:40:40.038884 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.037834 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2t66\" (UniqueName: \"kubernetes.io/projected/53cf7ed8-9b51-491c-802f-9c1a0da29378-kube-api-access-r2t66\") pod \"maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr\" (UID: \"53cf7ed8-9b51-491c-802f-9c1a0da29378\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" Oct 30 19:40:40.112014 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.111888 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:40.121808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.120955 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" Oct 30 19:40:40.501119 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.501075 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr"] Oct 30 19:40:40.503072 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.502860 5091 generic.go:356] "Generic (PLEG): container finished" podID="a00fed1d-f03c-4656-aad2-1f07ac428e68" containerID="f5621783477ffc2c474780bf33e695b6bbb35025506eab5410695fd0da603f3a" exitCode=0 Oct 30 19:40:40.503334 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.503309 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" event={"ID":"a00fed1d-f03c-4656-aad2-1f07ac428e68","Type":"ContainerDied","Data":"f5621783477ffc2c474780bf33e695b6bbb35025506eab5410695fd0da603f3a"} Oct 30 19:40:40.696070 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.695556 5091 patch_prober.go:29] interesting pod/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:40:40.696070 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.695623 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" podUID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 19:40:40.793033 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.792994 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" Oct 30 19:40:40.838459 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.837310 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" Oct 30 19:40:40.935639 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.935605 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-maas-api-xks-ca\") pod \"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa\" (UID: \"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa\") " Oct 30 19:40:40.935765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.935659 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-maas-api-openshift-ca\") pod \"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa\" (UID: \"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa\") " Oct 30 19:40:40.935765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.935742 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7022c039-2eb1-4cb5-87b1-71d26169c4fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7022c039-2eb1-4cb5-87b1-71d26169c4fe-maas-api-openshift-ca\") pod \"7022c039-2eb1-4cb5-87b1-71d26169c4fe\" (UID: \"7022c039-2eb1-4cb5-87b1-71d26169c4fe\") " Oct 30 19:40:40.935881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.935817 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7022c039-2eb1-4cb5-87b1-71d26169c4fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7022c039-2eb1-4cb5-87b1-71d26169c4fe-maas-api-xks-ca\") pod \"7022c039-2eb1-4cb5-87b1-71d26169c4fe\" (UID: \"7022c039-2eb1-4cb5-87b1-71d26169c4fe\") " Oct 30 19:40:40.935935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.935903 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-kube-api-access-bnt7k\" (UniqueName: \"kubernetes.io/projected/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-kube-api-access-bnt7k\") pod \"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa\" (UID: \"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa\") " Oct 30 19:40:40.935995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.935954 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7022c039-2eb1-4cb5-87b1-71d26169c4fe-kube-api-access-r96g5\" (UniqueName: \"kubernetes.io/projected/7022c039-2eb1-4cb5-87b1-71d26169c4fe-kube-api-access-r96g5\") pod \"7022c039-2eb1-4cb5-87b1-71d26169c4fe\" (UID: \"7022c039-2eb1-4cb5-87b1-71d26169c4fe\") " Oct 30 19:40:40.936715 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.936675 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7022c039-2eb1-4cb5-87b1-71d26169c4fe-maas-api-openshift-ca" pod "7022c039-2eb1-4cb5-87b1-71d26169c4fe" (UID: "7022c039-2eb1-4cb5-87b1-71d26169c4fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:40.936991 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.936959 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-maas-api-openshift-ca" pod "ce0f96ef-166c-4b4c-97fb-d08beb3b36aa" (UID: "ce0f96ef-166c-4b4c-97fb-d08beb3b36aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:40.938899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.938875 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7022c039-2eb1-4cb5-87b1-71d26169c4fe-maas-api-xks-ca" pod "7022c039-2eb1-4cb5-87b1-71d26169c4fe" (UID: "7022c039-2eb1-4cb5-87b1-71d26169c4fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:40.939001 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.938900 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-kube-api-access-bnt7k" pod "ce0f96ef-166c-4b4c-97fb-d08beb3b36aa" (UID: "ce0f96ef-166c-4b4c-97fb-d08beb3b36aa"). InnerVolumeSpecName "kube-api-access-bnt7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:40.939312 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.939280 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7022c039-2eb1-4cb5-87b1-71d26169c4fe-kube-api-access-r96g5" pod "7022c039-2eb1-4cb5-87b1-71d26169c4fe" (UID: "7022c039-2eb1-4cb5-87b1-71d26169c4fe"). InnerVolumeSpecName "kube-api-access-r96g5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:40.939312 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:40.939289 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-maas-api-xks-ca" pod "ce0f96ef-166c-4b4c-97fb-d08beb3b36aa" (UID: "ce0f96ef-166c-4b4c-97fb-d08beb3b36aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:41.037395 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.037310 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7022c039-2eb1-4cb5-87b1-71d26169c4fe-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:41.037395 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.037357 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnt7k\" (UniqueName: \"kubernetes.io/projected/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-kube-api-access-bnt7k\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:41.037395 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.037377 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r96g5\" (UniqueName: \"kubernetes.io/projected/7022c039-2eb1-4cb5-87b1-71d26169c4fe-kube-api-access-r96g5\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:41.037395 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.037392 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:41.037626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.037407 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:41.037626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.037422 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7022c039-2eb1-4cb5-87b1-71d26169c4fe-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:41.188803 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.182673 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:41.510703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.510669 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" event={"ID":"53cf7ed8-9b51-491c-802f-9c1a0da29378","Type":"ContainerStarted","Data":"58d105fb956464b97eaf5120956b753bc27d9f770c09ad72ddd9017a71b62e99"} Oct 30 19:40:41.512870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.510722 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" event={"ID":"53cf7ed8-9b51-491c-802f-9c1a0da29378","Type":"ContainerStarted","Data":"3ccb81ebb127d09addde9950556f6ea2ec432cfe353fa94faa43ace8d435ec6d"} Oct 30 19:40:41.512870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.512801 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" Oct 30 19:40:41.512870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.512815 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2" event={"ID":"7022c039-2eb1-4cb5-87b1-71d26169c4fe","Type":"ContainerDied","Data":"f1d77c0a878a23f64aaa36b17550f93cfd83620df2fea562b11eeb6d894ecc97"} Oct 30 19:40:41.512870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.512851 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1d77c0a878a23f64aaa36b17550f93cfd83620df2fea562b11eeb6d894ecc97" Oct 30 19:40:41.514332 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.514313 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" Oct 30 19:40:41.516241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.516217 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf" event={"ID":"ce0f96ef-166c-4b4c-97fb-d08beb3b36aa","Type":"ContainerDied","Data":"1f9fb3b9287b9c0e7adc1b770f02fe113e49bfb976bc4cbb146ec7a0cdc2aa95"} Oct 30 19:40:41.516345 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.516263 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f9fb3b9287b9c0e7adc1b770f02fe113e49bfb976bc4cbb146ec7a0cdc2aa95" Oct 30 19:40:41.537315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.537265 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" podStartSLOduration=2.537205673 podStartE2EDuration="2.537205673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:41.534624317 +0000 UTC m=+561.229832277" watchObservedRunningTime="2026-10-30 19:40:41.537205673 +0000 UTC m=+561.232413636" Oct 30 19:40:41.697255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.696657 5091 patch_prober.go:29] interesting pod/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:40:41.697255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.696704 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" podUID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 19:40:41.735368 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.735346 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" Oct 30 19:40:41.784358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.782540 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a00fed1d-f03c-4656-aad2-1f07ac428e68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00fed1d-f03c-4656-aad2-1f07ac428e68-maas-api-xks-ca\") pod \"a00fed1d-f03c-4656-aad2-1f07ac428e68\" (UID: \"a00fed1d-f03c-4656-aad2-1f07ac428e68\") " Oct 30 19:40:41.784358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.782680 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a00fed1d-f03c-4656-aad2-1f07ac428e68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00fed1d-f03c-4656-aad2-1f07ac428e68-maas-api-openshift-ca\") pod \"a00fed1d-f03c-4656-aad2-1f07ac428e68\" (UID: \"a00fed1d-f03c-4656-aad2-1f07ac428e68\") " Oct 30 19:40:41.784358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.782758 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a00fed1d-f03c-4656-aad2-1f07ac428e68-kube-api-access-7jbqn\" (UniqueName: \"kubernetes.io/projected/a00fed1d-f03c-4656-aad2-1f07ac428e68-kube-api-access-7jbqn\") pod \"a00fed1d-f03c-4656-aad2-1f07ac428e68\" (UID: \"a00fed1d-f03c-4656-aad2-1f07ac428e68\") " Oct 30 19:40:41.784358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.784030 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a00fed1d-f03c-4656-aad2-1f07ac428e68-maas-api-openshift-ca" pod "a00fed1d-f03c-4656-aad2-1f07ac428e68" (UID: "a00fed1d-f03c-4656-aad2-1f07ac428e68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:41.787770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.787743 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a00fed1d-f03c-4656-aad2-1f07ac428e68-kube-api-access-7jbqn" pod "a00fed1d-f03c-4656-aad2-1f07ac428e68" (UID: "a00fed1d-f03c-4656-aad2-1f07ac428e68"). InnerVolumeSpecName "kube-api-access-7jbqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:41.788001 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.787933 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a00fed1d-f03c-4656-aad2-1f07ac428e68-maas-api-xks-ca" pod "a00fed1d-f03c-4656-aad2-1f07ac428e68" (UID: "a00fed1d-f03c-4656-aad2-1f07ac428e68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:41.884697 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.884646 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jbqn\" (UniqueName: \"kubernetes.io/projected/a00fed1d-f03c-4656-aad2-1f07ac428e68-kube-api-access-7jbqn\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:41.884976 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.884959 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a00fed1d-f03c-4656-aad2-1f07ac428e68-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:41.885100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:41.885087 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a00fed1d-f03c-4656-aad2-1f07ac428e68-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:42.173437 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.167651 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:42.309381 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.309334 5091 patch_prober.go:29] interesting pod/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:40:42.309556 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.309415 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" podUID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:40:42.499045 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.499012 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:42.525580 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.525539 5091 generic.go:356] "Generic (PLEG): container finished" podID="d9cb9939-3547-4232-b6c0-d2a40803a666" containerID="fefd57fd8b93e0f45f5d08c0c0e949e413a155fdede32c576f4403595fafd81d" exitCode=0 Oct 30 19:40:42.526058 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.525600 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" event={"ID":"d9cb9939-3547-4232-b6c0-d2a40803a666","Type":"ContainerDied","Data":"fefd57fd8b93e0f45f5d08c0c0e949e413a155fdede32c576f4403595fafd81d"} Oct 30 19:40:42.527920 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.527892 5091 generic.go:356] "Generic (PLEG): container finished" podID="53cf7ed8-9b51-491c-802f-9c1a0da29378" containerID="58d105fb956464b97eaf5120956b753bc27d9f770c09ad72ddd9017a71b62e99" exitCode=0 Oct 30 19:40:42.528065 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.527993 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" event={"ID":"53cf7ed8-9b51-491c-802f-9c1a0da29378","Type":"ContainerDied","Data":"58d105fb956464b97eaf5120956b753bc27d9f770c09ad72ddd9017a71b62e99"} Oct 30 19:40:42.528520 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.528494 5091 scope.go:122] "RemoveContainer" containerID="fefd57fd8b93e0f45f5d08c0c0e949e413a155fdede32c576f4403595fafd81d" Oct 30 19:40:42.530345 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.530317 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:42.530435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.530381 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:42.530435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.530414 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:42.531446 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.531421 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" event={"ID":"a00fed1d-f03c-4656-aad2-1f07ac428e68","Type":"ContainerDied","Data":"1eecf0593f962a436bc5d2dc395206f8bba0fd87edd79e8093676020a0c6c4ae"} Oct 30 19:40:42.531446 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.531441 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf" Oct 30 19:40:42.531590 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.531467 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1eecf0593f962a436bc5d2dc395206f8bba0fd87edd79e8093676020a0c6c4ae" Oct 30 19:40:42.679654 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.679307 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:42.696164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.695116 5091 patch_prober.go:29] interesting pod/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:40:42.696164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:42.695231 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" podUID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 19:40:43.213942 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:43.213902 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:43.540645 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:43.540612 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" event={"ID":"d9cb9939-3547-4232-b6c0-d2a40803a666","Type":"ContainerStarted","Data":"d28cf35e02c44c0cdacaa0a9a423dd3bf0e7c2109c891d72f2087520d1effe42"} Oct 30 19:40:43.696170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:43.695589 5091 patch_prober.go:29] interesting pod/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body= Oct 30 19:40:43.696170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:43.695648 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" podUID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" Oct 30 19:40:43.836946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:43.836923 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" Oct 30 19:40:43.919485 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:43.919170 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53cf7ed8-9b51-491c-802f-9c1a0da29378-kube-api-access-r2t66\" (UniqueName: \"kubernetes.io/projected/53cf7ed8-9b51-491c-802f-9c1a0da29378-kube-api-access-r2t66\") pod \"53cf7ed8-9b51-491c-802f-9c1a0da29378\" (UID: \"53cf7ed8-9b51-491c-802f-9c1a0da29378\") " Oct 30 19:40:43.919485 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:43.919226 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53cf7ed8-9b51-491c-802f-9c1a0da29378-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53cf7ed8-9b51-491c-802f-9c1a0da29378-maas-api-openshift-ca\") pod \"53cf7ed8-9b51-491c-802f-9c1a0da29378\" (UID: \"53cf7ed8-9b51-491c-802f-9c1a0da29378\") " Oct 30 19:40:43.919485 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:43.919306 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53cf7ed8-9b51-491c-802f-9c1a0da29378-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53cf7ed8-9b51-491c-802f-9c1a0da29378-maas-api-xks-ca\") pod \"53cf7ed8-9b51-491c-802f-9c1a0da29378\" (UID: \"53cf7ed8-9b51-491c-802f-9c1a0da29378\") " Oct 30 19:40:43.919863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:43.919596 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53cf7ed8-9b51-491c-802f-9c1a0da29378-maas-api-openshift-ca" pod "53cf7ed8-9b51-491c-802f-9c1a0da29378" (UID: "53cf7ed8-9b51-491c-802f-9c1a0da29378"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:43.919863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:43.919769 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53cf7ed8-9b51-491c-802f-9c1a0da29378-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:43.921935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:43.921873 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cf7ed8-9b51-491c-802f-9c1a0da29378-kube-api-access-r2t66" pod "53cf7ed8-9b51-491c-802f-9c1a0da29378" (UID: "53cf7ed8-9b51-491c-802f-9c1a0da29378"). InnerVolumeSpecName "kube-api-access-r2t66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:43.925440 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:43.925411 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53cf7ed8-9b51-491c-802f-9c1a0da29378-maas-api-xks-ca" pod "53cf7ed8-9b51-491c-802f-9c1a0da29378" (UID: "53cf7ed8-9b51-491c-802f-9c1a0da29378"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:44.023778 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.023719 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2t66\" (UniqueName: \"kubernetes.io/projected/53cf7ed8-9b51-491c-802f-9c1a0da29378-kube-api-access-r2t66\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:44.023778 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.023761 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53cf7ed8-9b51-491c-802f-9c1a0da29378-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:44.142842 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.142750 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:44.310486 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.310450 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:44.311990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.311963 5091 patch_prober.go:29] interesting pod/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:40:44.312125 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.312020 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" podUID="d9cb9939-3547-4232-b6c0-d2a40803a666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 19:40:44.378580 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.378535 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:44.432361 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.431659 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf"] Oct 30 19:40:44.441937 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.441914 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-1d482894-grcnf"] Oct 30 19:40:44.539207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.539168 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:44.551128 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.551100 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" Oct 30 19:40:44.552350 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.551738 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr" event={"ID":"53cf7ed8-9b51-491c-802f-9c1a0da29378","Type":"ContainerDied","Data":"3ccb81ebb127d09addde9950556f6ea2ec432cfe353fa94faa43ace8d435ec6d"} Oct 30 19:40:44.552678 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.552662 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ccb81ebb127d09addde9950556f6ea2ec432cfe353fa94faa43ace8d435ec6d" Oct 30 19:40:44.699410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.699313 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:44.703067 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.701262 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:44.716826 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.716673 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:44.720993 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:44.720920 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:45.310756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.310707 5091 patch_prober.go:29] interesting pod/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:40:45.310937 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.310787 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" podUID="d9cb9939-3547-4232-b6c0-d2a40803a666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 19:40:45.544420 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.544377 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:45.692056 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.692023 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k"] Oct 30 19:40:45.696875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.692596 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7022c039-2eb1-4cb5-87b1-71d26169c4fe" containerName="revoke-keys" Oct 30 19:40:45.696875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.692612 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7022c039-2eb1-4cb5-87b1-71d26169c4fe" containerName="revoke-keys" Oct 30 19:40:45.696875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.692629 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ce0f96ef-166c-4b4c-97fb-d08beb3b36aa" containerName="revoke-keys" Oct 30 19:40:45.696875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.692636 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce0f96ef-166c-4b4c-97fb-d08beb3b36aa" containerName="revoke-keys" Oct 30 19:40:45.696875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.692645 5091 cpu_manager.go:414] "containerMap: removing container" podUID="53cf7ed8-9b51-491c-802f-9c1a0da29378" containerName="revoke-keys" Oct 30 19:40:45.696875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.692650 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53cf7ed8-9b51-491c-802f-9c1a0da29378" containerName="revoke-keys" Oct 30 19:40:45.696875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.692658 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a00fed1d-f03c-4656-aad2-1f07ac428e68" containerName="revoke-keys" Oct 30 19:40:45.696875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.692663 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a00fed1d-f03c-4656-aad2-1f07ac428e68" containerName="revoke-keys" Oct 30 19:40:45.696875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.692740 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7022c039-2eb1-4cb5-87b1-71d26169c4fe" containerName="revoke-keys" Oct 30 19:40:45.696875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.692748 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce0f96ef-166c-4b4c-97fb-d08beb3b36aa" containerName="revoke-keys" Oct 30 19:40:45.696875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.692755 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="53cf7ed8-9b51-491c-802f-9c1a0da29378" containerName="revoke-keys" Oct 30 19:40:45.696875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.692762 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a00fed1d-f03c-4656-aad2-1f07ac428e68" containerName="revoke-keys" Oct 30 19:40:45.696875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.695838 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:45.697748 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.697727 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" Oct 30 19:40:45.705628 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.705602 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k"] Oct 30 19:40:45.713821 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.713701 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:40:45.713821 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.713798 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:40:45.744637 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.744610 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efad46d4-e7ef-4139-9397-7d1b56faec0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k\" (UID: \"efad46d4-e7ef-4139-9397-7d1b56faec0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" Oct 30 19:40:45.744751 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.744666 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efad46d4-e7ef-4139-9397-7d1b56faec0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k\" (UID: \"efad46d4-e7ef-4139-9397-7d1b56faec0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" Oct 30 19:40:45.744837 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.744817 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dk2f\" (UniqueName: \"kubernetes.io/projected/efad46d4-e7ef-4139-9397-7d1b56faec0d-kube-api-access-7dk2f\") pod \"maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k\" (UID: \"efad46d4-e7ef-4139-9397-7d1b56faec0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" Oct 30 19:40:45.870160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.868244 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efad46d4-e7ef-4139-9397-7d1b56faec0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k\" (UID: \"efad46d4-e7ef-4139-9397-7d1b56faec0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" Oct 30 19:40:45.870160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.868350 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efad46d4-e7ef-4139-9397-7d1b56faec0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k\" (UID: \"efad46d4-e7ef-4139-9397-7d1b56faec0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" Oct 30 19:40:45.870160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.868498 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dk2f\" (UniqueName: \"kubernetes.io/projected/efad46d4-e7ef-4139-9397-7d1b56faec0d-kube-api-access-7dk2f\") pod \"maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k\" (UID: \"efad46d4-e7ef-4139-9397-7d1b56faec0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" Oct 30 19:40:45.870160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.870055 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efad46d4-e7ef-4139-9397-7d1b56faec0d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k\" (UID: \"efad46d4-e7ef-4139-9397-7d1b56faec0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" Oct 30 19:40:45.877725 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.877697 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efad46d4-e7ef-4139-9397-7d1b56faec0d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k\" (UID: \"efad46d4-e7ef-4139-9397-7d1b56faec0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" Oct 30 19:40:45.897854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.897797 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dk2f\" (UniqueName: \"kubernetes.io/projected/efad46d4-e7ef-4139-9397-7d1b56faec0d-kube-api-access-7dk2f\") pod \"maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k\" (UID: \"efad46d4-e7ef-4139-9397-7d1b56faec0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" Oct 30 19:40:45.916925 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.916888 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2"] Oct 30 19:40:45.957032 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.954711 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2"] Oct 30 19:40:45.957032 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.955036 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:45.982626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:45.982597 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-c1fdd6-openshift-default-dockercfg-7mt7d\"" Oct 30 19:40:46.010347 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.010314 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:46.038616 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:40:46.038581 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:40:46.043242 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.043203 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" Oct 30 19:40:46.106224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.104793 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.106224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.104857 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.106224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.104906 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.106224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.105020 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.106224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.105066 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.106224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.105107 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.106224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.105162 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.106224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.105202 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.106224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.105238 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdcq6\" (UniqueName: \"kubernetes.io/projected/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-kube-api-access-rdcq6\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.209038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.205908 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.209038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.206057 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.209038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.206105 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.209038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.206165 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.209038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.206200 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.209038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.206237 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.209038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.206271 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdcq6\" (UniqueName: \"kubernetes.io/projected/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-kube-api-access-rdcq6\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.209038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.206318 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.209038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.206366 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.209038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.206799 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.209038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.207423 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.209038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.208070 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.209038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.208358 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.209038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.208605 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.211260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.211213 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.211369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.211338 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.222725 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.222689 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.222725 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.222717 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdcq6\" (UniqueName: \"kubernetes.io/projected/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-kube-api-access-rdcq6\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.299497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.299404 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:46.309750 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.309726 5091 patch_prober.go:29] interesting pod/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:40:46.309854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.309783 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" podUID="d9cb9939-3547-4232-b6c0-d2a40803a666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 19:40:46.366698 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.366615 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:46.425232 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.425200 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k"] Oct 30 19:40:46.452789 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.452756 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce0f96ef-166c-4b4c-97fb-d08beb3b36aa" path="/var/lib/kubelet/pods/ce0f96ef-166c-4b4c-97fb-d08beb3b36aa/volumes" Oct 30 19:40:46.581588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.581555 5091 generic.go:356] "Generic (PLEG): container finished" podID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerID="0e89c6f15fc7793aea59c2f8c71a6bd14f211ecdb5cde1b721757cb51982f197" exitCode=0 Oct 30 19:40:46.581788 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.581638 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" event={"ID":"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8","Type":"ContainerDied","Data":"0e89c6f15fc7793aea59c2f8c71a6bd14f211ecdb5cde1b721757cb51982f197"} Oct 30 19:40:46.582246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.582222 5091 scope.go:122] "RemoveContainer" containerID="0e89c6f15fc7793aea59c2f8c71a6bd14f211ecdb5cde1b721757cb51982f197" Oct 30 19:40:46.583514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.583492 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" event={"ID":"efad46d4-e7ef-4139-9397-7d1b56faec0d","Type":"ContainerStarted","Data":"95c41a3c8dff53589afd28b5c33eccf0e975bd2700caf0998d808fc46a6bf0f4"} Oct 30 19:40:46.583736 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.583707 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:46.583803 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.583790 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:46.583868 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.583854 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:46.630288 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.630259 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2"] Oct 30 19:40:46.666341 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:46.666311 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode65edc8b_97e7_40ac_9f21_e1ffd3aff71d.slice/crio-1e51e8015e7655b3aff01b8f125db14e97a13b0fe42b35b52e12ec83d20d237e WatchSource:0}: Error finding container 1e51e8015e7655b3aff01b8f125db14e97a13b0fe42b35b52e12ec83d20d237e: Status 404 returned error can't find the container with id 1e51e8015e7655b3aff01b8f125db14e97a13b0fe42b35b52e12ec83d20d237e Oct 30 19:40:46.682213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.682120 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:46.682347 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.682270 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:46.682347 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.682322 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:46.868705 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:46.868649 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:47.311164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:47.309719 5091 patch_prober.go:29] interesting pod/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:40:47.311164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:47.309779 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" podUID="d9cb9939-3547-4232-b6c0-d2a40803a666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 19:40:47.614611 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:47.614471 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" event={"ID":"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d","Type":"ContainerStarted","Data":"31057ad55c8374fd1f70af888ca4405fe2163b86852725c4f093f48249ea9145"} Oct 30 19:40:47.614611 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:47.614532 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" event={"ID":"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d","Type":"ContainerStarted","Data":"1e51e8015e7655b3aff01b8f125db14e97a13b0fe42b35b52e12ec83d20d237e"} Oct 30 19:40:47.616661 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:47.616635 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" event={"ID":"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8","Type":"ContainerStarted","Data":"a13c1ebc45dfc5896b52ffbd6ab01be98a408a78be9370d22f72a991ab85212b"} Oct 30 19:40:47.618290 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:47.618268 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" event={"ID":"efad46d4-e7ef-4139-9397-7d1b56faec0d","Type":"ContainerStarted","Data":"f046910eac154f85263424987bdf5e29e45102cf28dc379b0f01f23a33d1b802"} Oct 30 19:40:47.644980 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:47.644937 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" podStartSLOduration=2.644926248 podStartE2EDuration="2.644926248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:47.640356512 +0000 UTC m=+567.335564470" watchObservedRunningTime="2026-10-30 19:40:47.644926248 +0000 UTC m=+567.340134206" Oct 30 19:40:47.693990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:47.684231 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm"] Oct 30 19:40:47.715314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:47.715278 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz"] Oct 30 19:40:47.723912 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:47.723882 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:47.724258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:47.724236 5091 patch_prober.go:29] interesting pod/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:40:47.724414 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:47.724389 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" podUID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 19:40:47.732776 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:47.732726 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" podStartSLOduration=2.732708716 podStartE2EDuration="2.732708716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:47.723098852 +0000 UTC m=+567.418306811" watchObservedRunningTime="2026-10-30 19:40:47.732708716 +0000 UTC m=+567.427916674" Oct 30 19:40:47.815707 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:47.815653 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:47.981105 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:47.981002 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:48.152338 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:48.152291 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:48.302170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:48.301212 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:48.316178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:48.315487 5091 patch_prober.go:29] interesting pod/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Oct 30 19:40:48.316178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:48.315550 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" podUID="d9cb9939-3547-4232-b6c0-d2a40803a666" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Oct 30 19:40:48.327171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:48.327092 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:48.538847 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:48.538808 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz"] Oct 30 19:40:48.616292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:48.616210 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln"] Oct 30 19:40:48.632674 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:48.631252 5091 generic.go:356] "Generic (PLEG): container finished" podID="efad46d4-e7ef-4139-9397-7d1b56faec0d" containerID="f046910eac154f85263424987bdf5e29e45102cf28dc379b0f01f23a33d1b802" exitCode=0 Oct 30 19:40:48.632674 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:48.631970 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" event={"ID":"efad46d4-e7ef-4139-9397-7d1b56faec0d","Type":"ContainerDied","Data":"f046910eac154f85263424987bdf5e29e45102cf28dc379b0f01f23a33d1b802"} Oct 30 19:40:48.632674 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:48.632534 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:48.633532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:48.633491 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:40:48.693185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:48.691931 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:48.721914 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:48.721874 5091 patch_prober.go:29] interesting pod/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:40:48.722108 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:48.721949 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" podUID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 19:40:48.887225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:48.886721 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:49.314212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:49.314181 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:49.315229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:49.315212 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:40:49.720824 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:49.720795 5091 patch_prober.go:29] interesting pod/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:40:49.720991 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:49.720852 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" podUID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 19:40:49.738577 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:49.738543 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:49.879196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:49.875335 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" Oct 30 19:40:49.928836 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:49.928742 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28"] Oct 30 19:40:49.984256 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:49.982658 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h"] Oct 30 19:40:50.003374 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.003326 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:50.021655 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.021626 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efad46d4-e7ef-4139-9397-7d1b56faec0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efad46d4-e7ef-4139-9397-7d1b56faec0d-maas-api-xks-ca\") pod \"efad46d4-e7ef-4139-9397-7d1b56faec0d\" (UID: \"efad46d4-e7ef-4139-9397-7d1b56faec0d\") " Oct 30 19:40:50.021655 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.021660 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efad46d4-e7ef-4139-9397-7d1b56faec0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efad46d4-e7ef-4139-9397-7d1b56faec0d-maas-api-openshift-ca\") pod \"efad46d4-e7ef-4139-9397-7d1b56faec0d\" (UID: \"efad46d4-e7ef-4139-9397-7d1b56faec0d\") " Oct 30 19:40:50.021881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.021718 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efad46d4-e7ef-4139-9397-7d1b56faec0d-kube-api-access-7dk2f\" (UniqueName: \"kubernetes.io/projected/efad46d4-e7ef-4139-9397-7d1b56faec0d-kube-api-access-7dk2f\") pod \"efad46d4-e7ef-4139-9397-7d1b56faec0d\" (UID: \"efad46d4-e7ef-4139-9397-7d1b56faec0d\") " Oct 30 19:40:50.022317 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.022281 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efad46d4-e7ef-4139-9397-7d1b56faec0d-maas-api-openshift-ca" pod "efad46d4-e7ef-4139-9397-7d1b56faec0d" (UID: "efad46d4-e7ef-4139-9397-7d1b56faec0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:50.025977 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.025949 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efad46d4-e7ef-4139-9397-7d1b56faec0d-kube-api-access-7dk2f" pod "efad46d4-e7ef-4139-9397-7d1b56faec0d" (UID: "efad46d4-e7ef-4139-9397-7d1b56faec0d"). InnerVolumeSpecName "kube-api-access-7dk2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:50.026394 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.026368 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efad46d4-e7ef-4139-9397-7d1b56faec0d-maas-api-xks-ca" pod "efad46d4-e7ef-4139-9397-7d1b56faec0d" (UID: "efad46d4-e7ef-4139-9397-7d1b56faec0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:40:50.080092 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.079937 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2"] Oct 30 19:40:50.084502 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.084446 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-3a55af38-k76v2"] Oct 30 19:40:50.125096 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.124960 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efad46d4-e7ef-4139-9397-7d1b56faec0d-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:50.125096 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.124999 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efad46d4-e7ef-4139-9397-7d1b56faec0d-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:50.125096 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.125016 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dk2f\" (UniqueName: \"kubernetes.io/projected/efad46d4-e7ef-4139-9397-7d1b56faec0d-kube-api-access-7dk2f\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:50.180974 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.180833 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:50.243223 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.243192 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7"] Oct 30 19:40:50.243452 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.243425 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" podUID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerName="istio-proxy" containerID="cri-o://b582102bd1cdaa6e45a54f89d214bc3a335fb312298d96f6b2e2f4e805735cc6" gracePeriod=30 Oct 30 19:40:50.444063 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.443984 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7022c039-2eb1-4cb5-87b1-71d26169c4fe" path="/var/lib/kubelet/pods/7022c039-2eb1-4cb5-87b1-71d26169c4fe/volumes" Oct 30 19:40:50.505221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.505190 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:50.508022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.507997 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf"] Oct 30 19:40:50.511191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.511169 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b33c04be-cvkpf"] Oct 30 19:40:50.676358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.676322 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" event={"ID":"efad46d4-e7ef-4139-9397-7d1b56faec0d","Type":"ContainerDied","Data":"95c41a3c8dff53589afd28b5c33eccf0e975bd2700caf0998d808fc46a6bf0f4"} Oct 30 19:40:50.676629 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.676612 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95c41a3c8dff53589afd28b5c33eccf0e975bd2700caf0998d808fc46a6bf0f4" Oct 30 19:40:50.676717 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.676480 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k" Oct 30 19:40:50.676935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.676915 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:50.721382 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.721291 5091 patch_prober.go:29] interesting pod/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:40:50.721382 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.721345 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" podUID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 19:40:50.868168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:50.865129 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:51.110589 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:51.110549 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:51.582673 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:51.582624 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:51.721211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:51.721171 5091 patch_prober.go:29] interesting pod/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:40:51.721370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:51.721224 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" podUID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 19:40:51.823106 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:51.822439 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:51.994529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:51.994466 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:52.132378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:52.132300 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:52.442066 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:52.441983 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a00fed1d-f03c-4656-aad2-1f07ac428e68" path="/var/lib/kubelet/pods/a00fed1d-f03c-4656-aad2-1f07ac428e68/volumes" Oct 30 19:40:52.685135 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:52.685089 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" podUID="550f7a23-825c-45dc-a2bf-ee67264eed56" containerName="payload-processing" containerID="cri-o://114b0b4e66909979829e312a1d32dc13b65850c5d82c2c353d7db0f745737e4c" gracePeriod=25 Oct 30 19:40:52.715971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:52.715935 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" podUID="91802908-1257-4dc4-8e5b-baaeb4dbdfbb" containerName="payload-pre-processing" containerID="cri-o://f91b86465c1afc447ffd5907302fef0a12b3f1ea329c04363ab3dfa0dafa8d65" gracePeriod=25 Oct 30 19:40:52.721669 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:52.721645 5091 patch_prober.go:29] interesting pod/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Oct 30 19:40:52.721771 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:52.721693 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" podUID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Oct 30 19:40:52.842453 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:52.842353 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:53.031139 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.031104 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:53.053163 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.045358 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" Oct 30 19:40:53.122201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.121278 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" Oct 30 19:40:53.209665 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.209627 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:53.209975 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.209946 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/550f7a23-825c-45dc-a2bf-ee67264eed56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550f7a23-825c-45dc-a2bf-ee67264eed56-plugins-config-volume\") pod \"550f7a23-825c-45dc-a2bf-ee67264eed56\" (UID: \"550f7a23-825c-45dc-a2bf-ee67264eed56\") " Oct 30 19:40:53.210253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.210228 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/550f7a23-825c-45dc-a2bf-ee67264eed56-kube-api-access-5pcxk\" (UniqueName: \"kubernetes.io/projected/550f7a23-825c-45dc-a2bf-ee67264eed56-kube-api-access-5pcxk\") pod \"550f7a23-825c-45dc-a2bf-ee67264eed56\" (UID: \"550f7a23-825c-45dc-a2bf-ee67264eed56\") " Oct 30 19:40:53.210457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.210410 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/550f7a23-825c-45dc-a2bf-ee67264eed56-plugins-config-volume" pod "550f7a23-825c-45dc-a2bf-ee67264eed56" (UID: "550f7a23-825c-45dc-a2bf-ee67264eed56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:53.210764 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.210745 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/550f7a23-825c-45dc-a2bf-ee67264eed56-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:53.212706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.212671 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/550f7a23-825c-45dc-a2bf-ee67264eed56-kube-api-access-5pcxk" pod "550f7a23-825c-45dc-a2bf-ee67264eed56" (UID: "550f7a23-825c-45dc-a2bf-ee67264eed56"). InnerVolumeSpecName "kube-api-access-5pcxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:53.311357 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.311324 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91802908-1257-4dc4-8e5b-baaeb4dbdfbb-kube-api-access-gc79v\" (UniqueName: \"kubernetes.io/projected/91802908-1257-4dc4-8e5b-baaeb4dbdfbb-kube-api-access-gc79v\") pod \"91802908-1257-4dc4-8e5b-baaeb4dbdfbb\" (UID: \"91802908-1257-4dc4-8e5b-baaeb4dbdfbb\") " Oct 30 19:40:53.311530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.311464 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91802908-1257-4dc4-8e5b-baaeb4dbdfbb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91802908-1257-4dc4-8e5b-baaeb4dbdfbb-plugins-config-volume\") pod \"91802908-1257-4dc4-8e5b-baaeb4dbdfbb\" (UID: \"91802908-1257-4dc4-8e5b-baaeb4dbdfbb\") " Oct 30 19:40:53.311871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.311843 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pcxk\" (UniqueName: \"kubernetes.io/projected/550f7a23-825c-45dc-a2bf-ee67264eed56-kube-api-access-5pcxk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:53.312008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.311869 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91802908-1257-4dc4-8e5b-baaeb4dbdfbb-plugins-config-volume" pod "91802908-1257-4dc4-8e5b-baaeb4dbdfbb" (UID: "91802908-1257-4dc4-8e5b-baaeb4dbdfbb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:53.313609 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.313580 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91802908-1257-4dc4-8e5b-baaeb4dbdfbb-kube-api-access-gc79v" pod "91802908-1257-4dc4-8e5b-baaeb4dbdfbb" (UID: "91802908-1257-4dc4-8e5b-baaeb4dbdfbb"). InnerVolumeSpecName "kube-api-access-gc79v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:53.413172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.412175 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:53.413172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.412900 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91802908-1257-4dc4-8e5b-baaeb4dbdfbb-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:53.413172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.412920 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc79v\" (UniqueName: \"kubernetes.io/projected/91802908-1257-4dc4-8e5b-baaeb4dbdfbb-kube-api-access-gc79v\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:53.540199 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.540120 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" podUID="34ec41bf-040e-46ff-82df-7a3f73929cc8" containerName="payload-processing" containerID="cri-o://330b5530283ec52dbca78f56ff5180df87fb583b96680f101050db58829f0065" gracePeriod=25 Oct 30 19:40:53.576117 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.576046 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:53.617524 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.617459 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" podUID="6733505d-3dd3-462d-a96a-ee60c3764e64" containerName="payload-pre-processing" containerID="cri-o://a413811902b0492fb459760fac0dd802ef2b8d2b9258c3c700fad3ca0fa4892c" gracePeriod=25 Oct 30 19:40:53.708777 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.708471 5091 generic.go:356] "Generic (PLEG): container finished" podID="91802908-1257-4dc4-8e5b-baaeb4dbdfbb" containerID="f91b86465c1afc447ffd5907302fef0a12b3f1ea329c04363ab3dfa0dafa8d65" exitCode=0 Oct 30 19:40:53.708777 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.708537 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" Oct 30 19:40:53.708777 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.708538 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" event={"ID":"91802908-1257-4dc4-8e5b-baaeb4dbdfbb","Type":"ContainerDied","Data":"f91b86465c1afc447ffd5907302fef0a12b3f1ea329c04363ab3dfa0dafa8d65"} Oct 30 19:40:53.708777 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.708607 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz" event={"ID":"91802908-1257-4dc4-8e5b-baaeb4dbdfbb","Type":"ContainerDied","Data":"7e67d0450b0ab138a73045e788af6f7f6a6d6a1bc922f681d0a86d04efe8ad2d"} Oct 30 19:40:53.708777 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.708636 5091 scope.go:122] "RemoveContainer" containerID="f91b86465c1afc447ffd5907302fef0a12b3f1ea329c04363ab3dfa0dafa8d65" Oct 30 19:40:53.713231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.713172 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" event={"ID":"34ec41bf-040e-46ff-82df-7a3f73929cc8","Type":"ContainerDied","Data":"330b5530283ec52dbca78f56ff5180df87fb583b96680f101050db58829f0065"} Oct 30 19:40:53.713364 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.713138 5091 generic.go:356] "Generic (PLEG): container finished" podID="34ec41bf-040e-46ff-82df-7a3f73929cc8" containerID="330b5530283ec52dbca78f56ff5180df87fb583b96680f101050db58829f0065" exitCode=0 Oct 30 19:40:53.714946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.714924 5091 generic.go:356] "Generic (PLEG): container finished" podID="550f7a23-825c-45dc-a2bf-ee67264eed56" containerID="114b0b4e66909979829e312a1d32dc13b65850c5d82c2c353d7db0f745737e4c" exitCode=0 Oct 30 19:40:53.715042 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.714968 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" Oct 30 19:40:53.715042 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.714978 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" event={"ID":"550f7a23-825c-45dc-a2bf-ee67264eed56","Type":"ContainerDied","Data":"114b0b4e66909979829e312a1d32dc13b65850c5d82c2c353d7db0f745737e4c"} Oct 30 19:40:53.715042 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.715013 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm" event={"ID":"550f7a23-825c-45dc-a2bf-ee67264eed56","Type":"ContainerDied","Data":"4f1c9d4d771a0cdc4ef065f93e6bb660f358d4c8994f117e403fe3b4f5efb1de"} Oct 30 19:40:53.726087 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.726065 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:53.726521 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.726504 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:53.727347 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.727327 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:40:53.730670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.730654 5091 scope.go:122] "RemoveContainer" containerID="f91b86465c1afc447ffd5907302fef0a12b3f1ea329c04363ab3dfa0dafa8d65" Oct 30 19:40:53.730950 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:40:53.730930 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f91b86465c1afc447ffd5907302fef0a12b3f1ea329c04363ab3dfa0dafa8d65\": container with ID starting with f91b86465c1afc447ffd5907302fef0a12b3f1ea329c04363ab3dfa0dafa8d65 not found: ID does not exist" containerID="f91b86465c1afc447ffd5907302fef0a12b3f1ea329c04363ab3dfa0dafa8d65" Oct 30 19:40:53.731034 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.730960 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f91b86465c1afc447ffd5907302fef0a12b3f1ea329c04363ab3dfa0dafa8d65"} err="failed to get container status \"f91b86465c1afc447ffd5907302fef0a12b3f1ea329c04363ab3dfa0dafa8d65\": rpc error: code = NotFound desc = could not find container \"f91b86465c1afc447ffd5907302fef0a12b3f1ea329c04363ab3dfa0dafa8d65\": container with ID starting with f91b86465c1afc447ffd5907302fef0a12b3f1ea329c04363ab3dfa0dafa8d65 not found: ID does not exist" Oct 30 19:40:53.731034 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.730984 5091 scope.go:122] "RemoveContainer" containerID="114b0b4e66909979829e312a1d32dc13b65850c5d82c2c353d7db0f745737e4c" Oct 30 19:40:53.739472 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.739448 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz"] Oct 30 19:40:53.742809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.742785 5091 scope.go:122] "RemoveContainer" containerID="114b0b4e66909979829e312a1d32dc13b65850c5d82c2c353d7db0f745737e4c" Oct 30 19:40:53.743114 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:40:53.743096 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114b0b4e66909979829e312a1d32dc13b65850c5d82c2c353d7db0f745737e4c\": container with ID starting with 114b0b4e66909979829e312a1d32dc13b65850c5d82c2c353d7db0f745737e4c not found: ID does not exist" containerID="114b0b4e66909979829e312a1d32dc13b65850c5d82c2c353d7db0f745737e4c" Oct 30 19:40:53.743210 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.743121 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114b0b4e66909979829e312a1d32dc13b65850c5d82c2c353d7db0f745737e4c"} err="failed to get container status \"114b0b4e66909979829e312a1d32dc13b65850c5d82c2c353d7db0f745737e4c\": rpc error: code = NotFound desc = could not find container \"114b0b4e66909979829e312a1d32dc13b65850c5d82c2c353d7db0f745737e4c\": container with ID starting with 114b0b4e66909979829e312a1d32dc13b65850c5d82c2c353d7db0f745737e4c not found: ID does not exist" Oct 30 19:40:53.749657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.749634 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3a55af38-65c8454759-5gbpz"] Oct 30 19:40:53.772494 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.772448 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm"] Oct 30 19:40:53.774622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.774597 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3a55af38-6f454d75c4-s62sm"] Oct 30 19:40:53.904890 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.904863 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" Oct 30 19:40:53.965193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:53.965165 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" Oct 30 19:40:54.020082 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.020050 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34ec41bf-040e-46ff-82df-7a3f73929cc8-kube-api-access-9n77s\" (UniqueName: \"kubernetes.io/projected/34ec41bf-040e-46ff-82df-7a3f73929cc8-kube-api-access-9n77s\") pod \"34ec41bf-040e-46ff-82df-7a3f73929cc8\" (UID: \"34ec41bf-040e-46ff-82df-7a3f73929cc8\") " Oct 30 19:40:54.020273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.020231 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34ec41bf-040e-46ff-82df-7a3f73929cc8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ec41bf-040e-46ff-82df-7a3f73929cc8-plugins-config-volume\") pod \"34ec41bf-040e-46ff-82df-7a3f73929cc8\" (UID: \"34ec41bf-040e-46ff-82df-7a3f73929cc8\") " Oct 30 19:40:54.021082 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.021044 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34ec41bf-040e-46ff-82df-7a3f73929cc8-plugins-config-volume" pod "34ec41bf-040e-46ff-82df-7a3f73929cc8" (UID: "34ec41bf-040e-46ff-82df-7a3f73929cc8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:54.022463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.022440 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34ec41bf-040e-46ff-82df-7a3f73929cc8-kube-api-access-9n77s" pod "34ec41bf-040e-46ff-82df-7a3f73929cc8" (UID: "34ec41bf-040e-46ff-82df-7a3f73929cc8"). InnerVolumeSpecName "kube-api-access-9n77s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:54.121269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.121197 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6733505d-3dd3-462d-a96a-ee60c3764e64-kube-api-access-qfm9h\" (UniqueName: \"kubernetes.io/projected/6733505d-3dd3-462d-a96a-ee60c3764e64-kube-api-access-qfm9h\") pod \"6733505d-3dd3-462d-a96a-ee60c3764e64\" (UID: \"6733505d-3dd3-462d-a96a-ee60c3764e64\") " Oct 30 19:40:54.121422 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.121318 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6733505d-3dd3-462d-a96a-ee60c3764e64-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6733505d-3dd3-462d-a96a-ee60c3764e64-plugins-config-volume\") pod \"6733505d-3dd3-462d-a96a-ee60c3764e64\" (UID: \"6733505d-3dd3-462d-a96a-ee60c3764e64\") " Oct 30 19:40:54.121706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.121685 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34ec41bf-040e-46ff-82df-7a3f73929cc8-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:54.121758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.121716 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n77s\" (UniqueName: \"kubernetes.io/projected/34ec41bf-040e-46ff-82df-7a3f73929cc8-kube-api-access-9n77s\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:54.121758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.121725 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6733505d-3dd3-462d-a96a-ee60c3764e64-plugins-config-volume" pod "6733505d-3dd3-462d-a96a-ee60c3764e64" (UID: "6733505d-3dd3-462d-a96a-ee60c3764e64"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:54.123463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.123435 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6733505d-3dd3-462d-a96a-ee60c3764e64-kube-api-access-qfm9h" pod "6733505d-3dd3-462d-a96a-ee60c3764e64" (UID: "6733505d-3dd3-462d-a96a-ee60c3764e64"). InnerVolumeSpecName "kube-api-access-qfm9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:54.199648 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.199609 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:54.222658 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.222626 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfm9h\" (UniqueName: \"kubernetes.io/projected/6733505d-3dd3-462d-a96a-ee60c3764e64-kube-api-access-qfm9h\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:54.222658 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.222657 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6733505d-3dd3-462d-a96a-ee60c3764e64-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:54.276259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.276229 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr"] Oct 30 19:40:54.281728 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.281703 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-0897eecc-gv5vr"] Oct 30 19:40:54.405167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.405066 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:54.442254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.442218 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53cf7ed8-9b51-491c-802f-9c1a0da29378" path="/var/lib/kubelet/pods/53cf7ed8-9b51-491c-802f-9c1a0da29378/volumes" Oct 30 19:40:54.442641 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.442626 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="550f7a23-825c-45dc-a2bf-ee67264eed56" path="/var/lib/kubelet/pods/550f7a23-825c-45dc-a2bf-ee67264eed56/volumes" Oct 30 19:40:54.442984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.442971 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91802908-1257-4dc4-8e5b-baaeb4dbdfbb" path="/var/lib/kubelet/pods/91802908-1257-4dc4-8e5b-baaeb4dbdfbb/volumes" Oct 30 19:40:54.724043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.724005 5091 generic.go:356] "Generic (PLEG): container finished" podID="6733505d-3dd3-462d-a96a-ee60c3764e64" containerID="a413811902b0492fb459760fac0dd802ef2b8d2b9258c3c700fad3ca0fa4892c" exitCode=0 Oct 30 19:40:54.724450 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.724063 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" Oct 30 19:40:54.724450 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.724086 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" event={"ID":"6733505d-3dd3-462d-a96a-ee60c3764e64","Type":"ContainerDied","Data":"a413811902b0492fb459760fac0dd802ef2b8d2b9258c3c700fad3ca0fa4892c"} Oct 30 19:40:54.724450 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.724186 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln" event={"ID":"6733505d-3dd3-462d-a96a-ee60c3764e64","Type":"ContainerDied","Data":"6f0169a4a8c3a104fe69873c9a0239cf1edb0cd8b4ebdeac0ace98c87a5be872"} Oct 30 19:40:54.724450 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.724213 5091 scope.go:122] "RemoveContainer" containerID="a413811902b0492fb459760fac0dd802ef2b8d2b9258c3c700fad3ca0fa4892c" Oct 30 19:40:54.725680 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.725659 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" event={"ID":"34ec41bf-040e-46ff-82df-7a3f73929cc8","Type":"ContainerDied","Data":"1f16240daa735a87a0bde27416aa1c36669e98190c936aa9f1cedfc63d888f2a"} Oct 30 19:40:54.725789 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.725685 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz" Oct 30 19:40:54.735653 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.735629 5091 scope.go:122] "RemoveContainer" containerID="a413811902b0492fb459760fac0dd802ef2b8d2b9258c3c700fad3ca0fa4892c" Oct 30 19:40:54.735984 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:40:54.735958 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a413811902b0492fb459760fac0dd802ef2b8d2b9258c3c700fad3ca0fa4892c\": container with ID starting with a413811902b0492fb459760fac0dd802ef2b8d2b9258c3c700fad3ca0fa4892c not found: ID does not exist" containerID="a413811902b0492fb459760fac0dd802ef2b8d2b9258c3c700fad3ca0fa4892c" Oct 30 19:40:54.736072 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.735993 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a413811902b0492fb459760fac0dd802ef2b8d2b9258c3c700fad3ca0fa4892c"} err="failed to get container status \"a413811902b0492fb459760fac0dd802ef2b8d2b9258c3c700fad3ca0fa4892c\": rpc error: code = NotFound desc = could not find container \"a413811902b0492fb459760fac0dd802ef2b8d2b9258c3c700fad3ca0fa4892c\": container with ID starting with a413811902b0492fb459760fac0dd802ef2b8d2b9258c3c700fad3ca0fa4892c not found: ID does not exist" Oct 30 19:40:54.736072 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.736017 5091 scope.go:122] "RemoveContainer" containerID="330b5530283ec52dbca78f56ff5180df87fb583b96680f101050db58829f0065" Oct 30 19:40:54.753033 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.753007 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln"] Oct 30 19:40:54.757813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.757794 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b33c04be-59455ff787-47bln"] Oct 30 19:40:54.781040 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.781020 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz"] Oct 30 19:40:54.786128 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.786109 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b33c04be-5c89c4c68b-59jpz"] Oct 30 19:40:54.929804 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.929742 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" podUID="8ec2b758-fec8-49ce-a483-e3f91af11c33" containerName="payload-processing" containerID="cri-o://9fb60a995645282e04d820d3f827bd708a35e3565fa6d00ad14abbb4a0cae8d6" gracePeriod=25 Oct 30 19:40:54.933860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.933827 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc"] Oct 30 19:40:54.934125 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.934099 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" podUID="d9cb9939-3547-4232-b6c0-d2a40803a666" containerName="istio-proxy" containerID="cri-o://d28cf35e02c44c0cdacaa0a9a423dd3bf0e7c2109c891d72f2087520d1effe42" gracePeriod=30 Oct 30 19:40:54.984031 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:54.983977 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" podUID="8f46b66e-1ca4-41ca-9be7-040cb8dec18a" containerName="payload-pre-processing" containerID="cri-o://92c85cbef00fddb14f9abe1d8451dd0ba232b9a2a40d2967fa85dafbb0452f6f" gracePeriod=25 Oct 30 19:40:55.011550 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.011517 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:55.240881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.240850 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" Oct 30 19:40:55.327276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.327251 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" Oct 30 19:40:55.340794 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.340758 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f46b66e-1ca4-41ca-9be7-040cb8dec18a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f46b66e-1ca4-41ca-9be7-040cb8dec18a-plugins-config-volume\") pod \"8f46b66e-1ca4-41ca-9be7-040cb8dec18a\" (UID: \"8f46b66e-1ca4-41ca-9be7-040cb8dec18a\") " Oct 30 19:40:55.340939 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.340915 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f46b66e-1ca4-41ca-9be7-040cb8dec18a-kube-api-access-76zj6\" (UniqueName: \"kubernetes.io/projected/8f46b66e-1ca4-41ca-9be7-040cb8dec18a-kube-api-access-76zj6\") pod \"8f46b66e-1ca4-41ca-9be7-040cb8dec18a\" (UID: \"8f46b66e-1ca4-41ca-9be7-040cb8dec18a\") " Oct 30 19:40:55.341002 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.340961 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec2b758-fec8-49ce-a483-e3f91af11c33-kube-api-access-vvgwh\" (UniqueName: \"kubernetes.io/projected/8ec2b758-fec8-49ce-a483-e3f91af11c33-kube-api-access-vvgwh\") pod \"8ec2b758-fec8-49ce-a483-e3f91af11c33\" (UID: \"8ec2b758-fec8-49ce-a483-e3f91af11c33\") " Oct 30 19:40:55.341058 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.341011 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec2b758-fec8-49ce-a483-e3f91af11c33-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec2b758-fec8-49ce-a483-e3f91af11c33-plugins-config-volume\") pod \"8ec2b758-fec8-49ce-a483-e3f91af11c33\" (UID: \"8ec2b758-fec8-49ce-a483-e3f91af11c33\") " Oct 30 19:40:55.341212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.341178 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f46b66e-1ca4-41ca-9be7-040cb8dec18a-plugins-config-volume" pod "8f46b66e-1ca4-41ca-9be7-040cb8dec18a" (UID: "8f46b66e-1ca4-41ca-9be7-040cb8dec18a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:55.341475 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.341444 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec2b758-fec8-49ce-a483-e3f91af11c33-plugins-config-volume" pod "8ec2b758-fec8-49ce-a483-e3f91af11c33" (UID: "8ec2b758-fec8-49ce-a483-e3f91af11c33"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:55.341582 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.341563 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ec2b758-fec8-49ce-a483-e3f91af11c33-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:55.341650 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.341586 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f46b66e-1ca4-41ca-9be7-040cb8dec18a-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:55.342967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.342936 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f46b66e-1ca4-41ca-9be7-040cb8dec18a-kube-api-access-76zj6" pod "8f46b66e-1ca4-41ca-9be7-040cb8dec18a" (UID: "8f46b66e-1ca4-41ca-9be7-040cb8dec18a"). InnerVolumeSpecName "kube-api-access-76zj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:55.343273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.343071 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec2b758-fec8-49ce-a483-e3f91af11c33-kube-api-access-vvgwh" pod "8ec2b758-fec8-49ce-a483-e3f91af11c33" (UID: "8ec2b758-fec8-49ce-a483-e3f91af11c33"). InnerVolumeSpecName "kube-api-access-vvgwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:55.376906 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.370238 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:55.426926 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.426895 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:55.442223 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.442192 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-podinfo\") pod \"8253e5fe-3ea9-4367-b43d-274dc85176f8\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " Oct 30 19:40:55.442393 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.442272 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8253e5fe-3ea9-4367-b43d-274dc85176f8-kube-api-access-62bpj\" (UniqueName: \"kubernetes.io/projected/8253e5fe-3ea9-4367-b43d-274dc85176f8-kube-api-access-62bpj\") pod \"8253e5fe-3ea9-4367-b43d-274dc85176f8\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " Oct 30 19:40:55.442393 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.442370 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-envoy\") pod \"8253e5fe-3ea9-4367-b43d-274dc85176f8\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " Oct 30 19:40:55.442503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.442454 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-workload-certs\") pod \"8253e5fe-3ea9-4367-b43d-274dc85176f8\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " Oct 30 19:40:55.442503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.442491 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-workload-socket\") pod \"8253e5fe-3ea9-4367-b43d-274dc85176f8\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " Oct 30 19:40:55.442609 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.442542 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-token\" (UniqueName: \"kubernetes.io/projected/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-token\") pod \"8253e5fe-3ea9-4367-b43d-274dc85176f8\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " Oct 30 19:40:55.442659 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.442617 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-credential-socket\") pod \"8253e5fe-3ea9-4367-b43d-274dc85176f8\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " Oct 30 19:40:55.442659 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.442649 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-data\") pod \"8253e5fe-3ea9-4367-b43d-274dc85176f8\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " Oct 30 19:40:55.442758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.442694 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8253e5fe-3ea9-4367-b43d-274dc85176f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8253e5fe-3ea9-4367-b43d-274dc85176f8-istiod-ca-cert\") pod \"8253e5fe-3ea9-4367-b43d-274dc85176f8\" (UID: \"8253e5fe-3ea9-4367-b43d-274dc85176f8\") " Oct 30 19:40:55.443222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.443196 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76zj6\" (UniqueName: \"kubernetes.io/projected/8f46b66e-1ca4-41ca-9be7-040cb8dec18a-kube-api-access-76zj6\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:55.443344 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.443224 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvgwh\" (UniqueName: \"kubernetes.io/projected/8ec2b758-fec8-49ce-a483-e3f91af11c33-kube-api-access-vvgwh\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:55.443344 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.443234 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-data" pod "8253e5fe-3ea9-4367-b43d-274dc85176f8" (UID: "8253e5fe-3ea9-4367-b43d-274dc85176f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:55.443462 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.443350 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-credential-socket" pod "8253e5fe-3ea9-4367-b43d-274dc85176f8" (UID: "8253e5fe-3ea9-4367-b43d-274dc85176f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:55.443462 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.443451 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-workload-certs" pod "8253e5fe-3ea9-4367-b43d-274dc85176f8" (UID: "8253e5fe-3ea9-4367-b43d-274dc85176f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:55.443673 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.443648 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-workload-socket" pod "8253e5fe-3ea9-4367-b43d-274dc85176f8" (UID: "8253e5fe-3ea9-4367-b43d-274dc85176f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:55.443673 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.443658 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8253e5fe-3ea9-4367-b43d-274dc85176f8-istiod-ca-cert" pod "8253e5fe-3ea9-4367-b43d-274dc85176f8" (UID: "8253e5fe-3ea9-4367-b43d-274dc85176f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:40:55.445173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.445029 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8253e5fe-3ea9-4367-b43d-274dc85176f8-kube-api-access-62bpj" pod "8253e5fe-3ea9-4367-b43d-274dc85176f8" (UID: "8253e5fe-3ea9-4367-b43d-274dc85176f8"). InnerVolumeSpecName "kube-api-access-62bpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:55.445173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.445111 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-token" pod "8253e5fe-3ea9-4367-b43d-274dc85176f8" (UID: "8253e5fe-3ea9-4367-b43d-274dc85176f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:40:55.445920 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.445901 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-podinfo" pod "8253e5fe-3ea9-4367-b43d-274dc85176f8" (UID: "8253e5fe-3ea9-4367-b43d-274dc85176f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:40:55.446433 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.446413 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-envoy" pod "8253e5fe-3ea9-4367-b43d-274dc85176f8" (UID: "8253e5fe-3ea9-4367-b43d-274dc85176f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:40:55.544670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.544638 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:55.544670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.544668 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:55.544871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.544678 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8253e5fe-3ea9-4367-b43d-274dc85176f8-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:55.544871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.544687 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:55.544871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.544696 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62bpj\" (UniqueName: \"kubernetes.io/projected/8253e5fe-3ea9-4367-b43d-274dc85176f8-kube-api-access-62bpj\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:55.544871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.544705 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:55.544871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.544713 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:55.544871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.544721 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8253e5fe-3ea9-4367-b43d-274dc85176f8-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:55.544871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.544729 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8253e5fe-3ea9-4367-b43d-274dc85176f8-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:40:55.685556 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.685517 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:55.733783 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.733750 5091 generic.go:356] "Generic (PLEG): container finished" podID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerID="b582102bd1cdaa6e45a54f89d214bc3a335fb312298d96f6b2e2f4e805735cc6" exitCode=0 Oct 30 19:40:55.734231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.733806 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" Oct 30 19:40:55.734231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.733838 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" event={"ID":"8253e5fe-3ea9-4367-b43d-274dc85176f8","Type":"ContainerDied","Data":"b582102bd1cdaa6e45a54f89d214bc3a335fb312298d96f6b2e2f4e805735cc6"} Oct 30 19:40:55.734231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.733891 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7" event={"ID":"8253e5fe-3ea9-4367-b43d-274dc85176f8","Type":"ContainerDied","Data":"b10319e26f314b824cd76dd9aef19801cb38c8ea9bbce194dee728a06348bb75"} Oct 30 19:40:55.734231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.733922 5091 scope.go:122] "RemoveContainer" containerID="b582102bd1cdaa6e45a54f89d214bc3a335fb312298d96f6b2e2f4e805735cc6" Oct 30 19:40:55.736824 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.736798 5091 generic.go:356] "Generic (PLEG): container finished" podID="8ec2b758-fec8-49ce-a483-e3f91af11c33" containerID="9fb60a995645282e04d820d3f827bd708a35e3565fa6d00ad14abbb4a0cae8d6" exitCode=0 Oct 30 19:40:55.736952 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.736833 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" Oct 30 19:40:55.736952 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.736892 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" event={"ID":"8ec2b758-fec8-49ce-a483-e3f91af11c33","Type":"ContainerDied","Data":"9fb60a995645282e04d820d3f827bd708a35e3565fa6d00ad14abbb4a0cae8d6"} Oct 30 19:40:55.737080 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.736953 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28" event={"ID":"8ec2b758-fec8-49ce-a483-e3f91af11c33","Type":"ContainerDied","Data":"50d5bd57fa26687db9cb633e1a9f0e600840f9ee7383939fd93493ed03edd1f1"} Oct 30 19:40:55.741508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.741485 5091 generic.go:356] "Generic (PLEG): container finished" podID="8f46b66e-1ca4-41ca-9be7-040cb8dec18a" containerID="92c85cbef00fddb14f9abe1d8451dd0ba232b9a2a40d2967fa85dafbb0452f6f" exitCode=0 Oct 30 19:40:55.741620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.741521 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" Oct 30 19:40:55.741620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.741564 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" event={"ID":"8f46b66e-1ca4-41ca-9be7-040cb8dec18a","Type":"ContainerDied","Data":"92c85cbef00fddb14f9abe1d8451dd0ba232b9a2a40d2967fa85dafbb0452f6f"} Oct 30 19:40:55.741742 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.741618 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h" event={"ID":"8f46b66e-1ca4-41ca-9be7-040cb8dec18a","Type":"ContainerDied","Data":"6f6a40ab13a93d727e5e6fb34e30dd553214b55b683154d0fc859a37622c2b91"} Oct 30 19:40:55.746790 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.746768 5091 scope.go:122] "RemoveContainer" containerID="ca9f3c187630b8610dfd3333be4cd7315d86c363552b1a47e09f5bc6a311353f" Oct 30 19:40:55.756739 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.756719 5091 scope.go:122] "RemoveContainer" containerID="b582102bd1cdaa6e45a54f89d214bc3a335fb312298d96f6b2e2f4e805735cc6" Oct 30 19:40:55.757009 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:40:55.756991 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b582102bd1cdaa6e45a54f89d214bc3a335fb312298d96f6b2e2f4e805735cc6\": container with ID starting with b582102bd1cdaa6e45a54f89d214bc3a335fb312298d96f6b2e2f4e805735cc6 not found: ID does not exist" containerID="b582102bd1cdaa6e45a54f89d214bc3a335fb312298d96f6b2e2f4e805735cc6" Oct 30 19:40:55.757085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.757015 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b582102bd1cdaa6e45a54f89d214bc3a335fb312298d96f6b2e2f4e805735cc6"} err="failed to get container status \"b582102bd1cdaa6e45a54f89d214bc3a335fb312298d96f6b2e2f4e805735cc6\": rpc error: code = NotFound desc = could not find container \"b582102bd1cdaa6e45a54f89d214bc3a335fb312298d96f6b2e2f4e805735cc6\": container with ID starting with b582102bd1cdaa6e45a54f89d214bc3a335fb312298d96f6b2e2f4e805735cc6 not found: ID does not exist" Oct 30 19:40:55.757085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.757032 5091 scope.go:122] "RemoveContainer" containerID="ca9f3c187630b8610dfd3333be4cd7315d86c363552b1a47e09f5bc6a311353f" Oct 30 19:40:55.757299 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:40:55.757281 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca9f3c187630b8610dfd3333be4cd7315d86c363552b1a47e09f5bc6a311353f\": container with ID starting with ca9f3c187630b8610dfd3333be4cd7315d86c363552b1a47e09f5bc6a311353f not found: ID does not exist" containerID="ca9f3c187630b8610dfd3333be4cd7315d86c363552b1a47e09f5bc6a311353f" Oct 30 19:40:55.757358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.757305 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca9f3c187630b8610dfd3333be4cd7315d86c363552b1a47e09f5bc6a311353f"} err="failed to get container status \"ca9f3c187630b8610dfd3333be4cd7315d86c363552b1a47e09f5bc6a311353f\": rpc error: code = NotFound desc = could not find container \"ca9f3c187630b8610dfd3333be4cd7315d86c363552b1a47e09f5bc6a311353f\": container with ID starting with ca9f3c187630b8610dfd3333be4cd7315d86c363552b1a47e09f5bc6a311353f not found: ID does not exist" Oct 30 19:40:55.757358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.757322 5091 scope.go:122] "RemoveContainer" containerID="9fb60a995645282e04d820d3f827bd708a35e3565fa6d00ad14abbb4a0cae8d6" Oct 30 19:40:55.767528 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.767506 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7"] Oct 30 19:40:55.768415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.768394 5091 scope.go:122] "RemoveContainer" containerID="9fb60a995645282e04d820d3f827bd708a35e3565fa6d00ad14abbb4a0cae8d6" Oct 30 19:40:55.768718 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:40:55.768694 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fb60a995645282e04d820d3f827bd708a35e3565fa6d00ad14abbb4a0cae8d6\": container with ID starting with 9fb60a995645282e04d820d3f827bd708a35e3565fa6d00ad14abbb4a0cae8d6 not found: ID does not exist" containerID="9fb60a995645282e04d820d3f827bd708a35e3565fa6d00ad14abbb4a0cae8d6" Oct 30 19:40:55.768792 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.768737 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fb60a995645282e04d820d3f827bd708a35e3565fa6d00ad14abbb4a0cae8d6"} err="failed to get container status \"9fb60a995645282e04d820d3f827bd708a35e3565fa6d00ad14abbb4a0cae8d6\": rpc error: code = NotFound desc = could not find container \"9fb60a995645282e04d820d3f827bd708a35e3565fa6d00ad14abbb4a0cae8d6\": container with ID starting with 9fb60a995645282e04d820d3f827bd708a35e3565fa6d00ad14abbb4a0cae8d6 not found: ID does not exist" Oct 30 19:40:55.768792 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.768762 5091 scope.go:122] "RemoveContainer" containerID="92c85cbef00fddb14f9abe1d8451dd0ba232b9a2a40d2967fa85dafbb0452f6f" Oct 30 19:40:55.773924 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.773902 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3a55af38-openshift-default-5bb87d5f65-hkdt7"] Oct 30 19:40:55.778425 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.778406 5091 scope.go:122] "RemoveContainer" containerID="92c85cbef00fddb14f9abe1d8451dd0ba232b9a2a40d2967fa85dafbb0452f6f" Oct 30 19:40:55.778669 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:40:55.778651 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92c85cbef00fddb14f9abe1d8451dd0ba232b9a2a40d2967fa85dafbb0452f6f\": container with ID starting with 92c85cbef00fddb14f9abe1d8451dd0ba232b9a2a40d2967fa85dafbb0452f6f not found: ID does not exist" containerID="92c85cbef00fddb14f9abe1d8451dd0ba232b9a2a40d2967fa85dafbb0452f6f" Oct 30 19:40:55.778734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.778674 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92c85cbef00fddb14f9abe1d8451dd0ba232b9a2a40d2967fa85dafbb0452f6f"} err="failed to get container status \"92c85cbef00fddb14f9abe1d8451dd0ba232b9a2a40d2967fa85dafbb0452f6f\": rpc error: code = NotFound desc = could not find container \"92c85cbef00fddb14f9abe1d8451dd0ba232b9a2a40d2967fa85dafbb0452f6f\": container with ID starting with 92c85cbef00fddb14f9abe1d8451dd0ba232b9a2a40d2967fa85dafbb0452f6f not found: ID does not exist" Oct 30 19:40:55.801576 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.801529 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28"] Oct 30 19:40:55.810339 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.810313 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0897eecc-75774c584b-cqx28"] Oct 30 19:40:55.830478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.830456 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h"] Oct 30 19:40:55.839632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.839610 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0897eecc-f5b95bc79-4x62h"] Oct 30 19:40:55.887256 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:55.887077 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:56.089127 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.088993 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:56.153316 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.153285 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc"] Oct 30 19:40:56.153911 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.153896 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerName="istio-proxy" Oct 30 19:40:56.153963 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.153914 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerName="istio-proxy" Oct 30 19:40:56.153963 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.153934 5091 cpu_manager.go:414] "containerMap: removing container" podUID="efad46d4-e7ef-4139-9397-7d1b56faec0d" containerName="revoke-keys" Oct 30 19:40:56.153963 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.153941 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efad46d4-e7ef-4139-9397-7d1b56faec0d" containerName="revoke-keys" Oct 30 19:40:56.154060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.153970 5091 cpu_manager.go:414] "containerMap: removing container" podUID="91802908-1257-4dc4-8e5b-baaeb4dbdfbb" containerName="payload-pre-processing" Oct 30 19:40:56.154060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.153979 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91802908-1257-4dc4-8e5b-baaeb4dbdfbb" containerName="payload-pre-processing" Oct 30 19:40:56.154060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154003 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8f46b66e-1ca4-41ca-9be7-040cb8dec18a" containerName="payload-pre-processing" Oct 30 19:40:56.154060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154011 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f46b66e-1ca4-41ca-9be7-040cb8dec18a" containerName="payload-pre-processing" Oct 30 19:40:56.154060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154037 5091 cpu_manager.go:414] "containerMap: removing container" podUID="34ec41bf-040e-46ff-82df-7a3f73929cc8" containerName="payload-processing" Oct 30 19:40:56.154060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154044 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34ec41bf-040e-46ff-82df-7a3f73929cc8" containerName="payload-processing" Oct 30 19:40:56.154060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154054 5091 cpu_manager.go:414] "containerMap: removing container" podUID="550f7a23-825c-45dc-a2bf-ee67264eed56" containerName="payload-processing" Oct 30 19:40:56.154279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154061 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="550f7a23-825c-45dc-a2bf-ee67264eed56" containerName="payload-processing" Oct 30 19:40:56.154279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154072 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6733505d-3dd3-462d-a96a-ee60c3764e64" containerName="payload-pre-processing" Oct 30 19:40:56.154279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154079 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6733505d-3dd3-462d-a96a-ee60c3764e64" containerName="payload-pre-processing" Oct 30 19:40:56.154279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154094 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8ec2b758-fec8-49ce-a483-e3f91af11c33" containerName="payload-processing" Oct 30 19:40:56.154279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154101 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec2b758-fec8-49ce-a483-e3f91af11c33" containerName="payload-processing" Oct 30 19:40:56.154279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154214 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f46b66e-1ca4-41ca-9be7-040cb8dec18a" containerName="payload-pre-processing" Oct 30 19:40:56.154279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154229 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="34ec41bf-040e-46ff-82df-7a3f73929cc8" containerName="payload-processing" Oct 30 19:40:56.154279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154238 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerName="istio-proxy" Oct 30 19:40:56.154279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154245 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="550f7a23-825c-45dc-a2bf-ee67264eed56" containerName="payload-processing" Oct 30 19:40:56.154279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154254 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6733505d-3dd3-462d-a96a-ee60c3764e64" containerName="payload-pre-processing" Oct 30 19:40:56.154279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154264 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec2b758-fec8-49ce-a483-e3f91af11c33" containerName="payload-processing" Oct 30 19:40:56.154279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154274 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="efad46d4-e7ef-4139-9397-7d1b56faec0d" containerName="revoke-keys" Oct 30 19:40:56.154598 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.154285 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="91802908-1257-4dc4-8e5b-baaeb4dbdfbb" containerName="payload-pre-processing" Oct 30 19:40:56.159436 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.159415 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.165140 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.165119 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-8836a8-openshift-default-dockercfg-dvhtk\"" Oct 30 19:40:56.174850 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.174811 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc"] Oct 30 19:40:56.257170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.252398 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.257170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.252498 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjg48\" (UniqueName: \"kubernetes.io/projected/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-kube-api-access-sjg48\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.257170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.252551 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.257170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.252583 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.257170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.252608 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.257170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.252646 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.257170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.252704 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.257170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.252770 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.257170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.252807 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.293705 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.293660 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:56.354129 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.354038 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.354129 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.354124 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.354405 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.354198 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.354405 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.354270 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.354555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.354527 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.354991 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.354964 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.355068 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.355004 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjg48\" (UniqueName: \"kubernetes.io/projected/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-kube-api-access-sjg48\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.355068 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.355055 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.355201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.355091 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.355201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.355117 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.355201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.355184 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.355735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.355533 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.355735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.355626 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.355735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.355685 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.358506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.357395 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.359031 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.359010 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.373459 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.373432 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.374486 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.374462 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjg48\" (UniqueName: \"kubernetes.io/projected/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-kube-api-access-sjg48\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.462639 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.462597 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34ec41bf-040e-46ff-82df-7a3f73929cc8" path="/var/lib/kubelet/pods/34ec41bf-040e-46ff-82df-7a3f73929cc8/volumes" Oct 30 19:40:56.463133 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.463108 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6733505d-3dd3-462d-a96a-ee60c3764e64" path="/var/lib/kubelet/pods/6733505d-3dd3-462d-a96a-ee60c3764e64/volumes" Oct 30 19:40:56.464690 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.464666 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8253e5fe-3ea9-4367-b43d-274dc85176f8" path="/var/lib/kubelet/pods/8253e5fe-3ea9-4367-b43d-274dc85176f8/volumes" Oct 30 19:40:56.465615 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.465595 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ec2b758-fec8-49ce-a483-e3f91af11c33" path="/var/lib/kubelet/pods/8ec2b758-fec8-49ce-a483-e3f91af11c33/volumes" Oct 30 19:40:56.466001 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.465982 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f46b66e-1ca4-41ca-9be7-040cb8dec18a" path="/var/lib/kubelet/pods/8f46b66e-1ca4-41ca-9be7-040cb8dec18a/volumes" Oct 30 19:40:56.477233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.477208 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:56.478916 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.478894 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:56.634600 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.634390 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:56.772406 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.772380 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc"] Oct 30 19:40:56.773045 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:56.773020 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2fb8632_e65d_4559_a9ad_1a6ecd4e978a.slice/crio-0c5c3ffedcd73d0d5a402bf28ddb89cb0f97d0f35f59f5c312afaea0cdd26df2 WatchSource:0}: Error finding container 0c5c3ffedcd73d0d5a402bf28ddb89cb0f97d0f35f59f5c312afaea0cdd26df2: Status 404 returned error can't find the container with id 0c5c3ffedcd73d0d5a402bf28ddb89cb0f97d0f35f59f5c312afaea0cdd26df2 Oct 30 19:40:56.775188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.775139 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:56.775288 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.775235 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:56.775288 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.775276 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:40:56.804200 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.804140 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:57.000178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:56.999774 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:57.089619 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.089579 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs"] Oct 30 19:40:57.090332 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.090312 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerName="istio-proxy" Oct 30 19:40:57.090377 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.090335 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerName="istio-proxy" Oct 30 19:40:57.090478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.090466 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8253e5fe-3ea9-4367-b43d-274dc85176f8" containerName="istio-proxy" Oct 30 19:40:57.094101 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.094080 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" Oct 30 19:40:57.104789 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.104766 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-f06af6-dockercfg-j854r\"" Oct 30 19:40:57.105551 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.105531 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-f06af6\"" Oct 30 19:40:57.107853 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.107818 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs"] Oct 30 19:40:57.167485 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.167374 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlzv2\" (UniqueName: \"kubernetes.io/projected/77c5214d-bdc6-4294-b702-704c79eaa0aa-kube-api-access-wlzv2\") pod \"payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs\" (UID: \"77c5214d-bdc6-4294-b702-704c79eaa0aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" Oct 30 19:40:57.167652 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.167524 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c5214d-bdc6-4294-b702-704c79eaa0aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs\" (UID: \"77c5214d-bdc6-4294-b702-704c79eaa0aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" Oct 30 19:40:57.203184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.197324 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:57.268911 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.268879 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlzv2\" (UniqueName: \"kubernetes.io/projected/77c5214d-bdc6-4294-b702-704c79eaa0aa-kube-api-access-wlzv2\") pod \"payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs\" (UID: \"77c5214d-bdc6-4294-b702-704c79eaa0aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" Oct 30 19:40:57.269095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.268997 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c5214d-bdc6-4294-b702-704c79eaa0aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs\" (UID: \"77c5214d-bdc6-4294-b702-704c79eaa0aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" Oct 30 19:40:57.269833 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.269812 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c5214d-bdc6-4294-b702-704c79eaa0aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs\" (UID: \"77c5214d-bdc6-4294-b702-704c79eaa0aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" Oct 30 19:40:57.295669 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.295641 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlzv2\" (UniqueName: \"kubernetes.io/projected/77c5214d-bdc6-4294-b702-704c79eaa0aa-kube-api-access-wlzv2\") pod \"payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs\" (UID: \"77c5214d-bdc6-4294-b702-704c79eaa0aa\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" Oct 30 19:40:57.414895 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.413560 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" Oct 30 19:40:57.772570 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.772431 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs"] Oct 30 19:40:57.787018 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.786989 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" event={"ID":"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a","Type":"ContainerStarted","Data":"53c68f7af2201706beb91d56fafb0e1723121d0329ed60ead9a128cc3c7d41dd"} Oct 30 19:40:57.787112 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.787052 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" event={"ID":"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a","Type":"ContainerStarted","Data":"0c5c3ffedcd73d0d5a402bf28ddb89cb0f97d0f35f59f5c312afaea0cdd26df2"} Oct 30 19:40:57.802408 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.802372 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:57.820616 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:57.820571 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" podStartSLOduration=1.820558106 podStartE2EDuration="1.820558106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:57.818975318 +0000 UTC m=+577.514183300" watchObservedRunningTime="2026-10-30 19:40:57.820558106 +0000 UTC m=+577.515766061" Oct 30 19:40:58.075620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.075523 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:58.121328 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.121292 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm"] Oct 30 19:40:58.125498 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.125476 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" Oct 30 19:40:58.135858 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.135836 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-2a3e1b\"" Oct 30 19:40:58.135858 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.137822 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-2a3e1b-dockercfg-vn5z8\"" Oct 30 19:40:58.140910 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.140887 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm"] Oct 30 19:40:58.180858 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.180818 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74"] Oct 30 19:40:58.186459 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.186409 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02aaf432-1c15-4e88-9978-23dc3f9ee27d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm\" (UID: \"02aaf432-1c15-4e88-9978-23dc3f9ee27d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" Oct 30 19:40:58.186639 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.186501 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfq6h\" (UniqueName: \"kubernetes.io/projected/02aaf432-1c15-4e88-9978-23dc3f9ee27d-kube-api-access-pfq6h\") pod \"payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm\" (UID: \"02aaf432-1c15-4e88-9978-23dc3f9ee27d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" Oct 30 19:40:58.186839 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.186818 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" Oct 30 19:40:58.194244 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.194220 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74"] Oct 30 19:40:58.195650 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.195626 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k"] Oct 30 19:40:58.200919 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.200894 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-d05eeacd-hgr2k"] Oct 30 19:40:58.267169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.265991 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:58.287455 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.287424 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76728a5-e155-435a-a692-cc9afee10063-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74\" (UID: \"f76728a5-e155-435a-a692-cc9afee10063\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" Oct 30 19:40:58.287590 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.287501 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02aaf432-1c15-4e88-9978-23dc3f9ee27d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm\" (UID: \"02aaf432-1c15-4e88-9978-23dc3f9ee27d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" Oct 30 19:40:58.287590 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.287567 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2v7v\" (UniqueName: \"kubernetes.io/projected/f76728a5-e155-435a-a692-cc9afee10063-kube-api-access-s2v7v\") pod \"payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74\" (UID: \"f76728a5-e155-435a-a692-cc9afee10063\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" Oct 30 19:40:58.287682 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.287609 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfq6h\" (UniqueName: \"kubernetes.io/projected/02aaf432-1c15-4e88-9978-23dc3f9ee27d-kube-api-access-pfq6h\") pod \"payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm\" (UID: \"02aaf432-1c15-4e88-9978-23dc3f9ee27d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" Oct 30 19:40:58.288251 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.288234 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02aaf432-1c15-4e88-9978-23dc3f9ee27d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm\" (UID: \"02aaf432-1c15-4e88-9978-23dc3f9ee27d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" Oct 30 19:40:58.317211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.317126 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfq6h\" (UniqueName: \"kubernetes.io/projected/02aaf432-1c15-4e88-9978-23dc3f9ee27d-kube-api-access-pfq6h\") pod \"payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm\" (UID: \"02aaf432-1c15-4e88-9978-23dc3f9ee27d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" Oct 30 19:40:58.343292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.343219 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7"] Oct 30 19:40:58.343508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.343479 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" podUID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerName="istio-proxy" containerID="cri-o://a13c1ebc45dfc5896b52ffbd6ab01be98a408a78be9370d22f72a991ab85212b" gracePeriod=30 Oct 30 19:40:58.389015 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.388984 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76728a5-e155-435a-a692-cc9afee10063-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74\" (UID: \"f76728a5-e155-435a-a692-cc9afee10063\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" Oct 30 19:40:58.389234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.389037 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2v7v\" (UniqueName: \"kubernetes.io/projected/f76728a5-e155-435a-a692-cc9afee10063-kube-api-access-s2v7v\") pod \"payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74\" (UID: \"f76728a5-e155-435a-a692-cc9afee10063\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" Oct 30 19:40:58.389747 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.389721 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76728a5-e155-435a-a692-cc9afee10063-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74\" (UID: \"f76728a5-e155-435a-a692-cc9afee10063\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" Oct 30 19:40:58.406874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.406853 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2v7v\" (UniqueName: \"kubernetes.io/projected/f76728a5-e155-435a-a692-cc9afee10063-kube-api-access-s2v7v\") pod \"payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74\" (UID: \"f76728a5-e155-435a-a692-cc9afee10063\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" Oct 30 19:40:58.428739 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.428709 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:58.441488 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.441461 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efad46d4-e7ef-4139-9397-7d1b56faec0d" path="/var/lib/kubelet/pods/efad46d4-e7ef-4139-9397-7d1b56faec0d/volumes" Oct 30 19:40:58.442140 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.441998 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" Oct 30 19:40:58.484166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.481483 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:58.500168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.499434 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:58.509803 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.509335 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" Oct 30 19:40:58.802026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.801920 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" event={"ID":"77c5214d-bdc6-4294-b702-704c79eaa0aa","Type":"ContainerStarted","Data":"fe31f14a2f911716cde58898ba1e7e75f331e4b36ad089cf3915394547996c9c"} Oct 30 19:40:58.802026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.801997 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" event={"ID":"77c5214d-bdc6-4294-b702-704c79eaa0aa","Type":"ContainerStarted","Data":"d838031941d3f7a1ebd582267d70e7051cf69ba93a858374639071548237e018"} Oct 30 19:40:58.802543 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.802165 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" Oct 30 19:40:58.802543 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.802351 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:58.804249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.804230 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:40:58.819028 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:58.819003 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02aaf432_1c15_4e88_9978_23dc3f9ee27d.slice/crio-e83b7b522f4c50d320db6a00d61eb11b4d15b350ebfd383d77cfce5eb4690de5 WatchSource:0}: Error finding container e83b7b522f4c50d320db6a00d61eb11b4d15b350ebfd383d77cfce5eb4690de5: Status 404 returned error can't find the container with id e83b7b522f4c50d320db6a00d61eb11b4d15b350ebfd383d77cfce5eb4690de5 Oct 30 19:40:58.820284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.820260 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm"] Oct 30 19:40:58.828913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.828876 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" podStartSLOduration=1.8288646910000002 podStartE2EDuration="1.828864691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:58.828506487 +0000 UTC m=+578.523714448" watchObservedRunningTime="2026-10-30 19:40:58.828864691 +0000 UTC m=+578.524072648" Oct 30 19:40:58.843445 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:40:58.843422 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76728a5_e155_435a_a692_cc9afee10063.slice/crio-54b4a301931fc47759dd68e1ade1cca0761eb75049324aac448c33d335b829c1 WatchSource:0}: Error finding container 54b4a301931fc47759dd68e1ade1cca0761eb75049324aac448c33d335b829c1: Status 404 returned error can't find the container with id 54b4a301931fc47759dd68e1ade1cca0761eb75049324aac448c33d335b829c1 Oct 30 19:40:58.844240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:58.844211 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74"] Oct 30 19:40:59.234581 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:59.234454 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:59.386686 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:59.386625 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:59.671269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:59.671233 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:40:59.809670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:59.809630 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" event={"ID":"f76728a5-e155-435a-a692-cc9afee10063","Type":"ContainerStarted","Data":"da0387542447795de10080e05ff16384c4ec925ebb6021183f68263114349a9a"} Oct 30 19:40:59.810225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:59.810203 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" event={"ID":"f76728a5-e155-435a-a692-cc9afee10063","Type":"ContainerStarted","Data":"54b4a301931fc47759dd68e1ade1cca0761eb75049324aac448c33d335b829c1"} Oct 30 19:40:59.810325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:59.810315 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" Oct 30 19:40:59.811087 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:59.811069 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" event={"ID":"02aaf432-1c15-4e88-9978-23dc3f9ee27d","Type":"ContainerStarted","Data":"88fde8d5cb82274a117df04703609db069be952cfb2c7a947c3f76c20d9f8f76"} Oct 30 19:40:59.811202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:59.811189 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" event={"ID":"02aaf432-1c15-4e88-9978-23dc3f9ee27d","Type":"ContainerStarted","Data":"e83b7b522f4c50d320db6a00d61eb11b4d15b350ebfd383d77cfce5eb4690de5"} Oct 30 19:40:59.811359 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:59.811347 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" Oct 30 19:40:59.838379 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:59.838316 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" podStartSLOduration=1.838296374 podStartE2EDuration="1.838296374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:59.838068969 +0000 UTC m=+579.533276932" watchObservedRunningTime="2026-10-30 19:40:59.838296374 +0000 UTC m=+579.533504338" Oct 30 19:40:59.877169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:40:59.876501 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:00.286120 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.286088 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:41:00.313089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.313002 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" podStartSLOduration=2.31298893 podStartE2EDuration="2.31298893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:40:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:40:59.893629643 +0000 UTC m=+579.588837596" watchObservedRunningTime="2026-10-30 19:41:00.31298893 +0000 UTC m=+580.008196890" Oct 30 19:41:00.323664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.323629 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-credential-socket\") pod \"d9cb9939-3547-4232-b6c0-d2a40803a666\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " Oct 30 19:41:00.323816 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.323693 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-podinfo\") pod \"d9cb9939-3547-4232-b6c0-d2a40803a666\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " Oct 30 19:41:00.323816 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.323762 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-data\") pod \"d9cb9939-3547-4232-b6c0-d2a40803a666\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " Oct 30 19:41:00.323888 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.323827 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-token\" (UniqueName: \"kubernetes.io/projected/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-token\") pod \"d9cb9939-3547-4232-b6c0-d2a40803a666\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " Oct 30 19:41:00.323888 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.323859 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9cb9939-3547-4232-b6c0-d2a40803a666-kube-api-access-p867c\" (UniqueName: \"kubernetes.io/projected/d9cb9939-3547-4232-b6c0-d2a40803a666-kube-api-access-p867c\") pod \"d9cb9939-3547-4232-b6c0-d2a40803a666\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " Oct 30 19:41:00.323953 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.323899 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-workload-certs\") pod \"d9cb9939-3547-4232-b6c0-d2a40803a666\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " Oct 30 19:41:00.324186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.323981 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9cb9939-3547-4232-b6c0-d2a40803a666-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9cb9939-3547-4232-b6c0-d2a40803a666-istiod-ca-cert\") pod \"d9cb9939-3547-4232-b6c0-d2a40803a666\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " Oct 30 19:41:00.324186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.324027 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-envoy\") pod \"d9cb9939-3547-4232-b6c0-d2a40803a666\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " Oct 30 19:41:00.324186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.324078 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-workload-socket\") pod \"d9cb9939-3547-4232-b6c0-d2a40803a666\" (UID: \"d9cb9939-3547-4232-b6c0-d2a40803a666\") " Oct 30 19:41:00.324567 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.324501 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-credential-socket" pod "d9cb9939-3547-4232-b6c0-d2a40803a666" (UID: "d9cb9939-3547-4232-b6c0-d2a40803a666"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:00.324732 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.324698 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-data" pod "d9cb9939-3547-4232-b6c0-d2a40803a666" (UID: "d9cb9939-3547-4232-b6c0-d2a40803a666"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:00.324931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.324895 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-workload-certs" pod "d9cb9939-3547-4232-b6c0-d2a40803a666" (UID: "d9cb9939-3547-4232-b6c0-d2a40803a666"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:00.324998 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.324952 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-workload-socket" pod "d9cb9939-3547-4232-b6c0-d2a40803a666" (UID: "d9cb9939-3547-4232-b6c0-d2a40803a666"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:00.325060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.325000 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9cb9939-3547-4232-b6c0-d2a40803a666-istiod-ca-cert" pod "d9cb9939-3547-4232-b6c0-d2a40803a666" (UID: "d9cb9939-3547-4232-b6c0-d2a40803a666"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:00.327429 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.327403 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-token" pod "d9cb9939-3547-4232-b6c0-d2a40803a666" (UID: "d9cb9939-3547-4232-b6c0-d2a40803a666"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:00.327524 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.327488 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-podinfo" pod "d9cb9939-3547-4232-b6c0-d2a40803a666" (UID: "d9cb9939-3547-4232-b6c0-d2a40803a666"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:41:00.327664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.327636 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9cb9939-3547-4232-b6c0-d2a40803a666-kube-api-access-p867c" pod "d9cb9939-3547-4232-b6c0-d2a40803a666" (UID: "d9cb9939-3547-4232-b6c0-d2a40803a666"). InnerVolumeSpecName "kube-api-access-p867c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:00.327798 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.327779 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-envoy" pod "d9cb9939-3547-4232-b6c0-d2a40803a666" (UID: "d9cb9939-3547-4232-b6c0-d2a40803a666"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:00.425984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.425869 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:00.425984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.425898 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:00.425984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.425907 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:00.425984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.425918 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p867c\" (UniqueName: \"kubernetes.io/projected/d9cb9939-3547-4232-b6c0-d2a40803a666-kube-api-access-p867c\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:00.425984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.425930 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:00.425984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.425938 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9cb9939-3547-4232-b6c0-d2a40803a666-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:00.425984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.425947 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:00.425984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.425955 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:00.425984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.425965 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9cb9939-3547-4232-b6c0-d2a40803a666-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:00.600782 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.600742 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:00.776268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.770082 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:00.823168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.822724 5091 generic.go:356] "Generic (PLEG): container finished" podID="d9cb9939-3547-4232-b6c0-d2a40803a666" containerID="d28cf35e02c44c0cdacaa0a9a423dd3bf0e7c2109c891d72f2087520d1effe42" exitCode=0 Oct 30 19:41:00.823634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.823557 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" Oct 30 19:41:00.825590 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.824812 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" event={"ID":"d9cb9939-3547-4232-b6c0-d2a40803a666","Type":"ContainerDied","Data":"d28cf35e02c44c0cdacaa0a9a423dd3bf0e7c2109c891d72f2087520d1effe42"} Oct 30 19:41:00.825590 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.824865 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc" event={"ID":"d9cb9939-3547-4232-b6c0-d2a40803a666","Type":"ContainerDied","Data":"aed4529108ddb937b08a60f08ff6ab442ed75de62722178d00321df4044435f4"} Oct 30 19:41:00.825590 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.824898 5091 scope.go:122] "RemoveContainer" containerID="d28cf35e02c44c0cdacaa0a9a423dd3bf0e7c2109c891d72f2087520d1effe42" Oct 30 19:41:00.841064 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.841039 5091 scope.go:122] "RemoveContainer" containerID="fefd57fd8b93e0f45f5d08c0c0e949e413a155fdede32c576f4403595fafd81d" Oct 30 19:41:00.878258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.877610 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc"] Oct 30 19:41:00.890500 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.890472 5091 scope.go:122] "RemoveContainer" containerID="d28cf35e02c44c0cdacaa0a9a423dd3bf0e7c2109c891d72f2087520d1effe42" Oct 30 19:41:00.893742 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.893713 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0897eecc-openshift-default-7b455f4b85-bhftc"] Oct 30 19:41:00.895929 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:41:00.895896 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d28cf35e02c44c0cdacaa0a9a423dd3bf0e7c2109c891d72f2087520d1effe42\": container with ID starting with d28cf35e02c44c0cdacaa0a9a423dd3bf0e7c2109c891d72f2087520d1effe42 not found: ID does not exist" containerID="d28cf35e02c44c0cdacaa0a9a423dd3bf0e7c2109c891d72f2087520d1effe42" Oct 30 19:41:00.896036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.895943 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d28cf35e02c44c0cdacaa0a9a423dd3bf0e7c2109c891d72f2087520d1effe42"} err="failed to get container status \"d28cf35e02c44c0cdacaa0a9a423dd3bf0e7c2109c891d72f2087520d1effe42\": rpc error: code = NotFound desc = could not find container \"d28cf35e02c44c0cdacaa0a9a423dd3bf0e7c2109c891d72f2087520d1effe42\": container with ID starting with d28cf35e02c44c0cdacaa0a9a423dd3bf0e7c2109c891d72f2087520d1effe42 not found: ID does not exist" Oct 30 19:41:00.896036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.895986 5091 scope.go:122] "RemoveContainer" containerID="fefd57fd8b93e0f45f5d08c0c0e949e413a155fdede32c576f4403595fafd81d" Oct 30 19:41:00.896809 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:41:00.896781 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fefd57fd8b93e0f45f5d08c0c0e949e413a155fdede32c576f4403595fafd81d\": container with ID starting with fefd57fd8b93e0f45f5d08c0c0e949e413a155fdede32c576f4403595fafd81d not found: ID does not exist" containerID="fefd57fd8b93e0f45f5d08c0c0e949e413a155fdede32c576f4403595fafd81d" Oct 30 19:41:00.896926 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.896817 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fefd57fd8b93e0f45f5d08c0c0e949e413a155fdede32c576f4403595fafd81d"} err="failed to get container status \"fefd57fd8b93e0f45f5d08c0c0e949e413a155fdede32c576f4403595fafd81d\": rpc error: code = NotFound desc = could not find container \"fefd57fd8b93e0f45f5d08c0c0e949e413a155fdede32c576f4403595fafd81d\": container with ID starting with fefd57fd8b93e0f45f5d08c0c0e949e413a155fdede32c576f4403595fafd81d not found: ID does not exist" Oct 30 19:41:00.971388 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:00.971353 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:02.070171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.068559 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:02.082800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.082771 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk"] Oct 30 19:41:02.083540 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.083522 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d9cb9939-3547-4232-b6c0-d2a40803a666" containerName="istio-proxy" Oct 30 19:41:02.083581 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.083543 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9cb9939-3547-4232-b6c0-d2a40803a666" containerName="istio-proxy" Oct 30 19:41:02.083617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.083578 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d9cb9939-3547-4232-b6c0-d2a40803a666" containerName="istio-proxy" Oct 30 19:41:02.083617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.083586 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9cb9939-3547-4232-b6c0-d2a40803a666" containerName="istio-proxy" Oct 30 19:41:02.083693 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.083681 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9cb9939-3547-4232-b6c0-d2a40803a666" containerName="istio-proxy" Oct 30 19:41:02.083730 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.083700 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9cb9939-3547-4232-b6c0-d2a40803a666" containerName="istio-proxy" Oct 30 19:41:02.088747 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.088728 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" Oct 30 19:41:02.094879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.094766 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-8836a8\"" Oct 30 19:41:02.097122 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.096637 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk"] Oct 30 19:41:02.128455 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.128424 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7"] Oct 30 19:41:02.132092 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.132072 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" Oct 30 19:41:02.144317 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.144297 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-8836a8-dockercfg-tgqsv\"" Oct 30 19:41:02.144745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.144722 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7"] Oct 30 19:41:02.177167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.155876 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7ca223-be1f-4f97-aef8-cf286b756081-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk\" (UID: \"6f7ca223-be1f-4f97-aef8-cf286b756081\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" Oct 30 19:41:02.177167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.155921 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmvwg\" (UniqueName: \"kubernetes.io/projected/6f7ca223-be1f-4f97-aef8-cf286b756081-kube-api-access-xmvwg\") pod \"payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk\" (UID: \"6f7ca223-be1f-4f97-aef8-cf286b756081\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" Oct 30 19:41:02.177167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.155980 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5a523a-97be-4828-b79e-ec093df306ff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7\" (UID: \"8b5a523a-97be-4828-b79e-ec093df306ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" Oct 30 19:41:02.177167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.156054 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4p57\" (UniqueName: \"kubernetes.io/projected/8b5a523a-97be-4828-b79e-ec093df306ff-kube-api-access-m4p57\") pod \"payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7\" (UID: \"8b5a523a-97be-4828-b79e-ec093df306ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" Oct 30 19:41:02.233166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.227398 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:02.274169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.257021 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7ca223-be1f-4f97-aef8-cf286b756081-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk\" (UID: \"6f7ca223-be1f-4f97-aef8-cf286b756081\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" Oct 30 19:41:02.274169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.257073 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmvwg\" (UniqueName: \"kubernetes.io/projected/6f7ca223-be1f-4f97-aef8-cf286b756081-kube-api-access-xmvwg\") pod \"payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk\" (UID: \"6f7ca223-be1f-4f97-aef8-cf286b756081\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" Oct 30 19:41:02.274169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.257169 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5a523a-97be-4828-b79e-ec093df306ff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7\" (UID: \"8b5a523a-97be-4828-b79e-ec093df306ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" Oct 30 19:41:02.274169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.257264 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4p57\" (UniqueName: \"kubernetes.io/projected/8b5a523a-97be-4828-b79e-ec093df306ff-kube-api-access-m4p57\") pod \"payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7\" (UID: \"8b5a523a-97be-4828-b79e-ec093df306ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" Oct 30 19:41:02.274169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.258579 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7ca223-be1f-4f97-aef8-cf286b756081-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk\" (UID: \"6f7ca223-be1f-4f97-aef8-cf286b756081\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" Oct 30 19:41:02.274169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.259651 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5a523a-97be-4828-b79e-ec093df306ff-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7\" (UID: \"8b5a523a-97be-4828-b79e-ec093df306ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" Oct 30 19:41:02.295467 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.295264 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4p57\" (UniqueName: \"kubernetes.io/projected/8b5a523a-97be-4828-b79e-ec093df306ff-kube-api-access-m4p57\") pod \"payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7\" (UID: \"8b5a523a-97be-4828-b79e-ec093df306ff\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" Oct 30 19:41:02.299047 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.299018 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmvwg\" (UniqueName: \"kubernetes.io/projected/6f7ca223-be1f-4f97-aef8-cf286b756081-kube-api-access-xmvwg\") pod \"payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk\" (UID: \"6f7ca223-be1f-4f97-aef8-cf286b756081\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" Oct 30 19:41:02.407932 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.407842 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" Oct 30 19:41:02.441472 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.441443 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9cb9939-3547-4232-b6c0-d2a40803a666" path="/var/lib/kubelet/pods/d9cb9939-3547-4232-b6c0-d2a40803a666/volumes" Oct 30 19:41:02.484122 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.483341 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" Oct 30 19:41:02.493675 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.493643 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:02.741793 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.741764 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk"] Oct 30 19:41:02.839207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.839178 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" event={"ID":"6f7ca223-be1f-4f97-aef8-cf286b756081","Type":"ContainerStarted","Data":"3c8404bdd648daa3bd40a43025c14b86e9217c056b2e16747427bd45ee6c66f7"} Oct 30 19:41:02.855477 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:02.847931 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7"] Oct 30 19:41:03.258402 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.258318 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:03.414115 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.414086 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:03.683360 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.683333 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:41:03.813731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.811549 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:03.825582 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.825544 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-token\" (UniqueName: \"kubernetes.io/projected/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-token\") pod \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " Oct 30 19:41:03.825772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.825643 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-data\") pod \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " Oct 30 19:41:03.825772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.825700 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-podinfo\") pod \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " Oct 30 19:41:03.825772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.825732 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-credential-socket\") pod \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " Oct 30 19:41:03.825772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.825763 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-kube-api-access-czctk\" (UniqueName: \"kubernetes.io/projected/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-kube-api-access-czctk\") pod \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " Oct 30 19:41:03.825952 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.825819 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-envoy\") pod \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " Oct 30 19:41:03.825952 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.825895 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-workload-socket\") pod \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " Oct 30 19:41:03.825952 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.825935 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-data" pod "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" (UID: "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:03.826081 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.825944 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istiod-ca-cert\") pod \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " Oct 30 19:41:03.826124 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.826075 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-workload-certs\") pod \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\" (UID: \"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8\") " Oct 30 19:41:03.826516 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.826181 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-credential-socket" pod "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" (UID: "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:03.826516 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.826290 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istiod-ca-cert" pod "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" (UID: "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:41:03.826516 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.826490 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-workload-certs" pod "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" (UID: "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:03.826711 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.826513 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-workload-socket" pod "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" (UID: "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:03.826755 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.826741 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:03.826798 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.826762 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:03.826798 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.826777 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:03.826798 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.826790 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:03.826935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.826806 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:03.828173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.828121 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-podinfo" pod "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" (UID: "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:41:03.828611 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.828582 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-token" pod "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" (UID: "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:03.828706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.828683 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-envoy" pod "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" (UID: "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:41:03.831042 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.830997 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-kube-api-access-czctk" pod "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" (UID: "9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8"). InnerVolumeSpecName "kube-api-access-czctk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:41:03.848157 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.848112 5091 generic.go:356] "Generic (PLEG): container finished" podID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerID="a13c1ebc45dfc5896b52ffbd6ab01be98a408a78be9370d22f72a991ab85212b" exitCode=0 Oct 30 19:41:03.848260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.848184 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" Oct 30 19:41:03.848260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.848181 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" event={"ID":"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8","Type":"ContainerDied","Data":"a13c1ebc45dfc5896b52ffbd6ab01be98a408a78be9370d22f72a991ab85212b"} Oct 30 19:41:03.848260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.848243 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7" event={"ID":"9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8","Type":"ContainerDied","Data":"1ca4ce270011c4b7bbb800188f1dff2418c43364205807cbd391870a053bea90"} Oct 30 19:41:03.848412 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.848274 5091 scope.go:122] "RemoveContainer" containerID="a13c1ebc45dfc5896b52ffbd6ab01be98a408a78be9370d22f72a991ab85212b" Oct 30 19:41:03.850726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.850702 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" event={"ID":"6f7ca223-be1f-4f97-aef8-cf286b756081","Type":"ContainerStarted","Data":"35c301b70ba0f174f9a475980824e99b36ae596fc12b55cb617a4ee3199ff7ba"} Oct 30 19:41:03.850893 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.850877 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" Oct 30 19:41:03.852626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.852604 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" event={"ID":"8b5a523a-97be-4828-b79e-ec093df306ff","Type":"ContainerStarted","Data":"aea8a3e36f4b039fa2826060f60cac91afa29ba326559a23f428a1ba49d7412a"} Oct 30 19:41:03.852728 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.852648 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" event={"ID":"8b5a523a-97be-4828-b79e-ec093df306ff","Type":"ContainerStarted","Data":"7235132911b5418d13f6a3eb2bad614626b92008a98053456a7aa8bb839886c3"} Oct 30 19:41:03.853217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.853198 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" Oct 30 19:41:03.860238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.860212 5091 scope.go:122] "RemoveContainer" containerID="0e89c6f15fc7793aea59c2f8c71a6bd14f211ecdb5cde1b721757cb51982f197" Oct 30 19:41:03.869252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.869195 5091 scope.go:122] "RemoveContainer" containerID="a13c1ebc45dfc5896b52ffbd6ab01be98a408a78be9370d22f72a991ab85212b" Oct 30 19:41:03.869539 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:41:03.869518 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13c1ebc45dfc5896b52ffbd6ab01be98a408a78be9370d22f72a991ab85212b\": container with ID starting with a13c1ebc45dfc5896b52ffbd6ab01be98a408a78be9370d22f72a991ab85212b not found: ID does not exist" containerID="a13c1ebc45dfc5896b52ffbd6ab01be98a408a78be9370d22f72a991ab85212b" Oct 30 19:41:03.869609 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.869551 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13c1ebc45dfc5896b52ffbd6ab01be98a408a78be9370d22f72a991ab85212b"} err="failed to get container status \"a13c1ebc45dfc5896b52ffbd6ab01be98a408a78be9370d22f72a991ab85212b\": rpc error: code = NotFound desc = could not find container \"a13c1ebc45dfc5896b52ffbd6ab01be98a408a78be9370d22f72a991ab85212b\": container with ID starting with a13c1ebc45dfc5896b52ffbd6ab01be98a408a78be9370d22f72a991ab85212b not found: ID does not exist" Oct 30 19:41:03.869609 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.869578 5091 scope.go:122] "RemoveContainer" containerID="0e89c6f15fc7793aea59c2f8c71a6bd14f211ecdb5cde1b721757cb51982f197" Oct 30 19:41:03.869863 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:41:03.869845 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e89c6f15fc7793aea59c2f8c71a6bd14f211ecdb5cde1b721757cb51982f197\": container with ID starting with 0e89c6f15fc7793aea59c2f8c71a6bd14f211ecdb5cde1b721757cb51982f197 not found: ID does not exist" containerID="0e89c6f15fc7793aea59c2f8c71a6bd14f211ecdb5cde1b721757cb51982f197" Oct 30 19:41:03.869907 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.869871 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e89c6f15fc7793aea59c2f8c71a6bd14f211ecdb5cde1b721757cb51982f197"} err="failed to get container status \"0e89c6f15fc7793aea59c2f8c71a6bd14f211ecdb5cde1b721757cb51982f197\": rpc error: code = NotFound desc = could not find container \"0e89c6f15fc7793aea59c2f8c71a6bd14f211ecdb5cde1b721757cb51982f197\": container with ID starting with 0e89c6f15fc7793aea59c2f8c71a6bd14f211ecdb5cde1b721757cb51982f197 not found: ID does not exist" Oct 30 19:41:03.881266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.881219 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" podStartSLOduration=1.881203183 podStartE2EDuration="1.881203183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:03.880244836 +0000 UTC m=+583.575474908" watchObservedRunningTime="2026-10-30 19:41:03.881203183 +0000 UTC m=+583.576411142" Oct 30 19:41:03.909762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.909702 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" podStartSLOduration=1.9096866239999999 podStartE2EDuration="1.909686624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:03.907817204 +0000 UTC m=+583.603025163" watchObservedRunningTime="2026-10-30 19:41:03.909686624 +0000 UTC m=+583.604894583" Oct 30 19:41:03.928826 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.928797 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:03.929057 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.929039 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:03.929183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.929129 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czctk\" (UniqueName: \"kubernetes.io/projected/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-kube-api-access-czctk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:03.929183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.929180 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:41:03.932403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.932381 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7"] Oct 30 19:41:03.938112 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:03.938089 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-d05eeacd-openshift-default-86c5d8956f-fgrb7"] Oct 30 19:41:04.345778 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.345723 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:04.443094 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.443059 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" path="/var/lib/kubelet/pods/9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8/volumes" Oct 30 19:41:04.547048 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.547003 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:04.663265 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.663177 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk"] Oct 30 19:41:04.663735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.663717 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerName="istio-proxy" Oct 30 19:41:04.663819 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.663736 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerName="istio-proxy" Oct 30 19:41:04.663819 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.663749 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerName="istio-proxy" Oct 30 19:41:04.663819 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.663757 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerName="istio-proxy" Oct 30 19:41:04.663973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.663856 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerName="istio-proxy" Oct 30 19:41:04.663973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.663865 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cc5a3c7-ef39-48fc-bf4d-87ebc86c97f8" containerName="istio-proxy" Oct 30 19:41:04.668594 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.668574 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" Oct 30 19:41:04.678984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.678449 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk"] Oct 30 19:41:04.679447 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.679388 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-7a745c\"" Oct 30 19:41:04.692243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.691256 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:41:04.692243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.691313 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:41:04.714524 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.714481 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:04.733037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.733011 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc"] Oct 30 19:41:04.737471 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.737442 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" Oct 30 19:41:04.763361 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.763332 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc"] Oct 30 19:41:04.810563 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.810529 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-7a745c-dockercfg-b8xn6\"" Oct 30 19:41:04.847456 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.847418 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc\" (UID: \"f8550b23-cb62-4ddb-90b9-8d2b0706dd20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" Oct 30 19:41:04.847711 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.847472 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrvnk\" (UniqueName: \"kubernetes.io/projected/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-kube-api-access-zrvnk\") pod \"payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc\" (UID: \"f8550b23-cb62-4ddb-90b9-8d2b0706dd20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" Oct 30 19:41:04.847711 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.847612 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405da951-aef4-4187-8755-b264fff0c2a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk\" (UID: \"405da951-aef4-4187-8755-b264fff0c2a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" Oct 30 19:41:04.847711 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.847672 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkt6f\" (UniqueName: \"kubernetes.io/projected/405da951-aef4-4187-8755-b264fff0c2a0-kube-api-access-tkt6f\") pod \"payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk\" (UID: \"405da951-aef4-4187-8755-b264fff0c2a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" Oct 30 19:41:04.866584 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.866546 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:04.949579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.949482 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc\" (UID: \"f8550b23-cb62-4ddb-90b9-8d2b0706dd20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" Oct 30 19:41:04.949579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.949537 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrvnk\" (UniqueName: \"kubernetes.io/projected/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-kube-api-access-zrvnk\") pod \"payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc\" (UID: \"f8550b23-cb62-4ddb-90b9-8d2b0706dd20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" Oct 30 19:41:04.949813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.949609 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405da951-aef4-4187-8755-b264fff0c2a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk\" (UID: \"405da951-aef4-4187-8755-b264fff0c2a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" Oct 30 19:41:04.949813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.949680 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkt6f\" (UniqueName: \"kubernetes.io/projected/405da951-aef4-4187-8755-b264fff0c2a0-kube-api-access-tkt6f\") pod \"payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk\" (UID: \"405da951-aef4-4187-8755-b264fff0c2a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" Oct 30 19:41:04.950484 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.950455 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc\" (UID: \"f8550b23-cb62-4ddb-90b9-8d2b0706dd20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" Oct 30 19:41:04.951018 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.950994 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405da951-aef4-4187-8755-b264fff0c2a0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk\" (UID: \"405da951-aef4-4187-8755-b264fff0c2a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" Oct 30 19:41:04.967941 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.967910 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrvnk\" (UniqueName: \"kubernetes.io/projected/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-kube-api-access-zrvnk\") pod \"payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc\" (UID: \"f8550b23-cb62-4ddb-90b9-8d2b0706dd20\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" Oct 30 19:41:04.968540 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.968513 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkt6f\" (UniqueName: \"kubernetes.io/projected/405da951-aef4-4187-8755-b264fff0c2a0-kube-api-access-tkt6f\") pod \"payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk\" (UID: \"405da951-aef4-4187-8755-b264fff0c2a0\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" Oct 30 19:41:04.986473 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:04.986444 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" Oct 30 19:41:05.014352 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:05.014318 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:05.058942 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:05.058420 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" Oct 30 19:41:05.159993 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:05.159838 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:05.287688 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:05.287328 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk"] Oct 30 19:41:05.291254 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:41:05.290361 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405da951_aef4_4187_8755_b264fff0c2a0.slice/crio-c84e2db017a9d9e03937528d9518a9d06287f7041857921dc73db131234a960e WatchSource:0}: Error finding container c84e2db017a9d9e03937528d9518a9d06287f7041857921dc73db131234a960e: Status 404 returned error can't find the container with id c84e2db017a9d9e03937528d9518a9d06287f7041857921dc73db131234a960e Oct 30 19:41:05.426201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:05.417604 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc"] Oct 30 19:41:05.699980 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:05.699894 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:05.860655 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:05.860610 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:05.869871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:05.869839 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" event={"ID":"405da951-aef4-4187-8755-b264fff0c2a0","Type":"ContainerStarted","Data":"9885d65fc47a9858488a86c9c29c6839ce1b3063d236382ae821e0318aaefc61"} Oct 30 19:41:05.870039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:05.869889 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" event={"ID":"405da951-aef4-4187-8755-b264fff0c2a0","Type":"ContainerStarted","Data":"c84e2db017a9d9e03937528d9518a9d06287f7041857921dc73db131234a960e"} Oct 30 19:41:05.870039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:05.869929 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" Oct 30 19:41:05.872913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:05.872872 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" event={"ID":"f8550b23-cb62-4ddb-90b9-8d2b0706dd20","Type":"ContainerStarted","Data":"29e268e0fb5627d4bed7385eee0ab553f70cad063702b3398fbce6cda92c94ce"} Oct 30 19:41:05.873170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:05.873127 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" event={"ID":"f8550b23-cb62-4ddb-90b9-8d2b0706dd20","Type":"ContainerStarted","Data":"72c76b6c359eb3d40d0197545da52e5720e765e0dffa4ec6fac14043f4e42b22"} Oct 30 19:41:05.873319 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:05.873303 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" Oct 30 19:41:05.916699 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:05.916612 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" podStartSLOduration=1.916588468 podStartE2EDuration="1.916588468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:05.916362105 +0000 UTC m=+585.611570063" watchObservedRunningTime="2026-10-30 19:41:05.916588468 +0000 UTC m=+585.611796442" Oct 30 19:41:05.940219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:05.940141 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" podStartSLOduration=1.940128158 podStartE2EDuration="1.940128158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:05.939530712 +0000 UTC m=+585.634738669" watchObservedRunningTime="2026-10-30 19:41:05.940128158 +0000 UTC m=+585.635336116" Oct 30 19:41:06.018686 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:06.018647 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:06.226406 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:06.226371 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:06.420425 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:06.420332 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:06.880996 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:06.880911 5091 generic.go:356] "Generic (PLEG): container finished" podID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerID="31057ad55c8374fd1f70af888ca4405fe2163b86852725c4f093f48249ea9145" exitCode=0 Oct 30 19:41:06.881460 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:06.880996 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" event={"ID":"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d","Type":"ContainerDied","Data":"31057ad55c8374fd1f70af888ca4405fe2163b86852725c4f093f48249ea9145"} Oct 30 19:41:06.881708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:06.881693 5091 scope.go:122] "RemoveContainer" containerID="31057ad55c8374fd1f70af888ca4405fe2163b86852725c4f093f48249ea9145" Oct 30 19:41:06.883511 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:06.883483 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:41:06.883639 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:06.883551 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:41:06.883639 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:06.883590 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:41:07.470922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:07.470879 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:07.650664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:07.650626 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:07.892195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:07.891951 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" event={"ID":"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d","Type":"ContainerStarted","Data":"03b59195e4f6f2c3ac93f70feca33c6f70bc8def4f1822008f3b54dfadf4ae65"} Oct 30 19:41:08.299753 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:08.299715 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:41:08.300020 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:08.299998 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:41:08.300083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:08.300048 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:41:08.681609 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:08.669650 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:08.815797 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:08.815744 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:09.300062 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:09.300022 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:41:09.300464 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:09.300081 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:41:09.814756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:09.814724 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" Oct 30 19:41:10.099707 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:10.099615 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:10.300376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:10.300337 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:41:10.300787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:10.300388 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:41:10.720717 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:10.720679 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:10.828661 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:10.828628 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" Oct 30 19:41:11.222586 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:11.222544 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:11.307171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:11.306558 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:41:11.307171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:11.306625 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:41:11.833411 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:11.833383 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" Oct 30 19:41:12.218817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:12.218693 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:12.301170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:12.300536 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body= Oct 30 19:41:12.301170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:12.300600 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" Oct 30 19:41:13.188284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:13.186830 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:13.614709 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:13.614677 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:41:13.615066 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:13.615042 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:41:13.616035 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:13.616009 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 19:41:14.093855 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:14.093808 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:14.878661 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:14.878630 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" Oct 30 19:41:15.291698 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:15.291667 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:15.712863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:15.712764 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:15.876570 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:15.876543 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" Oct 30 19:41:16.002852 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:41:16.002823 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:41:16.359947 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:16.359861 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:16.512396 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:16.512354 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:16.678231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:16.678110 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:16.844743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:16.844707 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:16.884634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:16.884605 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" Oct 30 19:41:16.887111 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:16.886461 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" Oct 30 19:41:17.477684 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:17.477647 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:17.651436 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:17.651393 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:18.123299 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:18.123250 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:19.419779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:19.419739 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:20.025819 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:20.025787 5091 generic.go:356] "Generic (PLEG): container finished" podID="c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" containerID="53c68f7af2201706beb91d56fafb0e1723121d0329ed60ead9a128cc3c7d41dd" exitCode=0 Oct 30 19:41:20.026041 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:20.025873 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" event={"ID":"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a","Type":"ContainerDied","Data":"53c68f7af2201706beb91d56fafb0e1723121d0329ed60ead9a128cc3c7d41dd"} Oct 30 19:41:20.026399 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:20.026380 5091 scope.go:122] "RemoveContainer" containerID="53c68f7af2201706beb91d56fafb0e1723121d0329ed60ead9a128cc3c7d41dd" Oct 30 19:41:20.027765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:20.027738 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:41:20.027825 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:20.027810 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:41:20.027860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:20.027843 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:41:20.505217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:20.505193 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:41:20.506079 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:20.506059 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:41:20.659603 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:20.659566 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:21.034218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:21.034183 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" event={"ID":"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a","Type":"ContainerStarted","Data":"dae5d1a96ac8c54f041dfb2054e9f3a5d289d810c2c58f870e196ee9dafd753b"} Oct 30 19:41:21.479987 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:21.479899 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:41:21.480193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:21.480121 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:41:21.480277 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:21.480198 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" podUID="c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:41:22.015041 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:22.015009 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:22.480085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:22.479997 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:41:22.480085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:22.480045 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" podUID="c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:41:22.994026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:22.993992 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld"] Oct 30 19:41:23.000460 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.000436 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.011884 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.011860 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2a3e1b\"/\"default-dockercfg-9wcd7\"" Oct 30 19:41:23.012033 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.011903 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2a3e1b\"/\"openshift-service-ca.crt\"" Oct 30 19:41:23.012033 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.011943 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-2a3e1b\"/\"simulator-w4-2a3e1b-kserve-self-signed-certs\"" Oct 30 19:41:23.015326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.015308 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-2a3e1b\"/\"kube-root-ca.crt\"" Oct 30 19:41:23.075451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.075420 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld"] Oct 30 19:41:23.095842 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.095808 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-model-cache\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.096014 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.095852 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-home\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.096014 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.095935 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvvwx\" (UniqueName: \"kubernetes.io/projected/6425ff67-d04c-4a3f-a5df-fda1bf36872f-kube-api-access-xvvwx\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.096014 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.095976 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-dshm\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.096190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.096051 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6425ff67-d04c-4a3f-a5df-fda1bf36872f-tls-certs\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.096190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.096169 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-tmp-dir\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.108093 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.108053 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:23.197921 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.197883 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-model-cache\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.198233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.198199 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-home\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.198614 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.198553 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvvwx\" (UniqueName: \"kubernetes.io/projected/6425ff67-d04c-4a3f-a5df-fda1bf36872f-kube-api-access-xvvwx\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.198785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.198767 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-dshm\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.198958 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.198943 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6425ff67-d04c-4a3f-a5df-fda1bf36872f-tls-certs\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.199159 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.199130 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-tmp-dir\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.199875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.199838 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-tmp-dir\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.201158 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.200986 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-model-cache\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.202130 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.201843 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-home\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.204097 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.204061 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-dshm\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.204657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.204631 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6425ff67-d04c-4a3f-a5df-fda1bf36872f-tls-certs\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.214691 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.214653 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvvwx\" (UniqueName: \"kubernetes.io/projected/6425ff67-d04c-4a3f-a5df-fda1bf36872f-kube-api-access-xvvwx\") pod \"simulator-w4-2a3e1b-kserve-545994d9d5-zxcld\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.270170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.269199 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:23.312922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.312887 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:23.480809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.480445 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:41:23.480809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.480500 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" podUID="c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:41:23.636322 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:23.636292 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld"] Oct 30 19:41:23.636559 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:41:23.636537 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6425ff67_d04c_4a3f_a5df_fda1bf36872f.slice/crio-d020aae7fa2db36a11f1fba8051dddc1ea9c0e9e0864126c4cdbd56c9939f65a WatchSource:0}: Error finding container d020aae7fa2db36a11f1fba8051dddc1ea9c0e9e0864126c4cdbd56c9939f65a: Status 404 returned error can't find the container with id d020aae7fa2db36a11f1fba8051dddc1ea9c0e9e0864126c4cdbd56c9939f65a Oct 30 19:41:24.079141 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:24.079101 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" event={"ID":"6425ff67-d04c-4a3f-a5df-fda1bf36872f","Type":"ContainerStarted","Data":"e9e294f0b6c8cece4596bd4ba4e0be3f916c0bdd065cab3999f38d68eba0013a"} Oct 30 19:41:24.079597 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:24.079192 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" event={"ID":"6425ff67-d04c-4a3f-a5df-fda1bf36872f","Type":"ContainerStarted","Data":"d020aae7fa2db36a11f1fba8051dddc1ea9c0e9e0864126c4cdbd56c9939f65a"} Oct 30 19:41:24.126775 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:24.126720 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podStartSLOduration=2.126706858 podStartE2EDuration="2.126706858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:24.124723654 +0000 UTC m=+603.819931613" watchObservedRunningTime="2026-10-30 19:41:24.126706858 +0000 UTC m=+603.821914815" Oct 30 19:41:24.311309 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:24.311272 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:24.313445 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:24.313416 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:24.479900 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:24.479808 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:41:24.479900 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:24.479876 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" podUID="c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:41:25.290775 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:25.290739 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:25.479491 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:25.479450 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:41:25.479678 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:25.479521 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" podUID="c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:41:25.887404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:25.887065 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:26.012258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.012217 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2"] Oct 30 19:41:26.017967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.017155 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.025170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.025129 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2"] Oct 30 19:41:26.029459 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.028393 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c1fdd6\"/\"kube-root-ca.crt\"" Oct 30 19:41:26.030343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.030317 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-c1fdd6\"/\"openshift-service-ca.crt\"" Oct 30 19:41:26.030443 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.030399 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c1fdd6\"/\"premium-w3-c1fdd6-kserve-self-signed-certs\"" Oct 30 19:41:26.030751 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.030734 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-c1fdd6\"/\"default-dockercfg-qq4v5\"" Oct 30 19:41:26.119758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.119724 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:26.133858 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.133823 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-home\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.133983 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.133874 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-dshm\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.134058 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.134018 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-model-cache\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.134117 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.134065 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-tmp-dir\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.134219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.134201 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-tls-certs\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.134319 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.134305 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq5xw\" (UniqueName: \"kubernetes.io/projected/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-kube-api-access-fq5xw\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.235834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.235742 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-model-cache\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.235834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.235796 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-tmp-dir\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.236117 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.235856 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-tls-certs\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.236117 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.235927 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq5xw\" (UniqueName: \"kubernetes.io/projected/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-kube-api-access-fq5xw\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.236117 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.235998 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-home\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.236117 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.236056 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-dshm\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.236433 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.236272 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-model-cache\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.236433 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.236308 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-tmp-dir\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.236563 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.236462 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-home\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.239401 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.239365 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-dshm\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.239563 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.239544 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-tls-certs\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.256180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.255604 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq5xw\" (UniqueName: \"kubernetes.io/projected/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-kube-api-access-fq5xw\") pod \"premium-w3-c1fdd6-kserve-54d49787bc-msgg2\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.331959 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.331921 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:26.480112 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.480073 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Oct 30 19:41:26.480314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.480137 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" podUID="c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Oct 30 19:41:26.585248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.585209 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:26.646351 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.646317 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2"] Oct 30 19:41:26.646615 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:41:26.646578 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4c14c99d_b8f0_4b81_ab2c_723529cb9dbb.slice/crio-11b4b7c636d4410fbdf8fe486bebf1ccf34d77757cbb91e7b56daadfe7d67226 WatchSource:0}: Error finding container 11b4b7c636d4410fbdf8fe486bebf1ccf34d77757cbb91e7b56daadfe7d67226: Status 404 returned error can't find the container with id 11b4b7c636d4410fbdf8fe486bebf1ccf34d77757cbb91e7b56daadfe7d67226 Oct 30 19:41:26.739781 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:26.739672 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:27.105071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.105030 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" event={"ID":"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb","Type":"ContainerStarted","Data":"00c2909f65532ffbe960d01bbb286fe704307163200c69bdcb2e5eae85b26c7b"} Oct 30 19:41:27.105071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.105073 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" event={"ID":"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb","Type":"ContainerStarted","Data":"11b4b7c636d4410fbdf8fe486bebf1ccf34d77757cbb91e7b56daadfe7d67226"} Oct 30 19:41:27.139583 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.139538 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podStartSLOduration=2.139524254 podStartE2EDuration="2.139524254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:27.137017385 +0000 UTC m=+606.832225347" watchObservedRunningTime="2026-10-30 19:41:27.139524254 +0000 UTC m=+606.834732208" Oct 30 19:41:27.332298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.332267 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:27.484668 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.484584 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:41:27.484973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.484950 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:41:27.485660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.485643 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 19:41:27.736568 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.736471 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:27.746331 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.746305 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p"] Oct 30 19:41:27.750406 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.750387 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.756934 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.756908 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 19:41:27.757049 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.757003 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-gmjw8\"" Oct 30 19:41:27.757176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.757123 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 19:41:27.757289 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.757184 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 19:41:27.771440 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.771396 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p"] Oct 30 19:41:27.854311 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.854269 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.854478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.854328 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.854478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.854383 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.854564 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.854470 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.854606 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.854588 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.854657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.854615 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.854657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.854634 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrbl\" (UniqueName: \"kubernetes.io/projected/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-kube-api-access-zvrbl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.956056 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.956020 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.956294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.956062 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.956294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.956124 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.956294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.956179 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.956294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.956198 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrbl\" (UniqueName: \"kubernetes.io/projected/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-kube-api-access-zvrbl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.956577 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.956305 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.956577 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.956334 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.957039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.957001 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.958776 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.958752 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.958934 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.958886 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.959079 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.958973 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.959234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.959110 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.973209 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.973180 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:27.975738 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:27.975708 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrbl\" (UniqueName: \"kubernetes.io/projected/e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628-kube-api-access-zvrbl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-h486p\" (UID: \"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:28.065194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:28.065139 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:28.362701 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:28.362673 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p"] Oct 30 19:41:28.363368 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:41:28.363342 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6760b67_0f0c_4bcc_b36f_f3d3f1bbe628.slice/crio-724564cf4412e77870d7007d46c09f0277a78e96eba014b3a41638e890709945 WatchSource:0}: Error finding container 724564cf4412e77870d7007d46c09f0277a78e96eba014b3a41638e890709945: Status 404 returned error can't find the container with id 724564cf4412e77870d7007d46c09f0277a78e96eba014b3a41638e890709945 Oct 30 19:41:28.365323 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:28.365305 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:41:28.399977 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:28.399943 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:28.921198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:28.921162 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:29.099017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:29.098960 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:29.120949 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:29.120904 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" event={"ID":"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628","Type":"ContainerStarted","Data":"724564cf4412e77870d7007d46c09f0277a78e96eba014b3a41638e890709945"} Oct 30 19:41:30.315870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:30.315812 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:31.381754 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:31.381699 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:31.822681 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:31.822650 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:41:31.822775 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:31.822712 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:41:32.144906 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:32.144807 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" event={"ID":"e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628","Type":"ContainerStarted","Data":"d3b61f925dfd514d37671dd3e4a623df8de5adc25adb846b453161129672222c"} Oct 30 19:41:32.145060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:32.144907 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:32.146493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:32.146462 5091 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-h486p container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:41:32.146634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:32.146519 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" podUID="e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:41:32.182397 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:32.182344 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" podStartSLOduration=1.726165189 podStartE2EDuration="5.182330387s" totalImagesPullingTime="3.456165198s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:41:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:41:32.174300946 +0000 UTC m=+611.869508905" watchObservedRunningTime="2026-10-30 19:41:32.182330387 +0000 UTC m=+611.877538345" Oct 30 19:41:32.596574 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:32.596524 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:33.151188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:33.151160 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-h486p" Oct 30 19:41:33.313672 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:33.313637 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:33.327125 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:33.327099 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:33.569160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:33.569118 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:33.939236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:33.939155 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:34.169261 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:34.169225 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:41:34.179243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:34.179206 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:34.704158 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:34.704107 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:35.660827 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:35.660783 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:36.332387 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:36.332357 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:36.346013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:36.345990 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:36.389481 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:36.389441 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:36.799857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:36.799807 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:37.190210 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:37.190123 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:41:38.000098 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:38.000062 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:39.154371 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:39.154330 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:40.086010 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:40.085972 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:41.156155 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:41.156100 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:42.069625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:42.069587 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:42.338187 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:42.338071 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:43.203054 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:43.203017 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:44.165134 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:44.165095 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:45.273874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:45.273831 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:45.978543 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:41:45.978499 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:41:46.288947 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:46.288118 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:47.278533 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:47.278502 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:48.468545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:48.468508 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:48.893004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:48.892968 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:49.693634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:49.692430 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:50.610600 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:50.610560 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:51.459722 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:51.459668 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:52.495515 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:52.495466 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:53.468701 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:53.468667 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:54.502767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:54.502732 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:55.045017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:55.044975 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:55.459051 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:55.458959 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:56.488265 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:56.488234 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:57.359899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:57.359864 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:58.371342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:58.371309 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:41:59.661162 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:41:59.661112 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:00.745939 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:00.745904 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:01.609837 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:01.605479 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:02.518683 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:02.518639 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:03.558442 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:03.558407 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:04.047656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:04.047616 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:04.615775 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:04.615742 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:05.883253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:05.880793 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:06.663402 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.663367 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2"] Oct 30 19:42:06.667485 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.667466 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.679920 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.679886 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f06af6-openshift-default-dockercfg-tcjql\"" Oct 30 19:42:06.683271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.683242 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2"] Oct 30 19:42:06.811595 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.811562 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.811768 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.811627 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.811768 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.811690 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czmxm\" (UniqueName: \"kubernetes.io/projected/fee7b3f2-9725-488d-8792-357afcb18f32-kube-api-access-czmxm\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.811768 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.811730 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.811768 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.811764 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.811915 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.811877 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fee7b3f2-9725-488d-8792-357afcb18f32-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.811915 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.811911 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fee7b3f2-9725-488d-8792-357afcb18f32-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.811992 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.811980 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.812027 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.811998 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fee7b3f2-9725-488d-8792-357afcb18f32-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.862942 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.862896 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:06.912877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.912842 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.913435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.912906 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.913435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.912940 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czmxm\" (UniqueName: \"kubernetes.io/projected/fee7b3f2-9725-488d-8792-357afcb18f32-kube-api-access-czmxm\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.913435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.912997 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.913435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.913042 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.913435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.913123 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fee7b3f2-9725-488d-8792-357afcb18f32-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.913435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.913194 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fee7b3f2-9725-488d-8792-357afcb18f32-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.913435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.913278 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.913435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.913302 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fee7b3f2-9725-488d-8792-357afcb18f32-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.913435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.913411 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.913887 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.913439 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.913887 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.913495 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.913887 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.913686 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.914036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.913889 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fee7b3f2-9725-488d-8792-357afcb18f32-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.915356 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.915335 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.915722 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.915703 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fee7b3f2-9725-488d-8792-357afcb18f32-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.931746 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.931716 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fee7b3f2-9725-488d-8792-357afcb18f32-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.932831 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.932810 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czmxm\" (UniqueName: \"kubernetes.io/projected/fee7b3f2-9725-488d-8792-357afcb18f32-kube-api-access-czmxm\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:06.982641 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:06.982611 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:07.270744 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:07.270713 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2"] Oct 30 19:42:07.271353 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:42:07.271326 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfee7b3f2_9725_488d_8792_357afcb18f32.slice/crio-d5e1ac03298c99b65ba91b93c56af922e47290a1e9e347396fa436dd03ec65e1 WatchSource:0}: Error finding container d5e1ac03298c99b65ba91b93c56af922e47290a1e9e347396fa436dd03ec65e1: Status 404 returned error can't find the container with id d5e1ac03298c99b65ba91b93c56af922e47290a1e9e347396fa436dd03ec65e1 Oct 30 19:42:07.273726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:07.273694 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:42:07.273793 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:07.273758 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:42:07.273793 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:07.273784 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:42:07.370421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:07.370376 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" event={"ID":"fee7b3f2-9725-488d-8792-357afcb18f32","Type":"ContainerStarted","Data":"8a2da82dd2e092be74e0a4ee2229ae925dea685b434d5731ece1b163a6a0f59e"} Oct 30 19:42:07.370604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:07.370440 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" event={"ID":"fee7b3f2-9725-488d-8792-357afcb18f32","Type":"ContainerStarted","Data":"d5e1ac03298c99b65ba91b93c56af922e47290a1e9e347396fa436dd03ec65e1"} Oct 30 19:42:07.406585 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:07.406510 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" podStartSLOduration=1.40649453 podStartE2EDuration="1.40649453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:07.405713085 +0000 UTC m=+647.100921063" watchObservedRunningTime="2026-10-30 19:42:07.40649453 +0000 UTC m=+647.101702491" Oct 30 19:42:07.867173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:07.863815 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:07.983761 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:07.983720 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:08.775257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:08.775110 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:08.985219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:08.985188 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:42:08.985629 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:08.985239 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" podUID="fee7b3f2-9725-488d-8792-357afcb18f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:42:09.683398 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:09.678938 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:09.983748 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:09.983362 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:42:09.983748 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:09.983418 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" podUID="fee7b3f2-9725-488d-8792-357afcb18f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:42:10.776181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:10.774736 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:10.923718 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:10.923681 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:10.986599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:10.986400 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:42:10.986599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:10.986482 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" podUID="fee7b3f2-9725-488d-8792-357afcb18f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:42:11.736479 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:11.735592 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:11.983296 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:11.983255 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:42:11.983785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:11.983336 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" podUID="fee7b3f2-9725-488d-8792-357afcb18f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:42:12.670382 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:12.670342 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:12.984662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:12.984570 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Oct 30 19:42:12.985276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:12.985243 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" podUID="fee7b3f2-9725-488d-8792-357afcb18f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Oct 30 19:42:13.766606 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:13.766560 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:14.656296 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:14.656268 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:14.656846 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:14.656654 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:14.658028 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:14.658006 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 19:42:14.773011 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:14.772976 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:15.675118 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:15.675059 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:15.976604 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:42:15.976501 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:42:16.676891 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:16.676854 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:17.797570 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:17.797538 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:18.762497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:18.762442 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:19.783007 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:19.782970 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:20.911913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:20.911875 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:21.977352 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:21.977316 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:22.986944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:22.986900 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:23.962231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:23.958420 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:24.865418 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:24.865386 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:25.891632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:25.891587 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:26.969623 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:26.969584 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:27.976482 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:27.976442 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:29.189230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:29.189197 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:30.434269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:30.434231 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:31.594371 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:31.594271 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:32.784367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:32.784321 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:33.805206 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:33.805167 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:34.877563 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:34.877514 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:35.188874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.188798 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps"] Oct 30 19:42:35.192787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.192767 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" Oct 30 19:42:35.217595 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.217568 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:42:35.217790 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.217697 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:42:35.220271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.220249 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps"] Oct 30 19:42:35.272740 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.272705 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlpz6\" (UniqueName: \"kubernetes.io/projected/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-kube-api-access-hlpz6\") pod \"maas-api-revoke-sub-1e4bc83d-p4qps\" (UID: \"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" Oct 30 19:42:35.272926 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.272764 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e4bc83d-p4qps\" (UID: \"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" Oct 30 19:42:35.272926 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.272878 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e4bc83d-p4qps\" (UID: \"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" Oct 30 19:42:35.373793 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.373752 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlpz6\" (UniqueName: \"kubernetes.io/projected/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-kube-api-access-hlpz6\") pod \"maas-api-revoke-sub-1e4bc83d-p4qps\" (UID: \"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" Oct 30 19:42:35.373996 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.373803 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e4bc83d-p4qps\" (UID: \"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" Oct 30 19:42:35.373996 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.373921 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e4bc83d-p4qps\" (UID: \"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" Oct 30 19:42:35.374492 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.374467 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1e4bc83d-p4qps\" (UID: \"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" Oct 30 19:42:35.376290 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.376272 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1e4bc83d-p4qps\" (UID: \"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" Oct 30 19:42:35.393072 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.393038 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlpz6\" (UniqueName: \"kubernetes.io/projected/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-kube-api-access-hlpz6\") pod \"maas-api-revoke-sub-1e4bc83d-p4qps\" (UID: \"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" Oct 30 19:42:35.507313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.507286 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" Oct 30 19:42:35.793418 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.793387 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps"] Oct 30 19:42:35.794290 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:42:35.794259 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb7bab1_6d8a_40ed_8c1a_2fca2afd4a8c.slice/crio-38f3884e9112e817a780f5f2d55a8ac638a05c3483b7563399d492c70cff2059 WatchSource:0}: Error finding container 38f3884e9112e817a780f5f2d55a8ac638a05c3483b7563399d492c70cff2059: Status 404 returned error can't find the container with id 38f3884e9112e817a780f5f2d55a8ac638a05c3483b7563399d492c70cff2059 Oct 30 19:42:35.973283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:35.973194 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:36.557038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:36.556997 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" event={"ID":"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c","Type":"ContainerStarted","Data":"34e5a6bef5345a8470f164411ff0084cfc0735ce9e6afacf959ce25af3517818"} Oct 30 19:42:36.557283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:36.557063 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" event={"ID":"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c","Type":"ContainerStarted","Data":"38f3884e9112e817a780f5f2d55a8ac638a05c3483b7563399d492c70cff2059"} Oct 30 19:42:36.581896 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:36.581841 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" podStartSLOduration=1.5818266250000002 podStartE2EDuration="1.581826625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:36.579176687 +0000 UTC m=+676.274384640" watchObservedRunningTime="2026-10-30 19:42:36.581826625 +0000 UTC m=+676.277034583" Oct 30 19:42:36.997024 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:36.996918 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:37.570348 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:37.570308 5091 generic.go:356] "Generic (PLEG): container finished" podID="5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c" containerID="34e5a6bef5345a8470f164411ff0084cfc0735ce9e6afacf959ce25af3517818" exitCode=0 Oct 30 19:42:37.570658 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:37.570376 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" event={"ID":"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c","Type":"ContainerDied","Data":"34e5a6bef5345a8470f164411ff0084cfc0735ce9e6afacf959ce25af3517818"} Oct 30 19:42:38.087901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:38.087859 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:38.726837 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:38.726814 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" Oct 30 19:42:38.815916 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:38.815887 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-maas-api-xks-ca\") pod \"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c\" (UID: \"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c\") " Oct 30 19:42:38.816096 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:38.815940 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-kube-api-access-hlpz6\" (UniqueName: \"kubernetes.io/projected/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-kube-api-access-hlpz6\") pod \"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c\" (UID: \"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c\") " Oct 30 19:42:38.816096 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:38.815992 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-maas-api-openshift-ca\") pod \"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c\" (UID: \"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c\") " Oct 30 19:42:38.816554 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:38.816512 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-maas-api-openshift-ca" pod "5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c" (UID: "5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:42:38.818302 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:38.818271 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-maas-api-xks-ca" pod "5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c" (UID: "5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:42:38.818404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:38.818378 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-kube-api-access-hlpz6" pod "5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c" (UID: "5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c"). InnerVolumeSpecName "kube-api-access-hlpz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:42:38.917860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:38.917829 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:38.917860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:38.917857 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlpz6\" (UniqueName: \"kubernetes.io/projected/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-kube-api-access-hlpz6\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:38.917860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:38.917867 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:42:39.200206 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:39.200092 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:39.584591 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:39.584552 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" event={"ID":"5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c","Type":"ContainerDied","Data":"38f3884e9112e817a780f5f2d55a8ac638a05c3483b7563399d492c70cff2059"} Oct 30 19:42:39.584591 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:39.584589 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps" Oct 30 19:42:39.584821 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:39.584614 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38f3884e9112e817a780f5f2d55a8ac638a05c3483b7563399d492c70cff2059" Oct 30 19:42:40.192464 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:40.192427 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:41.078739 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:41.078703 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:41.988988 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:41.988934 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:42.867804 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:42.867763 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:43.822048 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:43.822016 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:44.790121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:44.790091 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:45.983047 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:45.983001 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:45.986947 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:42:45.986892 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:42:46.191408 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.191367 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk"] Oct 30 19:42:46.192095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.192070 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c" containerName="revoke-keys" Oct 30 19:42:46.192095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.192094 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c" containerName="revoke-keys" Oct 30 19:42:46.192332 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.192299 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c" containerName="revoke-keys" Oct 30 19:42:46.210013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.208140 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.211458 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.211434 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk"] Oct 30 19:42:46.399913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.399874 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.400088 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.399986 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.400088 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.400029 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.400205 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.400109 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.400251 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.400234 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.400307 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.400292 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dv85\" (UniqueName: \"kubernetes.io/projected/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-kube-api-access-2dv85\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.400363 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.400350 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.400439 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.400420 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.400474 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.400461 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.501862 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.501827 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.501862 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.501869 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.502403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.501901 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.502403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.501944 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.502403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.501975 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.502403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.502006 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.502403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.502066 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.502403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.502116 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dv85\" (UniqueName: \"kubernetes.io/projected/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-kube-api-access-2dv85\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.502403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.502205 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.502403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.502311 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.502859 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.502542 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.502859 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.502684 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.503132 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.503102 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.503284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.503133 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.504355 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.504334 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.504686 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.504668 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.516767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.516745 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dv85\" (UniqueName: \"kubernetes.io/projected/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-kube-api-access-2dv85\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.516894 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.516878 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.526855 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.526837 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:46.818545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.818505 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk"] Oct 30 19:42:46.819038 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:42:46.819009 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8fca0c1_9b52_4e2d_9e1f_266cf303ffe1.slice/crio-b57626616209c94135724a96552112046d628c34c3c2cc66bd5cd98a853feb72 WatchSource:0}: Error finding container b57626616209c94135724a96552112046d628c34c3c2cc66bd5cd98a853feb72: Status 404 returned error can't find the container with id b57626616209c94135724a96552112046d628c34c3c2cc66bd5cd98a853feb72 Oct 30 19:42:46.821476 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.821441 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:42:46.821582 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.821502 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:42:46.821582 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:46.821529 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:42:47.190132 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:47.190020 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:47.420258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:47.420110 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:47.641033 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:47.641000 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" event={"ID":"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1","Type":"ContainerStarted","Data":"2e6cc51cbe80b1206d91fe7e6c24f43e07ef13f8ee924d3443cf4a5c0470ddf4"} Oct 30 19:42:47.641240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:47.641052 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" event={"ID":"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1","Type":"ContainerStarted","Data":"b57626616209c94135724a96552112046d628c34c3c2cc66bd5cd98a853feb72"} Oct 30 19:42:47.678363 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:47.678313 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" podStartSLOduration=1.6782990290000002 podStartE2EDuration="1.678299029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:42:47.675638296 +0000 UTC m=+687.370846256" watchObservedRunningTime="2026-10-30 19:42:47.678299029 +0000 UTC m=+687.373506986" Oct 30 19:42:47.896862 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:47.896776 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:48.466171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:48.458949 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:48.527597 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:48.527563 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:48.529455 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:48.529424 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:42:48.529584 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:48.529484 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" podUID="d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:42:49.532177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:49.530568 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:42:49.532177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:49.530631 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" podUID="d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:42:49.606170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:49.605461 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:50.528035 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:50.527562 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:42:50.528035 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:50.527626 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" podUID="d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:42:51.182610 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:51.182566 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:51.527966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:51.527932 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:42:51.528212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:51.527984 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" podUID="d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:42:52.484535 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:52.484483 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:52.527767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:52.527729 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:42:52.527950 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:52.527800 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" podUID="d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:42:52.970860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:52.970825 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:53.107559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:53.107526 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:53.529566 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:53.529529 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Oct 30 19:42:53.530159 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:53.530108 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" podUID="d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Oct 30 19:42:53.692171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:53.692102 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:55.108731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:55.108692 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:55.562307 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:55.562256 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:42:55.562643 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:55.562595 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" podUID="d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:42:55.686529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:55.686497 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:55.688052 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:55.688029 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:55.689136 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:55.689115 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 19:42:56.256594 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:56.256545 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:57.264295 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:57.264257 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:58.460825 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:58.460791 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:42:59.377226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:42:59.377186 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:00.460424 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:00.460391 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:01.381311 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:01.381215 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:02.383704 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:02.383669 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:03.291486 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:03.291449 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:04.286374 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:04.286337 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:05.177626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:05.177591 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:05.334373 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:05.334332 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:06.065294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:06.065260 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:07.080121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:07.080086 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:07.962466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:07.962431 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:08.986752 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:08.986716 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:10.316105 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:10.316071 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:10.899028 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:10.898990 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:11.475054 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:11.475022 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp"] Oct 30 19:43:11.923371 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:11.923338 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp"] Oct 30 19:43:11.926739 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:11.926627 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.136335 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.136297 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.136611 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.136586 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.136734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.136720 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.136913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.136897 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.137030 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.137017 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.137174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.137141 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.138571 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.137429 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.138747 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.138726 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqllx\" (UniqueName: \"kubernetes.io/projected/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-kube-api-access-qqllx\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.138878 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.138861 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.188439 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.172678 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:12.256176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.249067 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.278291 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.278260 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.278878 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.278856 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.279108 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.279090 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.279355 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.279336 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.279502 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.279484 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.279667 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.279651 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.287437 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.251100 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.291164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.278793 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.291305 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.281929 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.291399 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.282246 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.312933 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.295280 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.312933 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.295321 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqllx\" (UniqueName: \"kubernetes.io/projected/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-kube-api-access-qqllx\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.312933 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.295947 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.356179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.345704 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.356417 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.345937 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.356548 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.346730 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.447893 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.447820 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqllx\" (UniqueName: \"kubernetes.io/projected/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-kube-api-access-qqllx\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:12.730451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:12.730355 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:13.163441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:13.163239 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp"] Oct 30 19:43:13.171160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:13.171103 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:43:13.171269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:13.171245 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:43:13.171339 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:13.171304 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:43:13.205552 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:13.205522 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" event={"ID":"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57","Type":"ContainerStarted","Data":"713c055635411394ba11188fbd289a2c7cd2474d6294d819317672b780e66d71"} Oct 30 19:43:13.285340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:13.285223 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:14.215665 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:14.215624 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" event={"ID":"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57","Type":"ContainerStarted","Data":"7c5976ec74d6923e3a7861696dfd4231cce027ab2e8e451028747a33d5e2ba63"} Oct 30 19:43:14.243706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:14.243664 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" podStartSLOduration=3.243650809 podStartE2EDuration="3.243650809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:14.243502711 +0000 UTC m=+713.938710669" watchObservedRunningTime="2026-10-30 19:43:14.243650809 +0000 UTC m=+713.938858761" Oct 30 19:43:14.573169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:14.572645 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:14.731378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:14.731345 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:14.733012 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:14.732982 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:43:14.733171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:14.733030 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" podUID="1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:43:15.527320 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:15.527283 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:15.731424 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:15.731387 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:43:15.731670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:15.731451 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" podUID="1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:43:15.994422 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:43:15.994338 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:43:16.524726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:16.524688 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:16.731810 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:16.731775 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:43:16.732229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:16.731825 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" podUID="1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:43:16.926883 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:16.926799 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:17.606620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:17.606578 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:17.731548 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:17.731510 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:43:17.731723 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:17.731565 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" podUID="1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:43:18.581722 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:18.581685 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:18.731844 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:18.731810 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:43:18.732009 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:18.731886 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" podUID="1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:43:19.068371 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:19.068338 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:19.575201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:19.575162 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:19.731599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:19.731560 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Oct 30 19:43:19.731987 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:19.731626 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" podUID="1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Oct 30 19:43:20.590910 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:20.590841 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:20.735768 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:20.735738 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:20.736160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:20.735993 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:20.736813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:20.736795 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:43:21.544778 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:21.544742 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:21.907673 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:21.907597 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2"] Oct 30 19:43:21.913207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:21.913182 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:21.926077 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:21.926043 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2"] Oct 30 19:43:22.065874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.065834 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.066065 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.065908 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.066065 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.066021 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.066218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.066096 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.066218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.066134 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.066315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.066273 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.066357 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.066348 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8stcn\" (UniqueName: \"kubernetes.io/projected/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-kube-api-access-8stcn\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.066414 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.066375 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.066504 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.066487 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.168124 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.168031 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.168124 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.168073 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.168124 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.168099 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.168451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.168201 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8stcn\" (UniqueName: \"kubernetes.io/projected/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-kube-api-access-8stcn\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.168451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.168238 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.168451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.168289 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.168451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.168343 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.168451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.168402 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.168451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.168440 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.168786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.168517 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.168843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.168817 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.169244 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.169217 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.169356 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.169250 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.169499 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.169472 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.171008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.170985 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.171141 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.171122 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.184358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.184335 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.184500 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.184445 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8stcn\" (UniqueName: \"kubernetes.io/projected/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-kube-api-access-8stcn\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.232444 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.232410 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:22.534950 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.534876 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2"] Oct 30 19:43:22.538553 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:43:22.538497 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5b9c8e_97eb_4c35_8032_92bb7acb8d58.slice/crio-d44046cc4e6778d41154f5df4188557411e025b6126b4b7242ca38f51cc07d91 WatchSource:0}: Error finding container d44046cc4e6778d41154f5df4188557411e025b6126b4b7242ca38f51cc07d91: Status 404 returned error can't find the container with id d44046cc4e6778d41154f5df4188557411e025b6126b4b7242ca38f51cc07d91 Oct 30 19:43:22.541986 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.541942 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:43:22.542087 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.542034 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:43:22.542087 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.542079 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:43:22.706764 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:22.706731 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:23.294463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:23.294423 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" event={"ID":"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58","Type":"ContainerStarted","Data":"e85529baea4d463925b09ba162c122fccfa0c9f7fd8b805d2bd60a73c636c15b"} Oct 30 19:43:23.295019 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:23.294991 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" event={"ID":"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58","Type":"ContainerStarted","Data":"d44046cc4e6778d41154f5df4188557411e025b6126b4b7242ca38f51cc07d91"} Oct 30 19:43:23.491693 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:23.491632 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" podStartSLOduration=2.49161186 podStartE2EDuration="2.49161186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:23.354170647 +0000 UTC m=+723.049378594" watchObservedRunningTime="2026-10-30 19:43:23.49161186 +0000 UTC m=+723.186819820" Oct 30 19:43:24.070351 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:24.070313 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:24.237359 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:24.237329 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:24.237770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:24.237747 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:43:24.237930 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:24.237908 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" podUID="5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:43:25.239169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:25.236267 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:43:25.239169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:25.236329 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" podUID="5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:43:26.082169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:26.076421 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:26.163170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:26.162248 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:26.184199 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:26.183227 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:26.234656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:26.234621 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:43:26.234985 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:26.234913 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" podUID="5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:43:26.779186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:26.774242 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hc22z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:43:26.779186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:26.774307 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" podUID="892edc88-bb54-46e1-acd1-701313b9ae0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:27.174240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:27.173125 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:27.233238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:27.233202 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:43:27.233411 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:27.233258 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" podUID="5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:43:28.233528 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:28.233490 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:43:28.233949 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:28.233556 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" podUID="5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:43:28.369371 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:28.369334 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:29.232906 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:29.232871 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:43:29.233103 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:29.232925 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" podUID="5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:43:29.283295 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:29.283254 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:30.232741 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:30.232706 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Oct 30 19:43:30.232941 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:30.232762 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" podUID="5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Oct 30 19:43:30.269201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:30.269067 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:30.425497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:30.425463 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:31.178782 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:31.178743 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:31.237391 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:31.237353 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:31.237734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:31.237711 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:31.238375 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:31.238355 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:43:32.179289 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:32.179247 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:33.068887 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:33.068842 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:33.961108 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:33.961071 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:34.760306 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:34.760271 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:35.646571 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:35.646534 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:36.617106 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:36.617062 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:37.580165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:37.578758 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:38.755236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:38.755197 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:39.575591 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:39.575556 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:39.922856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:39.922775 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:40.470124 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:40.470063 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:41.588560 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:41.588514 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:42.460173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:42.460106 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:43.344623 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:43.344581 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:44.365928 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:44.365864 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:45.268981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:45.268947 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:45.978879 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:43:45.978838 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:43:46.157978 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.157946 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:46.303027 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.302993 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k"] Oct 30 19:43:46.307183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.307161 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.324191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.324136 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k"] Oct 30 19:43:46.375703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.375670 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.375881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.375801 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.375881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.375832 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.375964 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.375923 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.376017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.375986 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.376103 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.376085 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.376177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.376125 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.376227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.376209 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlqw8\" (UniqueName: \"kubernetes.io/projected/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-kube-api-access-wlqw8\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.376266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.376255 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.477876 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.477834 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.477876 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.477885 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.478122 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.477919 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.478122 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.477950 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlqw8\" (UniqueName: \"kubernetes.io/projected/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-kube-api-access-wlqw8\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.478283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.478118 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.478283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.478188 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.478458 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.478432 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.478536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.478435 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.478536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.478510 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.478656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.478530 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.478656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.478648 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.479059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.478990 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.479218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.479174 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.479302 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.479255 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.480554 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.480521 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.480869 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.480850 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.494123 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.494095 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.494247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.494227 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlqw8\" (UniqueName: \"kubernetes.io/projected/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-kube-api-access-wlqw8\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.623838 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.623762 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:46.923170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.921303 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k"] Oct 30 19:43:46.931545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.931498 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:43:46.931667 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.931594 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:43:46.931667 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:46.931636 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:43:47.190937 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:47.190393 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:47.581096 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:47.581052 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" event={"ID":"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f","Type":"ContainerStarted","Data":"43be83664ad5a73529879f057a787f6bc22dc5c75ef77db6f586fe2da0e80576"} Oct 30 19:43:47.581317 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:47.581117 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" event={"ID":"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f","Type":"ContainerStarted","Data":"362cf0027136b7cf48edb668a9a3e31a1d6ce89dc05df031e00ec53852e061cb"} Oct 30 19:43:47.611575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:47.611520 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" podStartSLOduration=1.611508282 podStartE2EDuration="1.611508282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:47.608746388 +0000 UTC m=+747.303954347" watchObservedRunningTime="2026-10-30 19:43:47.611508282 +0000 UTC m=+747.306716313" Oct 30 19:43:47.623911 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:47.623884 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:47.625390 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:47.625367 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:43:47.625492 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:47.625410 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" podUID="46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:43:48.078397 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:48.078357 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:48.227182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:48.224822 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:48.626168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:48.624604 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:43:48.626168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:48.624668 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" podUID="46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:43:49.565185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:49.565097 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:49.637186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:49.628555 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:43:49.637186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:49.628623 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" podUID="46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:43:50.625610 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:50.625573 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:43:50.815160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:50.815105 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" podUID="46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:43:51.168100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:51.168065 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:51.625532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:51.625496 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:43:51.625823 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:51.625785 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" podUID="46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:43:52.624615 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:52.624575 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:43:52.624905 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:52.624872 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" podUID="46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:43:52.659750 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:52.659712 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:53.033740 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:53.033701 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:43:53.033913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:53.033750 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:43:53.618823 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:53.618787 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:53.624803 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:53.624774 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:43:53.624940 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:53.624838 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" podUID="46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:43:53.926407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:53.926325 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:54.606380 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:54.605645 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:54.624861 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:54.624812 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Oct 30 19:43:54.625206 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:54.625125 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" podUID="46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Oct 30 19:43:55.449728 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:55.449683 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:56.308476 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.308445 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:56.308859 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.308831 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:56.309974 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.309952 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:43:56.323287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.323249 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:56.583184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.583094 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8"] Oct 30 19:43:56.588385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.588360 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.600780 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.600748 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8"] Oct 30 19:43:56.635298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.635266 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.635576 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.635550 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.635705 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.635586 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.635705 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.635603 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrwsb\" (UniqueName: \"kubernetes.io/projected/69602ea5-4de9-4b3a-8e3b-a3045d919c82-kube-api-access-rrwsb\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.635830 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.635727 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.635830 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.635752 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.635830 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.635779 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.635992 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.635938 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.635992 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.635983 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.736720 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.736690 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.736904 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.736728 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.736904 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.736745 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrwsb\" (UniqueName: \"kubernetes.io/projected/69602ea5-4de9-4b3a-8e3b-a3045d919c82-kube-api-access-rrwsb\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.736904 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.736788 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.736904 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.736817 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.736904 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.736849 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.737195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.737053 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.737195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.737111 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.737315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.737172 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.737315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.737185 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.737315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.737262 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.737459 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.737308 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.737459 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.737445 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.737756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.737736 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.739133 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.739107 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.739433 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.739417 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.756789 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.756767 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrwsb\" (UniqueName: \"kubernetes.io/projected/69602ea5-4de9-4b3a-8e3b-a3045d919c82-kube-api-access-rrwsb\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.757017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.756997 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:56.911872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:56.911795 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:57.211259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:57.211232 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8"] Oct 30 19:43:57.212084 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:43:57.212060 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69602ea5_4de9_4b3a_8e3b_a3045d919c82.slice/crio-7982f89f9f061becdfe2270c6c0b4316c1154afe56f5cd3ad3d69d0ef7460fc4 WatchSource:0}: Error finding container 7982f89f9f061becdfe2270c6c0b4316c1154afe56f5cd3ad3d69d0ef7460fc4: Status 404 returned error can't find the container with id 7982f89f9f061becdfe2270c6c0b4316c1154afe56f5cd3ad3d69d0ef7460fc4 Oct 30 19:43:57.214347 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:57.214318 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:43:57.214419 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:57.214374 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:43:57.214419 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:57.214405 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:43:57.288175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:57.285847 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:57.465207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:57.465063 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:57.621244 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:57.621207 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:58.215927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:58.215886 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" event={"ID":"69602ea5-4de9-4b3a-8e3b-a3045d919c82","Type":"ContainerStarted","Data":"01de8521f0a1db0c2ecdaa886a44908c271bf920b0eb7f9baf4f4b762bd27d3f"} Oct 30 19:43:58.216140 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:58.215950 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" event={"ID":"69602ea5-4de9-4b3a-8e3b-a3045d919c82","Type":"ContainerStarted","Data":"7982f89f9f061becdfe2270c6c0b4316c1154afe56f5cd3ad3d69d0ef7460fc4"} Oct 30 19:43:58.275169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:58.271805 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" podStartSLOduration=2.271784494 podStartE2EDuration="2.271784494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:43:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:43:58.24446591 +0000 UTC m=+757.939673891" watchObservedRunningTime="2026-10-30 19:43:58.271784494 +0000 UTC m=+757.966992453" Oct 30 19:43:58.588810 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:58.588385 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:58.913023 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:58.912942 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:43:58.914759 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:58.914734 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:43:58.914885 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:58.914781 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" podUID="69602ea5-4de9-4b3a-8e3b-a3045d919c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:43:59.592511 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:59.592468 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:43:59.913257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:59.913177 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:43:59.913257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:43:59.913241 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" podUID="69602ea5-4de9-4b3a-8e3b-a3045d919c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:44:00.813911 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:00.813872 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:00.912831 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:00.912744 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:44:00.913108 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:00.913058 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" podUID="69602ea5-4de9-4b3a-8e3b-a3045d919c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:44:01.912105 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:01.912067 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:01.913259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:01.913235 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:44:01.913465 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:01.913437 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" podUID="69602ea5-4de9-4b3a-8e3b-a3045d919c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:44:02.297275 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:02.297238 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:44:02.297556 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:02.297526 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" podUID="5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:02.889434 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:02.889396 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:02.913028 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:02.912995 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:44:02.913488 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:02.913046 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" podUID="69602ea5-4de9-4b3a-8e3b-a3045d919c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:44:03.049170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:03.045381 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:03.912572 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:03.912524 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Oct 30 19:44:03.912749 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:03.912611 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" podUID="69602ea5-4de9-4b3a-8e3b-a3045d919c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Oct 30 19:44:04.162960 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:04.162863 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:05.227977 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:05.227940 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:05.749013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:05.748978 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:44:05.749677 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:05.749647 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:44:05.750748 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:05.750725 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:44:06.175082 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:06.174990 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:07.173219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.173183 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:07.468959 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.468872 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw"] Oct 30 19:44:07.472998 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.472974 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" Oct 30 19:44:07.484461 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.484427 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:44:07.484702 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.484682 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:44:07.490771 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.490743 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw"] Oct 30 19:44:07.589167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.589114 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d846aa-4c80-4702-ab6e-9a2216de0b5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-514fad51-kf5gw\" (UID: \"41d846aa-4c80-4702-ab6e-9a2216de0b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" Oct 30 19:44:07.589342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.589207 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9d4gw\" (UniqueName: \"kubernetes.io/projected/41d846aa-4c80-4702-ab6e-9a2216de0b5b-kube-api-access-9d4gw\") pod \"maas-api-revoke-sub-514fad51-kf5gw\" (UID: \"41d846aa-4c80-4702-ab6e-9a2216de0b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" Oct 30 19:44:07.589342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.589309 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d846aa-4c80-4702-ab6e-9a2216de0b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-514fad51-kf5gw\" (UID: \"41d846aa-4c80-4702-ab6e-9a2216de0b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" Oct 30 19:44:07.691048 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.691014 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d846aa-4c80-4702-ab6e-9a2216de0b5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-514fad51-kf5gw\" (UID: \"41d846aa-4c80-4702-ab6e-9a2216de0b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" Oct 30 19:44:07.691282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.691070 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9d4gw\" (UniqueName: \"kubernetes.io/projected/41d846aa-4c80-4702-ab6e-9a2216de0b5b-kube-api-access-9d4gw\") pod \"maas-api-revoke-sub-514fad51-kf5gw\" (UID: \"41d846aa-4c80-4702-ab6e-9a2216de0b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" Oct 30 19:44:07.691282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.691193 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d846aa-4c80-4702-ab6e-9a2216de0b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-514fad51-kf5gw\" (UID: \"41d846aa-4c80-4702-ab6e-9a2216de0b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" Oct 30 19:44:07.691862 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.691839 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d846aa-4c80-4702-ab6e-9a2216de0b5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-514fad51-kf5gw\" (UID: \"41d846aa-4c80-4702-ab6e-9a2216de0b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" Oct 30 19:44:07.693418 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.693391 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d846aa-4c80-4702-ab6e-9a2216de0b5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-514fad51-kf5gw\" (UID: \"41d846aa-4c80-4702-ab6e-9a2216de0b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" Oct 30 19:44:07.706118 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.706092 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9d4gw\" (UniqueName: \"kubernetes.io/projected/41d846aa-4c80-4702-ab6e-9a2216de0b5b-kube-api-access-9d4gw\") pod \"maas-api-revoke-sub-514fad51-kf5gw\" (UID: \"41d846aa-4c80-4702-ab6e-9a2216de0b5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" Oct 30 19:44:07.792671 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.792634 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" Oct 30 19:44:07.996197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:07.995757 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:08.112221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:08.106340 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw"] Oct 30 19:44:08.124511 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:44:08.124469 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41d846aa_4c80_4702_ab6e_9a2216de0b5b.slice/crio-6f1e56d167cf16bccd71f5da79934d59abee0e32ad34a1a4fa575b6a002c759e WatchSource:0}: Error finding container 6f1e56d167cf16bccd71f5da79934d59abee0e32ad34a1a4fa575b6a002c759e: Status 404 returned error can't find the container with id 6f1e56d167cf16bccd71f5da79934d59abee0e32ad34a1a4fa575b6a002c759e Oct 30 19:44:08.220990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:08.220955 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:08.358738 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:08.358706 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" event={"ID":"41d846aa-4c80-4702-ab6e-9a2216de0b5b","Type":"ContainerStarted","Data":"6f1e56d167cf16bccd71f5da79934d59abee0e32ad34a1a4fa575b6a002c759e"} Oct 30 19:44:09.277171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:09.271535 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:09.366770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:09.366732 5091 generic.go:356] "Generic (PLEG): container finished" podID="41d846aa-4c80-4702-ab6e-9a2216de0b5b" containerID="cf3a6e2d4fb22549006e7e9f954659e271270eba0d65ad3f15b76982eb1c9126" exitCode=0 Oct 30 19:44:09.366938 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:09.366816 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" event={"ID":"41d846aa-4c80-4702-ab6e-9a2216de0b5b","Type":"ContainerDied","Data":"cf3a6e2d4fb22549006e7e9f954659e271270eba0d65ad3f15b76982eb1c9126"} Oct 30 19:44:10.494018 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:10.493791 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:10.541848 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:10.541821 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" Oct 30 19:44:10.639377 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:10.639291 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41d846aa-4c80-4702-ab6e-9a2216de0b5b-kube-api-access-9d4gw\" (UniqueName: \"kubernetes.io/projected/41d846aa-4c80-4702-ab6e-9a2216de0b5b-kube-api-access-9d4gw\") pod \"41d846aa-4c80-4702-ab6e-9a2216de0b5b\" (UID: \"41d846aa-4c80-4702-ab6e-9a2216de0b5b\") " Oct 30 19:44:10.639543 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:10.639389 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41d846aa-4c80-4702-ab6e-9a2216de0b5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d846aa-4c80-4702-ab6e-9a2216de0b5b-maas-api-xks-ca\") pod \"41d846aa-4c80-4702-ab6e-9a2216de0b5b\" (UID: \"41d846aa-4c80-4702-ab6e-9a2216de0b5b\") " Oct 30 19:44:10.639543 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:10.639451 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41d846aa-4c80-4702-ab6e-9a2216de0b5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d846aa-4c80-4702-ab6e-9a2216de0b5b-maas-api-openshift-ca\") pod \"41d846aa-4c80-4702-ab6e-9a2216de0b5b\" (UID: \"41d846aa-4c80-4702-ab6e-9a2216de0b5b\") " Oct 30 19:44:10.639899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:10.639870 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41d846aa-4c80-4702-ab6e-9a2216de0b5b-maas-api-openshift-ca" pod "41d846aa-4c80-4702-ab6e-9a2216de0b5b" (UID: "41d846aa-4c80-4702-ab6e-9a2216de0b5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:10.641506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:10.641478 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41d846aa-4c80-4702-ab6e-9a2216de0b5b-kube-api-access-9d4gw" pod "41d846aa-4c80-4702-ab6e-9a2216de0b5b" (UID: "41d846aa-4c80-4702-ab6e-9a2216de0b5b"). InnerVolumeSpecName "kube-api-access-9d4gw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:10.641846 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:10.641830 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41d846aa-4c80-4702-ab6e-9a2216de0b5b-maas-api-xks-ca" pod "41d846aa-4c80-4702-ab6e-9a2216de0b5b" (UID: "41d846aa-4c80-4702-ab6e-9a2216de0b5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:10.741047 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:10.741007 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9d4gw\" (UniqueName: \"kubernetes.io/projected/41d846aa-4c80-4702-ab6e-9a2216de0b5b-kube-api-access-9d4gw\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:10.741047 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:10.741041 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41d846aa-4c80-4702-ab6e-9a2216de0b5b-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:10.741047 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:10.741055 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41d846aa-4c80-4702-ab6e-9a2216de0b5b-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:10.957811 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:10.957718 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:11.368353 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:11.368317 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:11.381558 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:11.381530 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" event={"ID":"41d846aa-4c80-4702-ab6e-9a2216de0b5b","Type":"ContainerDied","Data":"6f1e56d167cf16bccd71f5da79934d59abee0e32ad34a1a4fa575b6a002c759e"} Oct 30 19:44:11.381703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:11.381566 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw" Oct 30 19:44:11.381703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:11.381584 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f1e56d167cf16bccd71f5da79934d59abee0e32ad34a1a4fa575b6a002c759e" Oct 30 19:44:12.467167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:12.466178 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:13.453209 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:13.453175 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:14.374179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:14.374102 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:15.669241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:15.669197 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:16.721179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.699611 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt"] Oct 30 19:44:16.721179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.700467 5091 cpu_manager.go:414] "containerMap: removing container" podUID="41d846aa-4c80-4702-ab6e-9a2216de0b5b" containerName="revoke-keys" Oct 30 19:44:16.721179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.700488 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41d846aa-4c80-4702-ab6e-9a2216de0b5b" containerName="revoke-keys" Oct 30 19:44:16.721179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.700715 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="41d846aa-4c80-4702-ab6e-9a2216de0b5b" containerName="revoke-keys" Oct 30 19:44:16.721179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.716006 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.752376 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:44:16.752340 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:44:16.834173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.826614 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt"] Oct 30 19:44:16.868681 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.868648 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wghc8\" (UniqueName: \"kubernetes.io/projected/d037606e-0d57-4527-b5a4-fa485eb976f3-kube-api-access-wghc8\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.868953 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.868926 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.869165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.869127 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d037606e-0d57-4527-b5a4-fa485eb976f3-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.869312 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.869293 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.869451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.869434 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.869592 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.869575 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.877703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.869731 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.877901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.877878 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.878077 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.878055 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.979230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.979068 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.979230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.979168 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.979230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.979224 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wghc8\" (UniqueName: \"kubernetes.io/projected/d037606e-0d57-4527-b5a4-fa485eb976f3-kube-api-access-wghc8\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.979557 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.979270 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.979557 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.979347 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d037606e-0d57-4527-b5a4-fa485eb976f3-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.979557 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.979372 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.979557 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.979414 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.979557 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.979457 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.979557 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.979506 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.979898 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.979585 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.979898 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.979871 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:16.980298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.980272 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d037606e-0d57-4527-b5a4-fa485eb976f3-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:17.023191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.980713 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:17.023191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:16.980949 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:17.027567 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:17.026206 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:17.027567 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:17.026608 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:17.033243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:17.028871 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:17.038820 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:17.038795 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wghc8\" (UniqueName: \"kubernetes.io/projected/d037606e-0d57-4527-b5a4-fa485eb976f3-kube-api-access-wghc8\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:17.173111 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:17.173077 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:17.395170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:17.392315 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:18.200228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:18.180277 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 19:44:18.200228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:18.194495 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:18.448179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:18.434019 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt"] Oct 30 19:44:18.499645 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:18.499603 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:44:18.499875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:18.499851 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:44:18.500018 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:18.499999 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:44:18.960687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:18.945590 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 19:44:19.186169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:19.182500 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:19.214170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:19.213231 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:19.258171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:19.243954 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:19.390921 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:19.390887 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:44:19.391517 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:19.391485 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" event={"ID":"d037606e-0d57-4527-b5a4-fa485eb976f3","Type":"ContainerStarted","Data":"56b7a0fddd44b2c48871eee8debf67225a6deaf60c2b3277869d9d89eb7aa16d"} Oct 30 19:44:19.392514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:19.392494 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:44:19.617222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:19.614652 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:44:19.729208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:19.708260 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:44:20.294243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:20.279870 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" event={"ID":"d037606e-0d57-4527-b5a4-fa485eb976f3","Type":"ContainerStarted","Data":"1db9a07cdd1181dea78fdc6eacbf6f9ac462c88e41a90fe3fe72f3b5faa8ef62"} Oct 30 19:44:20.581915 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:20.581806 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" podStartSLOduration=4.5817868520000005 podStartE2EDuration="4.581786852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:20.537212326 +0000 UTC m=+780.232420286" watchObservedRunningTime="2026-10-30 19:44:20.581786852 +0000 UTC m=+780.276994792" Oct 30 19:44:21.198875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:21.198798 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:21.200497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:21.200468 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:44:21.200708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:21.200677 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" podUID="d037606e-0d57-4527-b5a4-fa485eb976f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:44:21.578938 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:21.577350 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:22.118893 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.112206 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5"] Oct 30 19:44:22.136498 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.136445 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5"] Oct 30 19:44:22.136698 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.136624 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.180651 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.180611 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:44:22.180942 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.180913 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" podUID="d037606e-0d57-4527-b5a4-fa485eb976f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:44:22.233226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.233188 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.233377 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.233304 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.233377 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.233338 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.233483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.233426 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.233483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.233453 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.233559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.233490 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.233559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.233530 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.233642 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.233622 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.233686 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.233652 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzrn2\" (UniqueName: \"kubernetes.io/projected/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-kube-api-access-dzrn2\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.334535 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.334495 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.334703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.334575 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.334703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.334602 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.334703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.334645 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.334703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.334676 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.334949 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.334727 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.335079 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.335049 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.335286 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.335073 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.335682 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.335184 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.335759 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.335338 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.335759 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.335730 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzrn2\" (UniqueName: \"kubernetes.io/projected/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-kube-api-access-dzrn2\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.335866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.335375 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.335866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.335450 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.335866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.335556 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.337079 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.337059 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.337439 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.337421 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.373971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.373887 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzrn2\" (UniqueName: \"kubernetes.io/projected/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-kube-api-access-dzrn2\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.378257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.378222 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.483346 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.483314 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:22.776759 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.776733 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5"] Oct 30 19:44:22.779226 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:44:22.779191 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef8573b_3f96_4e5d_b9ff_7b0e3924b0a6.slice/crio-c7057fd72f869a7c6c6fb4ef29c48af5303a06e442288ecc58bac4708ea21e1c WatchSource:0}: Error finding container c7057fd72f869a7c6c6fb4ef29c48af5303a06e442288ecc58bac4708ea21e1c: Status 404 returned error can't find the container with id c7057fd72f869a7c6c6fb4ef29c48af5303a06e442288ecc58bac4708ea21e1c Oct 30 19:44:22.781795 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.781766 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:44:22.781870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.781832 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:44:22.781870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.781865 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:44:22.860194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:22.854261 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:23.177354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:23.176754 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:44:23.177354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:23.176827 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" podUID="d037606e-0d57-4527-b5a4-fa485eb976f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:44:23.302939 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:23.302903 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" event={"ID":"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6","Type":"ContainerStarted","Data":"d32608cb95951f9df5895e80a0aee54dfb7c209619e81dae4c80c27701d78af1"} Oct 30 19:44:23.303131 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:23.302959 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" event={"ID":"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6","Type":"ContainerStarted","Data":"c7057fd72f869a7c6c6fb4ef29c48af5303a06e442288ecc58bac4708ea21e1c"} Oct 30 19:44:23.359979 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:23.359912 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" podStartSLOduration=1.359891629 podStartE2EDuration="1.359891629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:23.35460605 +0000 UTC m=+783.049814031" watchObservedRunningTime="2026-10-30 19:44:23.359891629 +0000 UTC m=+783.055099588" Oct 30 19:44:23.503293 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:23.497901 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:23.503293 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:23.498023 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:44:23.503293 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:23.498064 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" podUID="8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:44:24.096042 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.095972 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:24.195039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.194596 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:44:24.195039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.194653 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" podUID="d037606e-0d57-4527-b5a4-fa485eb976f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:44:24.210168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.199058 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld"] Oct 30 19:44:24.227525 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.227494 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" Oct 30 19:44:24.228599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.227389 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld"] Oct 30 19:44:24.235421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.235370 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:44:24.236213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.236114 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:44:24.323089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.323052 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd8196c7-j48ld\" (UID: \"72865e5d-d9f3-49f8-a3a5-beedd320d8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" Oct 30 19:44:24.323286 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.323197 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd8196c7-j48ld\" (UID: \"72865e5d-d9f3-49f8-a3a5-beedd320d8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" Oct 30 19:44:24.323373 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.323350 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xrkb\" (UniqueName: \"kubernetes.io/projected/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-kube-api-access-8xrkb\") pod \"maas-api-revoke-sub-cd8196c7-j48ld\" (UID: \"72865e5d-d9f3-49f8-a3a5-beedd320d8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" Oct 30 19:44:24.424358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.424279 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd8196c7-j48ld\" (UID: \"72865e5d-d9f3-49f8-a3a5-beedd320d8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" Oct 30 19:44:24.424531 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.424363 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xrkb\" (UniqueName: \"kubernetes.io/projected/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-kube-api-access-8xrkb\") pod \"maas-api-revoke-sub-cd8196c7-j48ld\" (UID: \"72865e5d-d9f3-49f8-a3a5-beedd320d8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" Oct 30 19:44:24.424531 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.424510 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd8196c7-j48ld\" (UID: \"72865e5d-d9f3-49f8-a3a5-beedd320d8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" Oct 30 19:44:24.425111 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.425084 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd8196c7-j48ld\" (UID: \"72865e5d-d9f3-49f8-a3a5-beedd320d8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" Oct 30 19:44:24.426914 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.426886 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd8196c7-j48ld\" (UID: \"72865e5d-d9f3-49f8-a3a5-beedd320d8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" Oct 30 19:44:24.439357 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.439335 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xrkb\" (UniqueName: \"kubernetes.io/projected/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-kube-api-access-8xrkb\") pod \"maas-api-revoke-sub-cd8196c7-j48ld\" (UID: \"72865e5d-d9f3-49f8-a3a5-beedd320d8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" Oct 30 19:44:24.485951 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.485518 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:44:24.485951 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.485573 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" podUID="8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:44:24.565211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.549316 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" Oct 30 19:44:24.918755 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:24.918724 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld"] Oct 30 19:44:24.919827 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:44:24.919792 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72865e5d_d9f3_49f8_a3a5_beedd320d8e9.slice/crio-ea9bebf5c9fbdc54d22fe4581089674a652e49f88d077a15ee7bf2275bd0f1d8 WatchSource:0}: Error finding container ea9bebf5c9fbdc54d22fe4581089674a652e49f88d077a15ee7bf2275bd0f1d8: Status 404 returned error can't find the container with id ea9bebf5c9fbdc54d22fe4581089674a652e49f88d077a15ee7bf2275bd0f1d8 Oct 30 19:44:25.182382 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:25.180002 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:44:25.182382 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:25.180057 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" podUID="d037606e-0d57-4527-b5a4-fa485eb976f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:44:25.315379 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:25.315212 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:25.332188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:25.331966 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" event={"ID":"72865e5d-d9f3-49f8-a3a5-beedd320d8e9","Type":"ContainerStarted","Data":"ea9bebf5c9fbdc54d22fe4581089674a652e49f88d077a15ee7bf2275bd0f1d8"} Oct 30 19:44:25.491661 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:25.491571 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:44:25.491927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:25.491896 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" podUID="8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:44:26.179169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:26.178537 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:44:26.179169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:26.178597 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" podUID="d037606e-0d57-4527-b5a4-fa485eb976f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:44:26.410219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:26.409594 5091 generic.go:356] "Generic (PLEG): container finished" podID="72865e5d-d9f3-49f8-a3a5-beedd320d8e9" containerID="6fbc5aaf4fc82f744801fdf135b42ba839a6c5a707d051196e4793b47a99502f" exitCode=0 Oct 30 19:44:26.410219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:26.409683 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" event={"ID":"72865e5d-d9f3-49f8-a3a5-beedd320d8e9","Type":"ContainerDied","Data":"6fbc5aaf4fc82f744801fdf135b42ba839a6c5a707d051196e4793b47a99502f"} Oct 30 19:44:26.518242 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:26.491870 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:44:26.518242 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:26.491930 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" podUID="8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:44:26.518242 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:26.493567 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:27.175169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:27.174250 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body= Oct 30 19:44:27.175169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:27.174307 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" podUID="d037606e-0d57-4527-b5a4-fa485eb976f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" Oct 30 19:44:27.484399 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:27.484330 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:44:27.484828 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:27.484804 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" podUID="8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:44:27.713340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:27.713300 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:27.721628 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:27.721598 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" Oct 30 19:44:27.833974 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:27.833940 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-maas-api-xks-ca\") pod \"72865e5d-d9f3-49f8-a3a5-beedd320d8e9\" (UID: \"72865e5d-d9f3-49f8-a3a5-beedd320d8e9\") " Oct 30 19:44:27.834195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:27.834166 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-maas-api-openshift-ca\") pod \"72865e5d-d9f3-49f8-a3a5-beedd320d8e9\" (UID: \"72865e5d-d9f3-49f8-a3a5-beedd320d8e9\") " Oct 30 19:44:27.834259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:27.834199 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-kube-api-access-8xrkb\" (UniqueName: \"kubernetes.io/projected/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-kube-api-access-8xrkb\") pod \"72865e5d-d9f3-49f8-a3a5-beedd320d8e9\" (UID: \"72865e5d-d9f3-49f8-a3a5-beedd320d8e9\") " Oct 30 19:44:27.835269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:27.835237 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-maas-api-openshift-ca" pod "72865e5d-d9f3-49f8-a3a5-beedd320d8e9" (UID: "72865e5d-d9f3-49f8-a3a5-beedd320d8e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:27.836931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:27.836909 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-maas-api-xks-ca" pod "72865e5d-d9f3-49f8-a3a5-beedd320d8e9" (UID: "72865e5d-d9f3-49f8-a3a5-beedd320d8e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:27.839795 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:27.839773 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-kube-api-access-8xrkb" pod "72865e5d-d9f3-49f8-a3a5-beedd320d8e9" (UID: "72865e5d-d9f3-49f8-a3a5-beedd320d8e9"). InnerVolumeSpecName "kube-api-access-8xrkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:27.935776 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:27.935747 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:27.935776 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:27.935772 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xrkb\" (UniqueName: \"kubernetes.io/projected/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-kube-api-access-8xrkb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:27.936045 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:27.935787 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72865e5d-d9f3-49f8-a3a5-beedd320d8e9-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:28.427682 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:28.427627 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" event={"ID":"72865e5d-d9f3-49f8-a3a5-beedd320d8e9","Type":"ContainerDied","Data":"ea9bebf5c9fbdc54d22fe4581089674a652e49f88d077a15ee7bf2275bd0f1d8"} Oct 30 19:44:28.427682 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:28.427678 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea9bebf5c9fbdc54d22fe4581089674a652e49f88d077a15ee7bf2275bd0f1d8" Oct 30 19:44:28.427888 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:28.427734 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld" Oct 30 19:44:28.500169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:28.500007 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:44:28.500169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:28.500068 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" podUID="8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:44:28.596377 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:28.596346 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:28.598083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:28.598061 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:28.599888 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:28.599860 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:44:28.825808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:28.825769 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:29.483946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:29.483912 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body= Oct 30 19:44:29.484162 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:29.483977 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" podUID="8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" Oct 30 19:44:29.690194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:29.680970 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:30.489621 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:30.489576 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:30.491337 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:30.491314 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:30.492169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:30.492122 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 19:44:30.584339 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:30.584301 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:31.480189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:31.473182 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:32.397755 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:32.397697 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:33.675407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:33.675360 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:34.514248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.514212 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5"] Oct 30 19:44:34.514763 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.514750 5091 cpu_manager.go:414] "containerMap: removing container" podUID="72865e5d-d9f3-49f8-a3a5-beedd320d8e9" containerName="revoke-keys" Oct 30 19:44:34.514809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.514764 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72865e5d-d9f3-49f8-a3a5-beedd320d8e9" containerName="revoke-keys" Oct 30 19:44:34.514860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.514851 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="72865e5d-d9f3-49f8-a3a5-beedd320d8e9" containerName="revoke-keys" Oct 30 19:44:34.518082 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.518064 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" Oct 30 19:44:34.525407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.525377 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:44:34.525546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.525431 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:44:34.527389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.527367 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5"] Oct 30 19:44:34.649439 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.649408 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffgfp\" (UniqueName: \"kubernetes.io/projected/6a048461-6326-4544-a77f-830a1378c7bb-kube-api-access-ffgfp\") pod \"maas-api-revoke-sub-99fa903d-4mhr5\" (UID: \"6a048461-6326-4544-a77f-830a1378c7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" Oct 30 19:44:34.649606 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.649461 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a048461-6326-4544-a77f-830a1378c7bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99fa903d-4mhr5\" (UID: \"6a048461-6326-4544-a77f-830a1378c7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" Oct 30 19:44:34.649657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.649641 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a048461-6326-4544-a77f-830a1378c7bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99fa903d-4mhr5\" (UID: \"6a048461-6326-4544-a77f-830a1378c7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" Oct 30 19:44:34.695883 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.695842 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:34.751352 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.751312 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a048461-6326-4544-a77f-830a1378c7bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99fa903d-4mhr5\" (UID: \"6a048461-6326-4544-a77f-830a1378c7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" Oct 30 19:44:34.751539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.751394 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffgfp\" (UniqueName: \"kubernetes.io/projected/6a048461-6326-4544-a77f-830a1378c7bb-kube-api-access-ffgfp\") pod \"maas-api-revoke-sub-99fa903d-4mhr5\" (UID: \"6a048461-6326-4544-a77f-830a1378c7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" Oct 30 19:44:34.751539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.751431 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a048461-6326-4544-a77f-830a1378c7bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99fa903d-4mhr5\" (UID: \"6a048461-6326-4544-a77f-830a1378c7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" Oct 30 19:44:34.752055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.752028 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a048461-6326-4544-a77f-830a1378c7bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99fa903d-4mhr5\" (UID: \"6a048461-6326-4544-a77f-830a1378c7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" Oct 30 19:44:34.753812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.753790 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a048461-6326-4544-a77f-830a1378c7bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99fa903d-4mhr5\" (UID: \"6a048461-6326-4544-a77f-830a1378c7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" Oct 30 19:44:34.766417 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.766369 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffgfp\" (UniqueName: \"kubernetes.io/projected/6a048461-6326-4544-a77f-830a1378c7bb-kube-api-access-ffgfp\") pod \"maas-api-revoke-sub-99fa903d-4mhr5\" (UID: \"6a048461-6326-4544-a77f-830a1378c7bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" Oct 30 19:44:34.839895 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:34.839864 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" Oct 30 19:44:35.123859 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:35.123822 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5"] Oct 30 19:44:35.124724 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:44:35.124683 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a048461_6326_4544_a77f_830a1378c7bb.slice/crio-30bac1db48d03eb409798372d996e5cff964a7fe29d407d0cd051620b41cabfb WatchSource:0}: Error finding container 30bac1db48d03eb409798372d996e5cff964a7fe29d407d0cd051620b41cabfb: Status 404 returned error can't find the container with id 30bac1db48d03eb409798372d996e5cff964a7fe29d407d0cd051620b41cabfb Oct 30 19:44:35.553566 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:35.553531 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" event={"ID":"6a048461-6326-4544-a77f-830a1378c7bb","Type":"ContainerStarted","Data":"6189afd4db11ba12f5c4e3643433c23ec09188306a6f72ee975a9e605f653586"} Oct 30 19:44:35.553752 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:35.553592 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" event={"ID":"6a048461-6326-4544-a77f-830a1378c7bb","Type":"ContainerStarted","Data":"30bac1db48d03eb409798372d996e5cff964a7fe29d407d0cd051620b41cabfb"} Oct 30 19:44:35.595728 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:35.595691 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:35.602631 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:35.597863 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" podStartSLOduration=1.597841937 podStartE2EDuration="1.597841937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:35.59640274 +0000 UTC m=+795.291610699" watchObservedRunningTime="2026-10-30 19:44:35.597841937 +0000 UTC m=+795.293049908" Oct 30 19:44:36.473058 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.472956 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:36.563648 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.563615 5091 generic.go:356] "Generic (PLEG): container finished" podID="6a048461-6326-4544-a77f-830a1378c7bb" containerID="6189afd4db11ba12f5c4e3643433c23ec09188306a6f72ee975a9e605f653586" exitCode=0 Oct 30 19:44:36.563835 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.563716 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" event={"ID":"6a048461-6326-4544-a77f-830a1378c7bb","Type":"ContainerDied","Data":"6189afd4db11ba12f5c4e3643433c23ec09188306a6f72ee975a9e605f653586"} Oct 30 19:44:36.582897 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.582865 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm"] Oct 30 19:44:36.587962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.587937 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.601059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.600903 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm"] Oct 30 19:44:36.601404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.601334 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8836a8\"/\"default-dockercfg-vp47g\"" Oct 30 19:44:36.601514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.601449 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-8836a8\"/\"distinct-2-w0-8836a8-kserve-self-signed-certs\"" Oct 30 19:44:36.601579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.601524 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8836a8\"/\"openshift-service-ca.crt\"" Oct 30 19:44:36.601638 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.601621 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-8836a8\"/\"kube-root-ca.crt\"" Oct 30 19:44:36.679650 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.679621 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-home\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.679829 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.679691 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5j68n\" (UniqueName: \"kubernetes.io/projected/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-kube-api-access-5j68n\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.679829 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.679775 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-tls-certs\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.679947 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.679913 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-model-cache\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.680003 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.679959 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-tmp-dir\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.680078 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.680055 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-dshm\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.680330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.680298 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:36.786188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.784230 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-home\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.786188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.784324 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5j68n\" (UniqueName: \"kubernetes.io/projected/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-kube-api-access-5j68n\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.786188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.784363 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-tls-certs\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.786188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.784471 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-model-cache\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.786188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.784517 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-tmp-dir\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.786188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.784563 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-dshm\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.791493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.790549 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-dshm\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.791493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.790836 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-home\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.805168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.797516 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-model-cache\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.805168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.797772 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-tmp-dir\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.807881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.807849 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-tls-certs\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.817763 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.817740 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5j68n\" (UniqueName: \"kubernetes.io/projected/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-kube-api-access-5j68n\") pod \"distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:36.910463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:36.910434 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:37.094000 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.093917 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx"] Oct 30 19:44:37.100323 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.100298 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.108534 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.108444 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx"] Oct 30 19:44:37.191700 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.191563 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.191880 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.191703 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9lhv\" (UniqueName: \"kubernetes.io/projected/036310d3-eb3a-410b-8716-886f139c818c-kube-api-access-j9lhv\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.191880 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.191756 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.191880 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.191860 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.192044 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.191891 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036310d3-eb3a-410b-8716-886f139c818c-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.192044 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.191972 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.192044 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.191994 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036310d3-eb3a-410b-8716-886f139c818c-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.192244 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.192061 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.192244 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.192107 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036310d3-eb3a-410b-8716-886f139c818c-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.202141 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.202110 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm"] Oct 30 19:44:37.293110 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.293080 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.293278 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.293132 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9lhv\" (UniqueName: \"kubernetes.io/projected/036310d3-eb3a-410b-8716-886f139c818c-kube-api-access-j9lhv\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.293278 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.293175 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.293395 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.293316 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.293395 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.293359 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036310d3-eb3a-410b-8716-886f139c818c-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.293500 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.293441 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.293500 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.293479 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036310d3-eb3a-410b-8716-886f139c818c-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.293603 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.293510 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.293603 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.293553 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036310d3-eb3a-410b-8716-886f139c818c-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.293603 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.293558 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.293762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.293614 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.293849 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.293827 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.293908 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.293885 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.294530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.294507 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036310d3-eb3a-410b-8716-886f139c818c-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.295862 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.295839 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.296204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.296183 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036310d3-eb3a-410b-8716-886f139c818c-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.307233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.307200 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036310d3-eb3a-410b-8716-886f139c818c-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.307352 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.307201 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9lhv\" (UniqueName: \"kubernetes.io/projected/036310d3-eb3a-410b-8716-886f139c818c-kube-api-access-j9lhv\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.421517 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.421435 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:37.482032 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.481937 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:37.578243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.578206 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" event={"ID":"706afd71-a9e1-4b85-a243-b6a9f5e1edb1","Type":"ContainerStarted","Data":"259ab7b2c5a1b53a55ba0a41576190f3a2fd3bb6a3b68ab2dc1043edc55cd48f"} Oct 30 19:44:37.578424 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.578259 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" event={"ID":"706afd71-a9e1-4b85-a243-b6a9f5e1edb1","Type":"ContainerStarted","Data":"9ba84a1ad031948560b7020dcfdddaf2ddcaee7439279f33596f0cb6c2a00b4c"} Oct 30 19:44:37.611659 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.611593 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podStartSLOduration=1.611575437 podStartE2EDuration="1.611575437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:37.606319061 +0000 UTC m=+797.301527044" watchObservedRunningTime="2026-10-30 19:44:37.611575437 +0000 UTC m=+797.306783396" Oct 30 19:44:37.727803 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.727780 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" Oct 30 19:44:37.772719 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.749391 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx"] Oct 30 19:44:37.776231 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:44:37.773315 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod036310d3_eb3a_410b_8716_886f139c818c.slice/crio-59f4e765536af398d1de81079b516a203883bd58acdbd912982ae76fe5c1f15c WatchSource:0}: Error finding container 59f4e765536af398d1de81079b516a203883bd58acdbd912982ae76fe5c1f15c: Status 404 returned error can't find the container with id 59f4e765536af398d1de81079b516a203883bd58acdbd912982ae76fe5c1f15c Oct 30 19:44:37.787508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.787463 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:44:37.787752 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.787730 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:44:37.787882 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.787866 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:44:37.911077 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.911053 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:37.921683 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.921659 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a048461-6326-4544-a77f-830a1378c7bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a048461-6326-4544-a77f-830a1378c7bb-maas-api-openshift-ca\") pod \"6a048461-6326-4544-a77f-830a1378c7bb\" (UID: \"6a048461-6326-4544-a77f-830a1378c7bb\") " Oct 30 19:44:37.921809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.921792 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a048461-6326-4544-a77f-830a1378c7bb-kube-api-access-ffgfp\" (UniqueName: \"kubernetes.io/projected/6a048461-6326-4544-a77f-830a1378c7bb-kube-api-access-ffgfp\") pod \"6a048461-6326-4544-a77f-830a1378c7bb\" (UID: \"6a048461-6326-4544-a77f-830a1378c7bb\") " Oct 30 19:44:37.921924 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.921909 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a048461-6326-4544-a77f-830a1378c7bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a048461-6326-4544-a77f-830a1378c7bb-maas-api-xks-ca\") pod \"6a048461-6326-4544-a77f-830a1378c7bb\" (UID: \"6a048461-6326-4544-a77f-830a1378c7bb\") " Oct 30 19:44:37.922095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.922068 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a048461-6326-4544-a77f-830a1378c7bb-maas-api-openshift-ca" pod "6a048461-6326-4544-a77f-830a1378c7bb" (UID: "6a048461-6326-4544-a77f-830a1378c7bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:37.922423 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.922403 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a048461-6326-4544-a77f-830a1378c7bb-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:37.923861 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.923837 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a048461-6326-4544-a77f-830a1378c7bb-kube-api-access-ffgfp" pod "6a048461-6326-4544-a77f-830a1378c7bb" (UID: "6a048461-6326-4544-a77f-830a1378c7bb"). InnerVolumeSpecName "kube-api-access-ffgfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:37.923937 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:37.923891 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a048461-6326-4544-a77f-830a1378c7bb-maas-api-xks-ca" pod "6a048461-6326-4544-a77f-830a1378c7bb" (UID: "6a048461-6326-4544-a77f-830a1378c7bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:38.023508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:38.023476 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a048461-6326-4544-a77f-830a1378c7bb-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:38.023508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:38.023506 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffgfp\" (UniqueName: \"kubernetes.io/projected/6a048461-6326-4544-a77f-830a1378c7bb-kube-api-access-ffgfp\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:38.519126 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:38.518883 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:38.594475 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:38.594438 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" event={"ID":"6a048461-6326-4544-a77f-830a1378c7bb","Type":"ContainerDied","Data":"30bac1db48d03eb409798372d996e5cff964a7fe29d407d0cd051620b41cabfb"} Oct 30 19:44:38.594677 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:38.594491 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30bac1db48d03eb409798372d996e5cff964a7fe29d407d0cd051620b41cabfb" Oct 30 19:44:38.594677 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:38.594542 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5" Oct 30 19:44:38.597332 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:38.597186 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" event={"ID":"036310d3-eb3a-410b-8716-886f139c818c","Type":"ContainerStarted","Data":"89ebf0935a92e280530073a5021ebccb63e811aa67c619791d4dec6a7897b67e"} Oct 30 19:44:38.597332 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:38.597237 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" event={"ID":"036310d3-eb3a-410b-8716-886f139c818c","Type":"ContainerStarted","Data":"59f4e765536af398d1de81079b516a203883bd58acdbd912982ae76fe5c1f15c"} Oct 30 19:44:38.650955 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:38.645465 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" podStartSLOduration=1.645446967 podStartE2EDuration="1.645446967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:38.641555351 +0000 UTC m=+798.336763325" watchObservedRunningTime="2026-10-30 19:44:38.645446967 +0000 UTC m=+798.340654963" Oct 30 19:44:39.077170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.076649 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9"] Oct 30 19:44:39.078176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.077574 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6a048461-6326-4544-a77f-830a1378c7bb" containerName="revoke-keys" Oct 30 19:44:39.078176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.077598 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a048461-6326-4544-a77f-830a1378c7bb" containerName="revoke-keys" Oct 30 19:44:39.078176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.077773 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a048461-6326-4544-a77f-830a1378c7bb" containerName="revoke-keys" Oct 30 19:44:39.081831 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.081809 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" Oct 30 19:44:39.086718 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.086689 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9"] Oct 30 19:44:39.087346 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.087323 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:44:39.087449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.087422 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:44:39.140576 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.140545 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c069c460-436f-46d1-a508-6e094e66d9b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f79cb4a7-njft9\" (UID: \"c069c460-436f-46d1-a508-6e094e66d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" Oct 30 19:44:39.140701 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.140616 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c069c460-436f-46d1-a508-6e094e66d9b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f79cb4a7-njft9\" (UID: \"c069c460-436f-46d1-a508-6e094e66d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" Oct 30 19:44:39.140850 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.140705 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xghk\" (UniqueName: \"kubernetes.io/projected/c069c460-436f-46d1-a508-6e094e66d9b2-kube-api-access-4xghk\") pod \"maas-api-revoke-sub-f79cb4a7-njft9\" (UID: \"c069c460-436f-46d1-a508-6e094e66d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" Oct 30 19:44:39.241885 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.241849 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c069c460-436f-46d1-a508-6e094e66d9b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f79cb4a7-njft9\" (UID: \"c069c460-436f-46d1-a508-6e094e66d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" Oct 30 19:44:39.242069 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.241982 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c069c460-436f-46d1-a508-6e094e66d9b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f79cb4a7-njft9\" (UID: \"c069c460-436f-46d1-a508-6e094e66d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" Oct 30 19:44:39.242069 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.242006 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xghk\" (UniqueName: \"kubernetes.io/projected/c069c460-436f-46d1-a508-6e094e66d9b2-kube-api-access-4xghk\") pod \"maas-api-revoke-sub-f79cb4a7-njft9\" (UID: \"c069c460-436f-46d1-a508-6e094e66d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" Oct 30 19:44:39.242588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.242565 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c069c460-436f-46d1-a508-6e094e66d9b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f79cb4a7-njft9\" (UID: \"c069c460-436f-46d1-a508-6e094e66d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" Oct 30 19:44:39.244377 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.244357 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c069c460-436f-46d1-a508-6e094e66d9b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f79cb4a7-njft9\" (UID: \"c069c460-436f-46d1-a508-6e094e66d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" Oct 30 19:44:39.282804 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.282767 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xghk\" (UniqueName: \"kubernetes.io/projected/c069c460-436f-46d1-a508-6e094e66d9b2-kube-api-access-4xghk\") pod \"maas-api-revoke-sub-f79cb4a7-njft9\" (UID: \"c069c460-436f-46d1-a508-6e094e66d9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" Oct 30 19:44:39.408249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.408171 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" Oct 30 19:44:39.422545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.422517 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:39.423993 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.423966 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:44:39.424104 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.424020 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" podUID="036310d3-eb3a-410b-8716-886f139c818c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 19:44:39.569169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.526995 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:39.734182 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:44:39.734123 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc069c460_436f_46d1_a508_6e094e66d9b2.slice/crio-91251fe323fc4f301b5f0e29c950951b34c45ae451cd438612e9619ab85793cd WatchSource:0}: Error finding container 91251fe323fc4f301b5f0e29c950951b34c45ae451cd438612e9619ab85793cd: Status 404 returned error can't find the container with id 91251fe323fc4f301b5f0e29c950951b34c45ae451cd438612e9619ab85793cd Oct 30 19:44:39.734564 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:39.734540 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9"] Oct 30 19:44:40.422734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:40.422699 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:44:40.422913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:40.422758 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" podUID="036310d3-eb3a-410b-8716-886f139c818c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 19:44:40.631220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:40.631178 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:40.692947 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:40.692910 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" event={"ID":"c069c460-436f-46d1-a508-6e094e66d9b2","Type":"ContainerStarted","Data":"9ae0dbeba10bf7ceadae63779255d22566515bbab5f7bfc600712ad1a28a6f25"} Oct 30 19:44:40.693193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:40.692974 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" event={"ID":"c069c460-436f-46d1-a508-6e094e66d9b2","Type":"ContainerStarted","Data":"91251fe323fc4f301b5f0e29c950951b34c45ae451cd438612e9619ab85793cd"} Oct 30 19:44:40.717415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:40.717354 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" podStartSLOduration=1.717335819 podStartE2EDuration="1.717335819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:40.714703693 +0000 UTC m=+800.409911676" watchObservedRunningTime="2026-10-30 19:44:40.717335819 +0000 UTC m=+800.412543778" Oct 30 19:44:41.422197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.422159 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:44:41.422498 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.422463 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" podUID="036310d3-eb3a-410b-8716-886f139c818c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 19:44:41.788227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.732503 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz"] Oct 30 19:44:41.788227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.745389 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.788227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.753020 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz"] Oct 30 19:44:41.912170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.882957 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.912170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.883051 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.912170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.883169 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.912170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.883198 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.912170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.883233 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d77vb\" (UniqueName: \"kubernetes.io/projected/a95d6ded-89c1-4aee-b4e2-da19030afca8-kube-api-access-d77vb\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.912170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.883306 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.912170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.883336 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95d6ded-89c1-4aee-b4e2-da19030afca8-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.912170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.883371 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.912170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.883421 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.997259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.915727 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:41.997259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.996940 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.997259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.996989 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95d6ded-89c1-4aee-b4e2-da19030afca8-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.997259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.997023 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.997259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.997071 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.997259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.997193 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.997956 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.997273 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.997956 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.997377 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.997956 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.997404 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:41.997956 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.997439 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d77vb\" (UniqueName: \"kubernetes.io/projected/a95d6ded-89c1-4aee-b4e2-da19030afca8-kube-api-access-d77vb\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:42.049448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:41.999593 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:42.049448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:42.000296 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95d6ded-89c1-4aee-b4e2-da19030afca8-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:42.049448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:42.000570 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:42.049448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:42.000813 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:42.049448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:42.001053 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:42.056276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:42.050072 5091 generic.go:356] "Generic (PLEG): container finished" podID="c069c460-436f-46d1-a508-6e094e66d9b2" containerID="9ae0dbeba10bf7ceadae63779255d22566515bbab5f7bfc600712ad1a28a6f25" exitCode=0 Oct 30 19:44:42.056276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:42.050134 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" event={"ID":"c069c460-436f-46d1-a508-6e094e66d9b2","Type":"ContainerDied","Data":"9ae0dbeba10bf7ceadae63779255d22566515bbab5f7bfc600712ad1a28a6f25"} Oct 30 19:44:42.151164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:42.151114 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:42.251172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:42.249784 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d77vb\" (UniqueName: \"kubernetes.io/projected/a95d6ded-89c1-4aee-b4e2-da19030afca8-kube-api-access-d77vb\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:42.251172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:42.250021 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:42.299314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:42.296670 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:42.393221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:42.376440 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:42.440167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:42.428563 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:44:42.440167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:42.428620 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" podUID="036310d3-eb3a-410b-8716-886f139c818c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 19:44:42.968448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:42.966835 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:43.241424 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.241391 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz"] Oct 30 19:44:43.286332 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.286285 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:44:43.286462 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.286377 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:44:43.286462 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.286423 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:44:43.421857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.421761 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:44:43.421857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.421808 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" podUID="036310d3-eb3a-410b-8716-886f139c818c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 19:44:43.426754 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.426734 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" Oct 30 19:44:43.524258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.524225 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c069c460-436f-46d1-a508-6e094e66d9b2-kube-api-access-4xghk\" (UniqueName: \"kubernetes.io/projected/c069c460-436f-46d1-a508-6e094e66d9b2-kube-api-access-4xghk\") pod \"c069c460-436f-46d1-a508-6e094e66d9b2\" (UID: \"c069c460-436f-46d1-a508-6e094e66d9b2\") " Oct 30 19:44:43.524446 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.524283 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c069c460-436f-46d1-a508-6e094e66d9b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c069c460-436f-46d1-a508-6e094e66d9b2-maas-api-xks-ca\") pod \"c069c460-436f-46d1-a508-6e094e66d9b2\" (UID: \"c069c460-436f-46d1-a508-6e094e66d9b2\") " Oct 30 19:44:43.524446 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.524348 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c069c460-436f-46d1-a508-6e094e66d9b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c069c460-436f-46d1-a508-6e094e66d9b2-maas-api-openshift-ca\") pod \"c069c460-436f-46d1-a508-6e094e66d9b2\" (UID: \"c069c460-436f-46d1-a508-6e094e66d9b2\") " Oct 30 19:44:43.525025 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.525002 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c069c460-436f-46d1-a508-6e094e66d9b2-maas-api-openshift-ca" pod "c069c460-436f-46d1-a508-6e094e66d9b2" (UID: "c069c460-436f-46d1-a508-6e094e66d9b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:43.527452 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.527408 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c069c460-436f-46d1-a508-6e094e66d9b2-maas-api-xks-ca" pod "c069c460-436f-46d1-a508-6e094e66d9b2" (UID: "c069c460-436f-46d1-a508-6e094e66d9b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:43.527734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.527712 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c069c460-436f-46d1-a508-6e094e66d9b2-kube-api-access-4xghk" pod "c069c460-436f-46d1-a508-6e094e66d9b2" (UID: "c069c460-436f-46d1-a508-6e094e66d9b2"). InnerVolumeSpecName "kube-api-access-4xghk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:43.625652 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.625457 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c069c460-436f-46d1-a508-6e094e66d9b2-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:43.625652 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.625484 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xghk\" (UniqueName: \"kubernetes.io/projected/c069c460-436f-46d1-a508-6e094e66d9b2-kube-api-access-4xghk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:43.625652 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.625495 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c069c460-436f-46d1-a508-6e094e66d9b2-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:43.923166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:43.918127 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:44.120746 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:44.120707 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" Oct 30 19:44:44.121160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:44.120705 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9" event={"ID":"c069c460-436f-46d1-a508-6e094e66d9b2","Type":"ContainerDied","Data":"91251fe323fc4f301b5f0e29c950951b34c45ae451cd438612e9619ab85793cd"} Oct 30 19:44:44.121160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:44.120839 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91251fe323fc4f301b5f0e29c950951b34c45ae451cd438612e9619ab85793cd" Oct 30 19:44:44.122997 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:44.122973 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" event={"ID":"a95d6ded-89c1-4aee-b4e2-da19030afca8","Type":"ContainerStarted","Data":"22cd906037041447865cbba1675dbbf1cc763480e22e5f811a55961209058013"} Oct 30 19:44:44.123093 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:44.123014 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" event={"ID":"a95d6ded-89c1-4aee-b4e2-da19030afca8","Type":"ContainerStarted","Data":"c5d3aeb52b242df38980ce1368a5c91959b2c9b9ab3fd523fd9991174837dec6"} Oct 30 19:44:44.180430 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:44.180309 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" podStartSLOduration=3.180288782 podStartE2EDuration="3.180288782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:44:44.165315406 +0000 UTC m=+803.860523365" watchObservedRunningTime="2026-10-30 19:44:44.180288782 +0000 UTC m=+803.875496741" Oct 30 19:44:44.378467 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:44.378337 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:44.382171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:44.380265 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:44:44.382171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:44.380337 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" podUID="a95d6ded-89c1-4aee-b4e2-da19030afca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:44:44.422288 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:44.422253 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" start-of-body= Oct 30 19:44:44.422416 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:44.422312 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" podUID="036310d3-eb3a-410b-8716-886f139c818c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": dial tcp 10.132.0.92:15021: connect: connection refused" Oct 30 19:44:45.021285 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:45.021249 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:45.382269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:45.378600 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:44:45.382269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:45.378662 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" podUID="a95d6ded-89c1-4aee-b4e2-da19030afca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:44:45.903575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:45.903537 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:46.003253 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:44:46.003216 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:44:46.360691 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:46.360651 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:46.378588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:46.378551 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:44:46.378861 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:46.378831 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" podUID="a95d6ded-89c1-4aee-b4e2-da19030afca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:44:46.469171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:46.468261 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:44:46.469171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:46.468323 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" podUID="036310d3-eb3a-410b-8716-886f139c818c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:46.676917 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:46.676772 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:46.915172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:46.913697 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:46.929689 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:46.929618 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:47.204219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.203340 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:47.238431 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.237489 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:47.238431 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.238345 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:47.239419 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.239399 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:44:47.256490 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.256432 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:47.291692 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.291660 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47"] Oct 30 19:44:47.292344 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.292327 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c069c460-436f-46d1-a508-6e094e66d9b2" containerName="revoke-keys" Oct 30 19:44:47.292423 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.292344 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c069c460-436f-46d1-a508-6e094e66d9b2" containerName="revoke-keys" Oct 30 19:44:47.292522 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.292510 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c069c460-436f-46d1-a508-6e094e66d9b2" containerName="revoke-keys" Oct 30 19:44:47.297167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.297135 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" Oct 30 19:44:47.307494 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.307472 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:44:47.308387 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.308343 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:44:47.312376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.312355 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47"] Oct 30 19:44:47.384025 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.383988 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:44:47.384258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.384042 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" podUID="a95d6ded-89c1-4aee-b4e2-da19030afca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:44:47.420136 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.420103 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a27159b9-95c47\" (UID: \"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" Oct 30 19:44:47.420302 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.420194 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a27159b9-95c47\" (UID: \"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" Oct 30 19:44:47.420643 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.420619 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pbxp\" (UniqueName: \"kubernetes.io/projected/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-kube-api-access-6pbxp\") pod \"maas-api-revoke-sub-a27159b9-95c47\" (UID: \"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" Oct 30 19:44:47.522215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.522180 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a27159b9-95c47\" (UID: \"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" Oct 30 19:44:47.522692 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.522262 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a27159b9-95c47\" (UID: \"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" Oct 30 19:44:47.522692 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.522518 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pbxp\" (UniqueName: \"kubernetes.io/projected/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-kube-api-access-6pbxp\") pod \"maas-api-revoke-sub-a27159b9-95c47\" (UID: \"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" Oct 30 19:44:47.523046 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.523020 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a27159b9-95c47\" (UID: \"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" Oct 30 19:44:47.524783 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.524761 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a27159b9-95c47\" (UID: \"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" Oct 30 19:44:47.571800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.571769 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pbxp\" (UniqueName: \"kubernetes.io/projected/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-kube-api-access-6pbxp\") pod \"maas-api-revoke-sub-a27159b9-95c47\" (UID: \"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" Oct 30 19:44:47.614963 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.614932 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" Oct 30 19:44:47.916139 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:44:47.916110 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4da59d7_41f9_4a9d_bef6_1a5288bb88aa.slice/crio-58c083d544784602b13006b4e0ff363ea7f2eefacac8a85fd883ad83aa35814f WatchSource:0}: Error finding container 58c083d544784602b13006b4e0ff363ea7f2eefacac8a85fd883ad83aa35814f: Status 404 returned error can't find the container with id 58c083d544784602b13006b4e0ff363ea7f2eefacac8a85fd883ad83aa35814f Oct 30 19:44:47.916288 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:47.916196 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47"] Oct 30 19:44:48.192582 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:48.192498 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" event={"ID":"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa","Type":"ContainerStarted","Data":"58c083d544784602b13006b4e0ff363ea7f2eefacac8a85fd883ad83aa35814f"} Oct 30 19:44:48.310187 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:48.303281 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:48.377588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:48.377550 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:44:48.377828 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:48.377796 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" podUID="a95d6ded-89c1-4aee-b4e2-da19030afca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:44:49.182290 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:49.182097 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:49.223036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:49.222988 5091 generic.go:356] "Generic (PLEG): container finished" podID="a4da59d7-41f9-4a9d-bef6-1a5288bb88aa" containerID="8cecc0433cb3e81cac9b2391fc78560cc9fe977771aa86d1371b47b02a9e56df" exitCode=0 Oct 30 19:44:49.223326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:49.223245 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" event={"ID":"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa","Type":"ContainerDied","Data":"8cecc0433cb3e81cac9b2391fc78560cc9fe977771aa86d1371b47b02a9e56df"} Oct 30 19:44:49.377645 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:49.377556 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body= Oct 30 19:44:49.377645 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:49.377632 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" podUID="a95d6ded-89c1-4aee-b4e2-da19030afca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" Oct 30 19:44:50.021736 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:50.021696 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:50.185800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:50.185718 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:50.429888 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:50.429861 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" Oct 30 19:44:50.492929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:50.492888 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-kube-api-access-6pbxp\" (UniqueName: \"kubernetes.io/projected/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-kube-api-access-6pbxp\") pod \"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa\" (UID: \"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa\") " Oct 30 19:44:50.493092 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:50.493061 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-maas-api-xks-ca\") pod \"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa\" (UID: \"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa\") " Oct 30 19:44:50.493992 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:50.493959 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-maas-api-openshift-ca\") pod \"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa\" (UID: \"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa\") " Oct 30 19:44:50.494831 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:50.494794 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-maas-api-openshift-ca" pod "a4da59d7-41f9-4a9d-bef6-1a5288bb88aa" (UID: "a4da59d7-41f9-4a9d-bef6-1a5288bb88aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:50.495111 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:50.495088 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:50.496379 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:50.496357 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-maas-api-xks-ca" pod "a4da59d7-41f9-4a9d-bef6-1a5288bb88aa" (UID: "a4da59d7-41f9-4a9d-bef6-1a5288bb88aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:50.496487 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:50.496402 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-kube-api-access-6pbxp" pod "a4da59d7-41f9-4a9d-bef6-1a5288bb88aa" (UID: "a4da59d7-41f9-4a9d-bef6-1a5288bb88aa"). InnerVolumeSpecName "kube-api-access-6pbxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:50.596267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:50.596141 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:50.596267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:50.596216 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pbxp\" (UniqueName: \"kubernetes.io/projected/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa-kube-api-access-6pbxp\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:50.649227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:50.649196 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:50.649748 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:50.649717 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:50.650779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:50.650757 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 19:44:51.201166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.199842 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:51.240353 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.240322 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" Oct 30 19:44:51.240500 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.240359 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47" event={"ID":"a4da59d7-41f9-4a9d-bef6-1a5288bb88aa","Type":"ContainerDied","Data":"58c083d544784602b13006b4e0ff363ea7f2eefacac8a85fd883ad83aa35814f"} Oct 30 19:44:51.240500 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.240435 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58c083d544784602b13006b4e0ff363ea7f2eefacac8a85fd883ad83aa35814f" Oct 30 19:44:51.312357 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.312330 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc"] Oct 30 19:44:51.312931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.312917 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a4da59d7-41f9-4a9d-bef6-1a5288bb88aa" containerName="revoke-keys" Oct 30 19:44:51.312986 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.312932 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4da59d7-41f9-4a9d-bef6-1a5288bb88aa" containerName="revoke-keys" Oct 30 19:44:51.313039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.313029 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4da59d7-41f9-4a9d-bef6-1a5288bb88aa" containerName="revoke-keys" Oct 30 19:44:51.316627 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.316610 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" Oct 30 19:44:51.323601 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.321603 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:44:51.323601 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.322042 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:44:51.336838 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.326493 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc"] Oct 30 19:44:51.406288 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.406252 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10adce2f-b9d8-47e7-9546-a1b45813dc5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87bd8f73-fp7mc\" (UID: \"10adce2f-b9d8-47e7-9546-a1b45813dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" Oct 30 19:44:51.406426 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.406403 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10adce2f-b9d8-47e7-9546-a1b45813dc5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87bd8f73-fp7mc\" (UID: \"10adce2f-b9d8-47e7-9546-a1b45813dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" Oct 30 19:44:51.406478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.406463 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlz5r\" (UniqueName: \"kubernetes.io/projected/10adce2f-b9d8-47e7-9546-a1b45813dc5b-kube-api-access-jlz5r\") pod \"maas-api-revoke-sub-87bd8f73-fp7mc\" (UID: \"10adce2f-b9d8-47e7-9546-a1b45813dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" Oct 30 19:44:51.508455 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.508422 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10adce2f-b9d8-47e7-9546-a1b45813dc5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87bd8f73-fp7mc\" (UID: \"10adce2f-b9d8-47e7-9546-a1b45813dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" Oct 30 19:44:51.508628 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.508537 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10adce2f-b9d8-47e7-9546-a1b45813dc5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87bd8f73-fp7mc\" (UID: \"10adce2f-b9d8-47e7-9546-a1b45813dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" Oct 30 19:44:51.508628 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.508596 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlz5r\" (UniqueName: \"kubernetes.io/projected/10adce2f-b9d8-47e7-9546-a1b45813dc5b-kube-api-access-jlz5r\") pod \"maas-api-revoke-sub-87bd8f73-fp7mc\" (UID: \"10adce2f-b9d8-47e7-9546-a1b45813dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" Oct 30 19:44:51.509102 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.509072 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10adce2f-b9d8-47e7-9546-a1b45813dc5b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87bd8f73-fp7mc\" (UID: \"10adce2f-b9d8-47e7-9546-a1b45813dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" Oct 30 19:44:51.510859 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.510833 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10adce2f-b9d8-47e7-9546-a1b45813dc5b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87bd8f73-fp7mc\" (UID: \"10adce2f-b9d8-47e7-9546-a1b45813dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" Oct 30 19:44:51.524766 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.524741 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlz5r\" (UniqueName: \"kubernetes.io/projected/10adce2f-b9d8-47e7-9546-a1b45813dc5b-kube-api-access-jlz5r\") pod \"maas-api-revoke-sub-87bd8f73-fp7mc\" (UID: \"10adce2f-b9d8-47e7-9546-a1b45813dc5b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" Oct 30 19:44:51.634385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.634353 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" Oct 30 19:44:51.924753 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:51.924722 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc"] Oct 30 19:44:51.924958 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:44:51.924938 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10adce2f_b9d8_47e7_9546_a1b45813dc5b.slice/crio-ffa7891b89fc090ce7aedf109e3f3a23040444447e0c291e2243c4095ad7abff WatchSource:0}: Error finding container ffa7891b89fc090ce7aedf109e3f3a23040444447e0c291e2243c4095ad7abff: Status 404 returned error can't find the container with id ffa7891b89fc090ce7aedf109e3f3a23040444447e0c291e2243c4095ad7abff Oct 30 19:44:52.251712 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:52.251670 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" event={"ID":"10adce2f-b9d8-47e7-9546-a1b45813dc5b","Type":"ContainerStarted","Data":"ffa7891b89fc090ce7aedf109e3f3a23040444447e0c291e2243c4095ad7abff"} Oct 30 19:44:52.293539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:52.293501 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:53.290237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:53.290204 5091 generic.go:356] "Generic (PLEG): container finished" podID="10adce2f-b9d8-47e7-9546-a1b45813dc5b" containerID="cb25472ae1b8d2349cd50d95945719250f987681d7ece4e04d254c6353f67e63" exitCode=0 Oct 30 19:44:53.290785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:53.290763 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" event={"ID":"10adce2f-b9d8-47e7-9546-a1b45813dc5b","Type":"ContainerDied","Data":"cb25472ae1b8d2349cd50d95945719250f987681d7ece4e04d254c6353f67e63"} Oct 30 19:44:54.324105 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:54.324065 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:54.633225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:54.630318 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:54.642205 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:54.641666 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:54.699292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:54.699241 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:54.893226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:54.886423 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:55.204227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:55.189248 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:55.799234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:55.664973 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:55.799234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:55.665517 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:44:56.064981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:55.999578 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:56.064981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.000793 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:44:56.064981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.041327 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:56.064981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.042358 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:44:56.306172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.297493 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 19:44:56.327221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.320927 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:56.701246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.691768 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:56.701246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.692366 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:44:56.710737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.710709 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" Oct 30 19:44:56.811170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.806576 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" event={"ID":"10adce2f-b9d8-47e7-9546-a1b45813dc5b","Type":"ContainerDied","Data":"ffa7891b89fc090ce7aedf109e3f3a23040444447e0c291e2243c4095ad7abff"} Oct 30 19:44:56.811170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.806643 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffa7891b89fc090ce7aedf109e3f3a23040444447e0c291e2243c4095ad7abff" Oct 30 19:44:56.811170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.806716 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc" Oct 30 19:44:56.854809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.854771 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10adce2f-b9d8-47e7-9546-a1b45813dc5b-kube-api-access-jlz5r\" (UniqueName: \"kubernetes.io/projected/10adce2f-b9d8-47e7-9546-a1b45813dc5b-kube-api-access-jlz5r\") pod \"10adce2f-b9d8-47e7-9546-a1b45813dc5b\" (UID: \"10adce2f-b9d8-47e7-9546-a1b45813dc5b\") " Oct 30 19:44:56.873655 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.873616 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10adce2f-b9d8-47e7-9546-a1b45813dc5b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10adce2f-b9d8-47e7-9546-a1b45813dc5b-maas-api-openshift-ca\") pod \"10adce2f-b9d8-47e7-9546-a1b45813dc5b\" (UID: \"10adce2f-b9d8-47e7-9546-a1b45813dc5b\") " Oct 30 19:44:56.873987 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.873962 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10adce2f-b9d8-47e7-9546-a1b45813dc5b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10adce2f-b9d8-47e7-9546-a1b45813dc5b-maas-api-xks-ca\") pod \"10adce2f-b9d8-47e7-9546-a1b45813dc5b\" (UID: \"10adce2f-b9d8-47e7-9546-a1b45813dc5b\") " Oct 30 19:44:56.904204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.874880 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:44:56.904204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.894573 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10adce2f-b9d8-47e7-9546-a1b45813dc5b-maas-api-openshift-ca" pod "10adce2f-b9d8-47e7-9546-a1b45813dc5b" (UID: "10adce2f-b9d8-47e7-9546-a1b45813dc5b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:44:56.972231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.945913 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:44:56.972231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:56.945970 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" podUID="d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:57.020318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:57.008637 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10adce2f-b9d8-47e7-9546-a1b45813dc5b-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:57.041328 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:57.041289 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10adce2f-b9d8-47e7-9546-a1b45813dc5b-kube-api-access-jlz5r" pod "10adce2f-b9d8-47e7-9546-a1b45813dc5b" (UID: "10adce2f-b9d8-47e7-9546-a1b45813dc5b"). InnerVolumeSpecName "kube-api-access-jlz5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:44:57.041597 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:57.041566 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:57.141616 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:57.141574 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlz5r\" (UniqueName: \"kubernetes.io/projected/10adce2f-b9d8-47e7-9546-a1b45813dc5b-kube-api-access-jlz5r\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:57.325170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:57.323230 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:57.360627 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:57.360587 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:44:57.371275 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:57.371229 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" podUID="46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:57.375174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:57.371533 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:57.412178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:57.411322 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10adce2f-b9d8-47e7-9546-a1b45813dc5b-maas-api-xks-ca" pod "10adce2f-b9d8-47e7-9546-a1b45813dc5b" (UID: "10adce2f-b9d8-47e7-9546-a1b45813dc5b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:44:57.443246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:57.441237 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10adce2f-b9d8-47e7-9546-a1b45813dc5b-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:44:57.715225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:57.702140 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:57.802169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:57.802073 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:57.903536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:57.876331 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 19:44:57.927385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:57.927320 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:44:58.042263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.042224 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 19:44:58.285739 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.285651 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:44:58.480330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.480299 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:44:58.513126 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.513096 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:44:58.541943 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.541865 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh"] Oct 30 19:44:58.543600 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.543574 5091 cpu_manager.go:414] "containerMap: removing container" podUID="10adce2f-b9d8-47e7-9546-a1b45813dc5b" containerName="revoke-keys" Oct 30 19:44:58.552210 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.552184 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10adce2f-b9d8-47e7-9546-a1b45813dc5b" containerName="revoke-keys" Oct 30 19:44:58.552677 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.552657 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="10adce2f-b9d8-47e7-9546-a1b45813dc5b" containerName="revoke-keys" Oct 30 19:44:58.598842 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.598810 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh"] Oct 30 19:44:58.599126 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.598998 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" Oct 30 19:44:58.608233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.608041 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:44:58.608233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.608238 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:44:58.746429 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.746391 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8459a593-be25-4760-900e-bc08897d8294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb22e2c1-77qkh\" (UID: \"8459a593-be25-4760-900e-bc08897d8294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" Oct 30 19:44:58.746744 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.746534 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8459a593-be25-4760-900e-bc08897d8294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb22e2c1-77qkh\" (UID: \"8459a593-be25-4760-900e-bc08897d8294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" Oct 30 19:44:58.746744 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.746611 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q86bc\" (UniqueName: \"kubernetes.io/projected/8459a593-be25-4760-900e-bc08897d8294-kube-api-access-q86bc\") pod \"maas-api-revoke-sub-fb22e2c1-77qkh\" (UID: \"8459a593-be25-4760-900e-bc08897d8294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" Oct 30 19:44:58.852696 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.849168 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:58.855365 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.855095 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8459a593-be25-4760-900e-bc08897d8294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb22e2c1-77qkh\" (UID: \"8459a593-be25-4760-900e-bc08897d8294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" Oct 30 19:44:58.859473 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.855561 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8459a593-be25-4760-900e-bc08897d8294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb22e2c1-77qkh\" (UID: \"8459a593-be25-4760-900e-bc08897d8294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" Oct 30 19:44:58.866525 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.866494 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q86bc\" (UniqueName: \"kubernetes.io/projected/8459a593-be25-4760-900e-bc08897d8294-kube-api-access-q86bc\") pod \"maas-api-revoke-sub-fb22e2c1-77qkh\" (UID: \"8459a593-be25-4760-900e-bc08897d8294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" Oct 30 19:44:58.869216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.856394 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8459a593-be25-4760-900e-bc08897d8294-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb22e2c1-77qkh\" (UID: \"8459a593-be25-4760-900e-bc08897d8294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" Oct 30 19:44:58.900166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.882907 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8459a593-be25-4760-900e-bc08897d8294-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb22e2c1-77qkh\" (UID: \"8459a593-be25-4760-900e-bc08897d8294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" Oct 30 19:44:58.902676 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.902602 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q86bc\" (UniqueName: \"kubernetes.io/projected/8459a593-be25-4760-900e-bc08897d8294-kube-api-access-q86bc\") pod \"maas-api-revoke-sub-fb22e2c1-77qkh\" (UID: \"8459a593-be25-4760-900e-bc08897d8294\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" Oct 30 19:44:58.928036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:58.928001 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" Oct 30 19:44:59.211036 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:44:59.210995 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8459a593_be25_4760_900e_bc08897d8294.slice/crio-3702ffea916779073547a098866aa46266b4847d1cb9ffd8df0e0736983a059b WatchSource:0}: Error finding container 3702ffea916779073547a098866aa46266b4847d1cb9ffd8df0e0736983a059b: Status 404 returned error can't find the container with id 3702ffea916779073547a098866aa46266b4847d1cb9ffd8df0e0736983a059b Oct 30 19:44:59.211331 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:59.211307 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh"] Oct 30 19:44:59.878168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:59.875039 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:44:59.992905 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:59.992861 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" event={"ID":"8459a593-be25-4760-900e-bc08897d8294","Type":"ContainerStarted","Data":"324a95a81392ff1d07dd412e16fcc13064c260c7f03dd24ac4382eecae25c533"} Oct 30 19:44:59.993391 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:44:59.992932 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" event={"ID":"8459a593-be25-4760-900e-bc08897d8294","Type":"ContainerStarted","Data":"3702ffea916779073547a098866aa46266b4847d1cb9ffd8df0e0736983a059b"} Oct 30 19:45:00.016899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.016841 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" podStartSLOduration=2.016827186 podStartE2EDuration="2.016827186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:44:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:00.013823165 +0000 UTC m=+819.709031134" watchObservedRunningTime="2026-10-30 19:45:00.016827186 +0000 UTC m=+819.712035143" Oct 30 19:45:00.052426 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.052364 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:00.282976 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.282946 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk"] Oct 30 19:45:00.288709 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.288682 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72"] Oct 30 19:45:00.300431 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.289271 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:00.300431 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.291400 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" Oct 30 19:45:00.304187 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.304112 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" Oct 30 19:45:00.304471 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.304410 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk"] Oct 30 19:45:00.304568 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.304510 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72"] Oct 30 19:45:00.325408 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.325385 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288"] Oct 30 19:45:00.332306 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.332287 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" Oct 30 19:45:00.359927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.359894 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288"] Oct 30 19:45:00.430267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.430231 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06cdca52-kr288\" (UID: \"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" Oct 30 19:45:00.430435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.430351 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06cdca52-kr288\" (UID: \"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" Oct 30 19:45:00.430435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.430376 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpzxw\" (UniqueName: \"kubernetes.io/projected/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-kube-api-access-kpzxw\") pod \"maas-api-revoke-sub-06cdca52-kr288\" (UID: \"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" Oct 30 19:45:00.430514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.430459 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wggzk\" (UniqueName: \"kubernetes.io/projected/a932ddba-4214-4104-a02f-af5417aa345b-kube-api-access-wggzk\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk\" (UID: \"a932ddba-4214-4104-a02f-af5417aa345b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" Oct 30 19:45:00.430553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.430521 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a716551d-0905-4fab-889a-f0aa0e6019b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72\" (UID: \"a716551d-0905-4fab-889a-f0aa0e6019b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" Oct 30 19:45:00.430634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.430617 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmmqv\" (UniqueName: \"kubernetes.io/projected/a716551d-0905-4fab-889a-f0aa0e6019b3-kube-api-access-qmmqv\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72\" (UID: \"a716551d-0905-4fab-889a-f0aa0e6019b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" Oct 30 19:45:00.430756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.430740 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a932ddba-4214-4104-a02f-af5417aa345b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk\" (UID: \"a932ddba-4214-4104-a02f-af5417aa345b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" Oct 30 19:45:00.544072 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.544023 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a932ddba-4214-4104-a02f-af5417aa345b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk\" (UID: \"a932ddba-4214-4104-a02f-af5417aa345b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" Oct 30 19:45:00.544552 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.544513 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06cdca52-kr288\" (UID: \"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" Oct 30 19:45:00.544752 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.544734 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06cdca52-kr288\" (UID: \"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" Oct 30 19:45:00.544867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.544852 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpzxw\" (UniqueName: \"kubernetes.io/projected/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-kube-api-access-kpzxw\") pod \"maas-api-revoke-sub-06cdca52-kr288\" (UID: \"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" Oct 30 19:45:00.544991 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.544977 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wggzk\" (UniqueName: \"kubernetes.io/projected/a932ddba-4214-4104-a02f-af5417aa345b-kube-api-access-wggzk\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk\" (UID: \"a932ddba-4214-4104-a02f-af5417aa345b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" Oct 30 19:45:00.545135 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.545119 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a716551d-0905-4fab-889a-f0aa0e6019b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72\" (UID: \"a716551d-0905-4fab-889a-f0aa0e6019b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" Oct 30 19:45:00.545338 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.545304 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmmqv\" (UniqueName: \"kubernetes.io/projected/a716551d-0905-4fab-889a-f0aa0e6019b3-kube-api-access-qmmqv\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72\" (UID: \"a716551d-0905-4fab-889a-f0aa0e6019b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" Oct 30 19:45:00.567220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.546870 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a932ddba-4214-4104-a02f-af5417aa345b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk\" (UID: \"a932ddba-4214-4104-a02f-af5417aa345b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" Oct 30 19:45:00.567500 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.550981 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a716551d-0905-4fab-889a-f0aa0e6019b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72\" (UID: \"a716551d-0905-4fab-889a-f0aa0e6019b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" Oct 30 19:45:00.567620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.551834 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06cdca52-kr288\" (UID: \"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" Oct 30 19:45:00.575266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.573017 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06cdca52-kr288\" (UID: \"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" Oct 30 19:45:00.587218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.575792 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmmqv\" (UniqueName: \"kubernetes.io/projected/a716551d-0905-4fab-889a-f0aa0e6019b3-kube-api-access-qmmqv\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72\" (UID: \"a716551d-0905-4fab-889a-f0aa0e6019b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" Oct 30 19:45:00.587218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.585839 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wggzk\" (UniqueName: \"kubernetes.io/projected/a932ddba-4214-4104-a02f-af5417aa345b-kube-api-access-wggzk\") pod \"maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk\" (UID: \"a932ddba-4214-4104-a02f-af5417aa345b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" Oct 30 19:45:00.610176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.606927 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpzxw\" (UniqueName: \"kubernetes.io/projected/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-kube-api-access-kpzxw\") pod \"maas-api-revoke-sub-06cdca52-kr288\" (UID: \"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" Oct 30 19:45:00.635186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.634363 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" Oct 30 19:45:00.639133 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.638914 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" Oct 30 19:45:00.664574 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.664543 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" Oct 30 19:45:00.978588 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:45:00.978541 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda716551d_0905_4fab_889a_f0aa0e6019b3.slice/crio-6b1b3d6ce3bde1c6b4cc9d67b3395723de6be02e3b9d87bd54d1c3007fd1305c WatchSource:0}: Error finding container 6b1b3d6ce3bde1c6b4cc9d67b3395723de6be02e3b9d87bd54d1c3007fd1305c: Status 404 returned error can't find the container with id 6b1b3d6ce3bde1c6b4cc9d67b3395723de6be02e3b9d87bd54d1c3007fd1305c Oct 30 19:45:00.979571 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:00.979401 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72"] Oct 30 19:45:01.005971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:01.005934 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk"] Oct 30 19:45:01.015611 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:45:01.015577 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda932ddba_4214_4104_a02f_af5417aa345b.slice/crio-212822af0c360b9655d3691ba57fc1ae6f3af27e2f65eef6f48c6e3fbf5c0621 WatchSource:0}: Error finding container 212822af0c360b9655d3691ba57fc1ae6f3af27e2f65eef6f48c6e3fbf5c0621: Status 404 returned error can't find the container with id 212822af0c360b9655d3691ba57fc1ae6f3af27e2f65eef6f48c6e3fbf5c0621 Oct 30 19:45:01.016413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:01.016321 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288"] Oct 30 19:45:01.017730 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:01.017700 5091 generic.go:356] "Generic (PLEG): container finished" podID="8459a593-be25-4760-900e-bc08897d8294" containerID="324a95a81392ff1d07dd412e16fcc13064c260c7f03dd24ac4382eecae25c533" exitCode=0 Oct 30 19:45:01.017865 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:01.017834 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" event={"ID":"8459a593-be25-4760-900e-bc08897d8294","Type":"ContainerDied","Data":"324a95a81392ff1d07dd412e16fcc13064c260c7f03dd24ac4382eecae25c533"} Oct 30 19:45:01.018762 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:45:01.018739 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c8ad84f_80cb_4a36_8d65_f0eb2cef68c7.slice/crio-dcd89a0a1b8ee280e8b9795338122fde68142f5c1e54e9195868f2c27a636faf WatchSource:0}: Error finding container dcd89a0a1b8ee280e8b9795338122fde68142f5c1e54e9195868f2c27a636faf: Status 404 returned error can't find the container with id dcd89a0a1b8ee280e8b9795338122fde68142f5c1e54e9195868f2c27a636faf Oct 30 19:45:01.019585 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:01.019562 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" event={"ID":"a716551d-0905-4fab-889a-f0aa0e6019b3","Type":"ContainerStarted","Data":"6b1b3d6ce3bde1c6b4cc9d67b3395723de6be02e3b9d87bd54d1c3007fd1305c"} Oct 30 19:45:01.433876 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:01.433711 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:02.028293 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.028255 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" event={"ID":"a716551d-0905-4fab-889a-f0aa0e6019b3","Type":"ContainerStarted","Data":"feaebe3999a465470065747ca6c9b62c746cfc9fec24465f11e02ab42fa78ccf"} Oct 30 19:45:02.029989 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.029964 5091 generic.go:356] "Generic (PLEG): container finished" podID="a932ddba-4214-4104-a02f-af5417aa345b" containerID="097854721b13ed7b941fff206ce9cb520dbf85d93bb320f216162936e06d2dae" exitCode=60 Oct 30 19:45:02.030127 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.029991 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" event={"ID":"a932ddba-4214-4104-a02f-af5417aa345b","Type":"ContainerDied","Data":"097854721b13ed7b941fff206ce9cb520dbf85d93bb320f216162936e06d2dae"} Oct 30 19:45:02.030127 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.030030 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" event={"ID":"a932ddba-4214-4104-a02f-af5417aa345b","Type":"ContainerStarted","Data":"212822af0c360b9655d3691ba57fc1ae6f3af27e2f65eef6f48c6e3fbf5c0621"} Oct 30 19:45:02.030309 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.030295 5091 scope.go:122] "RemoveContainer" containerID="097854721b13ed7b941fff206ce9cb520dbf85d93bb320f216162936e06d2dae" Oct 30 19:45:02.031883 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.031857 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" event={"ID":"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7","Type":"ContainerStarted","Data":"63e39f8ea3567cd95d67e253c56201ea6cdfe08d89da44e007d5f65f0972e363"} Oct 30 19:45:02.031957 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.031891 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" event={"ID":"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7","Type":"ContainerStarted","Data":"dcd89a0a1b8ee280e8b9795338122fde68142f5c1e54e9195868f2c27a636faf"} Oct 30 19:45:02.055541 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.055495 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" podStartSLOduration=2.055480678 podStartE2EDuration="2.055480678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.050784229 +0000 UTC m=+821.745992188" watchObservedRunningTime="2026-10-30 19:45:02.055480678 +0000 UTC m=+821.750688636" Oct 30 19:45:02.081391 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.081344 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" podStartSLOduration=2.08132881 podStartE2EDuration="2.08132881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:02.079390435 +0000 UTC m=+821.774598399" watchObservedRunningTime="2026-10-30 19:45:02.08132881 +0000 UTC m=+821.776536767" Oct 30 19:45:02.212880 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.212847 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" Oct 30 19:45:02.286634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.286601 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8459a593-be25-4760-900e-bc08897d8294-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8459a593-be25-4760-900e-bc08897d8294-maas-api-xks-ca\") pod \"8459a593-be25-4760-900e-bc08897d8294\" (UID: \"8459a593-be25-4760-900e-bc08897d8294\") " Oct 30 19:45:02.286792 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.286752 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8459a593-be25-4760-900e-bc08897d8294-kube-api-access-q86bc\" (UniqueName: \"kubernetes.io/projected/8459a593-be25-4760-900e-bc08897d8294-kube-api-access-q86bc\") pod \"8459a593-be25-4760-900e-bc08897d8294\" (UID: \"8459a593-be25-4760-900e-bc08897d8294\") " Oct 30 19:45:02.286953 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.286940 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8459a593-be25-4760-900e-bc08897d8294-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8459a593-be25-4760-900e-bc08897d8294-maas-api-openshift-ca\") pod \"8459a593-be25-4760-900e-bc08897d8294\" (UID: \"8459a593-be25-4760-900e-bc08897d8294\") " Oct 30 19:45:02.287508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.287348 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8459a593-be25-4760-900e-bc08897d8294-maas-api-openshift-ca" pod "8459a593-be25-4760-900e-bc08897d8294" (UID: "8459a593-be25-4760-900e-bc08897d8294"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:02.287508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.287487 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8459a593-be25-4760-900e-bc08897d8294-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:02.288729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.288705 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8459a593-be25-4760-900e-bc08897d8294-maas-api-xks-ca" pod "8459a593-be25-4760-900e-bc08897d8294" (UID: "8459a593-be25-4760-900e-bc08897d8294"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:02.288983 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.288963 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8459a593-be25-4760-900e-bc08897d8294-kube-api-access-q86bc" pod "8459a593-be25-4760-900e-bc08897d8294" (UID: "8459a593-be25-4760-900e-bc08897d8294"). InnerVolumeSpecName "kube-api-access-q86bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:02.389092 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.389060 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8459a593-be25-4760-900e-bc08897d8294-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:02.389092 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.389093 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q86bc\" (UniqueName: \"kubernetes.io/projected/8459a593-be25-4760-900e-bc08897d8294-kube-api-access-q86bc\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:02.563251 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:02.563214 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:03.040031 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:03.039986 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" Oct 30 19:45:03.040031 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:03.040015 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh" event={"ID":"8459a593-be25-4760-900e-bc08897d8294","Type":"ContainerDied","Data":"3702ffea916779073547a098866aa46266b4847d1cb9ffd8df0e0736983a059b"} Oct 30 19:45:03.040567 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:03.040066 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3702ffea916779073547a098866aa46266b4847d1cb9ffd8df0e0736983a059b" Oct 30 19:45:03.041819 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:03.041794 5091 generic.go:356] "Generic (PLEG): container finished" podID="a716551d-0905-4fab-889a-f0aa0e6019b3" containerID="feaebe3999a465470065747ca6c9b62c746cfc9fec24465f11e02ab42fa78ccf" exitCode=60 Oct 30 19:45:03.041922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:03.041869 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" event={"ID":"a716551d-0905-4fab-889a-f0aa0e6019b3","Type":"ContainerDied","Data":"feaebe3999a465470065747ca6c9b62c746cfc9fec24465f11e02ab42fa78ccf"} Oct 30 19:45:03.042324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:03.042125 5091 scope.go:122] "RemoveContainer" containerID="feaebe3999a465470065747ca6c9b62c746cfc9fec24465f11e02ab42fa78ccf" Oct 30 19:45:03.044238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:03.044216 5091 generic.go:356] "Generic (PLEG): container finished" podID="a932ddba-4214-4104-a02f-af5417aa345b" containerID="ba401fcceda69bb6b3acdd55e683745c0045d55a3684148a4fa7ecab4d96db4e" exitCode=60 Oct 30 19:45:03.044324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:03.044292 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" event={"ID":"a932ddba-4214-4104-a02f-af5417aa345b","Type":"ContainerDied","Data":"ba401fcceda69bb6b3acdd55e683745c0045d55a3684148a4fa7ecab4d96db4e"} Oct 30 19:45:03.044380 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:03.044355 5091 scope.go:122] "RemoveContainer" containerID="097854721b13ed7b941fff206ce9cb520dbf85d93bb320f216162936e06d2dae" Oct 30 19:45:03.044634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:03.044616 5091 scope.go:122] "RemoveContainer" containerID="ba401fcceda69bb6b3acdd55e683745c0045d55a3684148a4fa7ecab4d96db4e" Oct 30 19:45:03.045256 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:45:03.045234 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk_odh-ai-gateway-infra(a932ddba-4214-4104-a02f-af5417aa345b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" podUID="a932ddba-4214-4104-a02f-af5417aa345b" Oct 30 19:45:03.046889 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:03.046869 5091 generic.go:356] "Generic (PLEG): container finished" podID="8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7" containerID="63e39f8ea3567cd95d67e253c56201ea6cdfe08d89da44e007d5f65f0972e363" exitCode=0 Oct 30 19:45:03.046985 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:03.046900 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" event={"ID":"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7","Type":"ContainerDied","Data":"63e39f8ea3567cd95d67e253c56201ea6cdfe08d89da44e007d5f65f0972e363"} Oct 30 19:45:03.785181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:03.782080 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:03.964188 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:45:03.964119 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda716551d_0905_4fab_889a_f0aa0e6019b3.slice/crio-ea7b90426ae368ccd9511c340e0649e45c54b7b958f43c18fd2b68d43337c676.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:45:04.056790 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.056702 5091 scope.go:122] "RemoveContainer" containerID="ba401fcceda69bb6b3acdd55e683745c0045d55a3684148a4fa7ecab4d96db4e" Oct 30 19:45:04.057287 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:45:04.057075 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk_odh-ai-gateway-infra(a932ddba-4214-4104-a02f-af5417aa345b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" podUID="a932ddba-4214-4104-a02f-af5417aa345b" Oct 30 19:45:04.058740 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.058710 5091 generic.go:356] "Generic (PLEG): container finished" podID="a716551d-0905-4fab-889a-f0aa0e6019b3" containerID="ea7b90426ae368ccd9511c340e0649e45c54b7b958f43c18fd2b68d43337c676" exitCode=60 Oct 30 19:45:04.058882 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.058789 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" event={"ID":"a716551d-0905-4fab-889a-f0aa0e6019b3","Type":"ContainerDied","Data":"ea7b90426ae368ccd9511c340e0649e45c54b7b958f43c18fd2b68d43337c676"} Oct 30 19:45:04.058882 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.058864 5091 scope.go:122] "RemoveContainer" containerID="feaebe3999a465470065747ca6c9b62c746cfc9fec24465f11e02ab42fa78ccf" Oct 30 19:45:04.059605 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.059050 5091 scope.go:122] "RemoveContainer" containerID="ea7b90426ae368ccd9511c340e0649e45c54b7b958f43c18fd2b68d43337c676" Oct 30 19:45:04.059605 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:45:04.059331 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72_odh-ai-gateway-infra(a716551d-0905-4fab-889a-f0aa0e6019b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" podUID="a716551d-0905-4fab-889a-f0aa0e6019b3" Oct 30 19:45:04.231699 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.231663 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" Oct 30 19:45:04.314626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.314538 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-kube-api-access-kpzxw\" (UniqueName: \"kubernetes.io/projected/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-kube-api-access-kpzxw\") pod \"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7\" (UID: \"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7\") " Oct 30 19:45:04.314626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.314589 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-maas-api-xks-ca\") pod \"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7\" (UID: \"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7\") " Oct 30 19:45:04.315523 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.314669 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-maas-api-openshift-ca\") pod \"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7\" (UID: \"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7\") " Oct 30 19:45:04.315523 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.315059 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-maas-api-openshift-ca" pod "8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7" (UID: "8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:04.316878 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.316854 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-maas-api-xks-ca" pod "8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7" (UID: "8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:04.317064 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.317037 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-kube-api-access-kpzxw" pod "8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7" (UID: "8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7"). InnerVolumeSpecName "kube-api-access-kpzxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:04.416386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.416352 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpzxw\" (UniqueName: \"kubernetes.io/projected/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-kube-api-access-kpzxw\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:04.416555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.416391 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:04.416555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.416408 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:04.713225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.710942 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:04.839004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.838963 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq"] Oct 30 19:45:04.839674 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.839653 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7" containerName="revoke-keys" Oct 30 19:45:04.839674 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.839675 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7" containerName="revoke-keys" Oct 30 19:45:04.839857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.839695 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8459a593-be25-4760-900e-bc08897d8294" containerName="revoke-keys" Oct 30 19:45:04.839857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.839702 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8459a593-be25-4760-900e-bc08897d8294" containerName="revoke-keys" Oct 30 19:45:04.839857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.839829 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7" containerName="revoke-keys" Oct 30 19:45:04.839857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.839844 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8459a593-be25-4760-900e-bc08897d8294" containerName="revoke-keys" Oct 30 19:45:04.843326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.843307 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" Oct 30 19:45:04.853705 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.853678 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq"] Oct 30 19:45:04.935480 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.935444 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e77306-zmxbq\" (UID: \"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" Oct 30 19:45:04.935643 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.935591 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e77306-zmxbq\" (UID: \"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" Oct 30 19:45:04.935682 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:04.935653 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnmgb\" (UniqueName: \"kubernetes.io/projected/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-kube-api-access-jnmgb\") pod \"maas-api-revoke-sub-d4e77306-zmxbq\" (UID: \"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" Oct 30 19:45:05.037550 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:05.037505 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnmgb\" (UniqueName: \"kubernetes.io/projected/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-kube-api-access-jnmgb\") pod \"maas-api-revoke-sub-d4e77306-zmxbq\" (UID: \"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" Oct 30 19:45:05.037746 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:05.037667 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e77306-zmxbq\" (UID: \"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" Oct 30 19:45:05.037828 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:05.037813 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e77306-zmxbq\" (UID: \"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" Oct 30 19:45:05.038502 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:05.038475 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4e77306-zmxbq\" (UID: \"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" Oct 30 19:45:05.040179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:05.040128 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4e77306-zmxbq\" (UID: \"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" Oct 30 19:45:05.056951 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:05.056923 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnmgb\" (UniqueName: \"kubernetes.io/projected/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-kube-api-access-jnmgb\") pod \"maas-api-revoke-sub-d4e77306-zmxbq\" (UID: \"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" Oct 30 19:45:05.069685 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:05.069657 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" Oct 30 19:45:05.069875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:05.069682 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288" event={"ID":"8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7","Type":"ContainerDied","Data":"dcd89a0a1b8ee280e8b9795338122fde68142f5c1e54e9195868f2c27a636faf"} Oct 30 19:45:05.069875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:05.069763 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dcd89a0a1b8ee280e8b9795338122fde68142f5c1e54e9195868f2c27a636faf" Oct 30 19:45:05.072375 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:05.072356 5091 scope.go:122] "RemoveContainer" containerID="ea7b90426ae368ccd9511c340e0649e45c54b7b958f43c18fd2b68d43337c676" Oct 30 19:45:05.072645 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:45:05.072624 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72_odh-ai-gateway-infra(a716551d-0905-4fab-889a-f0aa0e6019b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" podUID="a716551d-0905-4fab-889a-f0aa0e6019b3" Oct 30 19:45:05.187567 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:05.187533 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" Oct 30 19:45:05.476529 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:45:05.476489 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6b1d5a_a7c4_4d91_9c16_0330a02d4c86.slice/crio-fa7e6eda8dfcbba1596e2b28ff182fbc64d7f1daac351d7f4f13dffb32735b34 WatchSource:0}: Error finding container fa7e6eda8dfcbba1596e2b28ff182fbc64d7f1daac351d7f4f13dffb32735b34: Status 404 returned error can't find the container with id fa7e6eda8dfcbba1596e2b28ff182fbc64d7f1daac351d7f4f13dffb32735b34 Oct 30 19:45:05.478505 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:05.478478 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq"] Oct 30 19:45:05.675281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:05.675240 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:06.080703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:06.080666 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" event={"ID":"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86","Type":"ContainerStarted","Data":"ffb11d3e7901d13df1b5ffae1fbbec09e9ecf36210545cfba515a37e6fe98362"} Oct 30 19:45:06.081215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:06.080723 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" event={"ID":"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86","Type":"ContainerStarted","Data":"fa7e6eda8dfcbba1596e2b28ff182fbc64d7f1daac351d7f4f13dffb32735b34"} Oct 30 19:45:06.106669 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:06.106563 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" podStartSLOduration=2.106549105 podStartE2EDuration="2.106549105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:06.104434527 +0000 UTC m=+825.799642488" watchObservedRunningTime="2026-10-30 19:45:06.106549105 +0000 UTC m=+825.801757062" Oct 30 19:45:06.580195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:06.577041 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:07.089677 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:07.089639 5091 generic.go:356] "Generic (PLEG): container finished" podID="2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86" containerID="ffb11d3e7901d13df1b5ffae1fbbec09e9ecf36210545cfba515a37e6fe98362" exitCode=0 Oct 30 19:45:07.090042 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:07.089727 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" event={"ID":"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86","Type":"ContainerDied","Data":"ffb11d3e7901d13df1b5ffae1fbbec09e9ecf36210545cfba515a37e6fe98362"} Oct 30 19:45:07.605979 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:07.605932 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:08.255396 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:08.255370 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" Oct 30 19:45:08.387125 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:08.387040 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-kube-api-access-jnmgb\" (UniqueName: \"kubernetes.io/projected/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-kube-api-access-jnmgb\") pod \"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86\" (UID: \"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86\") " Oct 30 19:45:08.387125 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:08.387104 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-maas-api-xks-ca\") pod \"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86\" (UID: \"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86\") " Oct 30 19:45:08.387380 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:08.387221 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-maas-api-openshift-ca\") pod \"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86\" (UID: \"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86\") " Oct 30 19:45:08.387648 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:08.387616 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-maas-api-openshift-ca" pod "2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86" (UID: "2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:08.387754 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:08.387709 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:08.389304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:08.389269 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-maas-api-xks-ca" pod "2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86" (UID: "2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:08.389423 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:08.389349 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-kube-api-access-jnmgb" pod "2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86" (UID: "2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86"). InnerVolumeSpecName "kube-api-access-jnmgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:08.480389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:08.480290 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:08.491488 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:08.491448 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:08.492935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:08.492911 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnmgb\" (UniqueName: \"kubernetes.io/projected/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86-kube-api-access-jnmgb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:08.732360 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:08.732271 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:09.105613 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:09.105576 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" event={"ID":"2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86","Type":"ContainerDied","Data":"fa7e6eda8dfcbba1596e2b28ff182fbc64d7f1daac351d7f4f13dffb32735b34"} Oct 30 19:45:09.105770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:09.105631 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa7e6eda8dfcbba1596e2b28ff182fbc64d7f1daac351d7f4f13dffb32735b34" Oct 30 19:45:09.105770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:09.105586 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq" Oct 30 19:45:09.562737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:09.562701 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:10.731725 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:10.731690 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:10.832411 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:10.830187 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb"] Oct 30 19:45:10.832411 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:10.831070 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86" containerName="revoke-keys" Oct 30 19:45:10.832411 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:10.831086 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86" containerName="revoke-keys" Oct 30 19:45:10.832411 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:10.831280 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86" containerName="revoke-keys" Oct 30 19:45:10.851171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:10.836411 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" Oct 30 19:45:10.851171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:10.849474 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:45:10.852649 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:10.852624 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb"] Oct 30 19:45:10.975262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:10.959869 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98jh7\" (UniqueName: \"kubernetes.io/projected/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-kube-api-access-98jh7\") pod \"maas-api-revoke-sub-521aca49-xpkxb\" (UID: \"df2a21ed-1e2b-415b-b8c2-a04f82d7178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" Oct 30 19:45:10.975262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:10.959989 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521aca49-xpkxb\" (UID: \"df2a21ed-1e2b-415b-b8c2-a04f82d7178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" Oct 30 19:45:10.975262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:10.960091 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521aca49-xpkxb\" (UID: \"df2a21ed-1e2b-415b-b8c2-a04f82d7178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" Oct 30 19:45:11.107930 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:11.107882 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521aca49-xpkxb\" (UID: \"df2a21ed-1e2b-415b-b8c2-a04f82d7178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" Oct 30 19:45:11.117775 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:11.117738 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98jh7\" (UniqueName: \"kubernetes.io/projected/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-kube-api-access-98jh7\") pod \"maas-api-revoke-sub-521aca49-xpkxb\" (UID: \"df2a21ed-1e2b-415b-b8c2-a04f82d7178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" Oct 30 19:45:11.127173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:11.118080 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521aca49-xpkxb\" (UID: \"df2a21ed-1e2b-415b-b8c2-a04f82d7178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" Oct 30 19:45:11.281384 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:11.119987 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-521aca49-xpkxb\" (UID: \"df2a21ed-1e2b-415b-b8c2-a04f82d7178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" Oct 30 19:45:11.281384 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:11.258613 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:11.394765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:11.394682 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-521aca49-xpkxb\" (UID: \"df2a21ed-1e2b-415b-b8c2-a04f82d7178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" Oct 30 19:45:11.413901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:11.413860 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:11.450294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:11.450264 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98jh7\" (UniqueName: \"kubernetes.io/projected/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-kube-api-access-98jh7\") pod \"maas-api-revoke-sub-521aca49-xpkxb\" (UID: \"df2a21ed-1e2b-415b-b8c2-a04f82d7178e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" Oct 30 19:45:11.498704 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:11.498430 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" Oct 30 19:45:11.573441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:11.573363 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:12.296281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:12.296230 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:12.770278 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:12.746572 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:45:12.863440 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:12.863389 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:12.936171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:12.928169 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:13.010120 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:12.997837 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb"] Oct 30 19:45:13.341749 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:13.341508 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 19:45:13.397171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:13.353504 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:45:13.794412 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:13.783582 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:45:13.902375 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:13.821616 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:45:13.902375 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:13.783385 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:13.982723 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:13.982653 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:14.049189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:13.985259 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:45:14.080193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:14.077686 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" event={"ID":"df2a21ed-1e2b-415b-b8c2-a04f82d7178e","Type":"ContainerStarted","Data":"a665418ed03b598951b1e011e5a4b6f87e4b7e90975b2ebc1740e8d4add4ee0d"} Oct 30 19:45:14.130180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:14.129655 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:14.335171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:14.334279 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:14.374585 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:14.364475 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 19:45:14.394648 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:14.391808 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:14.565195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:14.565103 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:14.665589 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:14.665550 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:14.665886 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:14.665853 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:14.676262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:14.676231 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:14.676490 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:14.676464 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" podUID="d037606e-0d57-4527-b5a4-fa485eb976f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:14.787462 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:14.787422 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:45:14.859226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:14.858247 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 19:45:14.859226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:14.859179 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:45:15.015283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:15.015231 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:15.305646 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:15.305604 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" event={"ID":"df2a21ed-1e2b-415b-b8c2-a04f82d7178e","Type":"ContainerStarted","Data":"0f352e1c7af0bf83bf1f7d828ff1da393847d4de4d7edf17c5cdcd6b6748e96e"} Oct 30 19:45:15.447240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:15.433095 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:15.631170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:15.622913 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf"] Oct 30 19:45:15.756441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:15.751209 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:15.756441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:15.751281 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" podUID="fee7b3f2-9725-488d-8792-357afcb18f32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:15.797499 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:15.796409 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:45:15.797879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:15.797849 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf"] Oct 30 19:45:15.798121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:15.798104 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" Oct 30 19:45:15.881254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:15.864411 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:16.054421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:15.997926 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6g76\" (UniqueName: \"kubernetes.io/projected/7218f8cd-7572-479a-a136-a31fbcfab76c-kube-api-access-b6g76\") pod \"maas-api-revoke-sub-c359af9c-rk6sf\" (UID: \"7218f8cd-7572-479a-a136-a31fbcfab76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" Oct 30 19:45:16.054421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:15.998097 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7218f8cd-7572-479a-a136-a31fbcfab76c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c359af9c-rk6sf\" (UID: \"7218f8cd-7572-479a-a136-a31fbcfab76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" Oct 30 19:45:16.054421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:15.998215 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7218f8cd-7572-479a-a136-a31fbcfab76c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c359af9c-rk6sf\" (UID: \"7218f8cd-7572-479a-a136-a31fbcfab76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" Oct 30 19:45:16.054421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:16.026067 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:16.117172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:16.101219 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7218f8cd-7572-479a-a136-a31fbcfab76c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c359af9c-rk6sf\" (UID: \"7218f8cd-7572-479a-a136-a31fbcfab76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" Oct 30 19:45:16.200170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:16.166724 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7218f8cd-7572-479a-a136-a31fbcfab76c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c359af9c-rk6sf\" (UID: \"7218f8cd-7572-479a-a136-a31fbcfab76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" Oct 30 19:45:16.200500 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:16.200470 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7218f8cd-7572-479a-a136-a31fbcfab76c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c359af9c-rk6sf\" (UID: \"7218f8cd-7572-479a-a136-a31fbcfab76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" Oct 30 19:45:16.200808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:16.200782 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6g76\" (UniqueName: \"kubernetes.io/projected/7218f8cd-7572-479a-a136-a31fbcfab76c-kube-api-access-b6g76\") pod \"maas-api-revoke-sub-c359af9c-rk6sf\" (UID: \"7218f8cd-7572-479a-a136-a31fbcfab76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" Oct 30 19:45:16.224172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:16.202166 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7218f8cd-7572-479a-a136-a31fbcfab76c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c359af9c-rk6sf\" (UID: \"7218f8cd-7572-479a-a136-a31fbcfab76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" Oct 30 19:45:16.537274 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:16.497379 5091 scope.go:122] "RemoveContainer" containerID="ea7b90426ae368ccd9511c340e0649e45c54b7b958f43c18fd2b68d43337c676" Oct 30 19:45:16.565174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:16.539766 5091 scope.go:122] "RemoveContainer" containerID="ba401fcceda69bb6b3acdd55e683745c0045d55a3684148a4fa7ecab4d96db4e" Oct 30 19:45:16.586814 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:16.579792 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:16.711170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:16.686528 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:16.955351 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:16.955311 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" podUID="8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:16.955616 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:16.844039 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:16.956376 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:45:16.690577 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:45:16.960163 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:16.955220 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6g76\" (UniqueName: \"kubernetes.io/projected/7218f8cd-7572-479a-a136-a31fbcfab76c-kube-api-access-b6g76\") pod \"maas-api-revoke-sub-c359af9c-rk6sf\" (UID: \"7218f8cd-7572-479a-a136-a31fbcfab76c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" Oct 30 19:45:16.960862 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:16.878206 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:17.360287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:17.360258 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" Oct 30 19:45:17.581974 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:17.567586 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:17.581974 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:17.567643 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" podUID="5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:17.753159 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:17.753103 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:17.755529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:17.753312 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 19:45:17.755852 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:17.754349 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:17.998572 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:45:17.998538 5091 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.524s" Oct 30 19:45:17.998863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:17.998845 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:45:18.026380 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:18.026340 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:45:18.045868 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:18.031530 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:18.046679 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:18.031884 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:18.091198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:18.078247 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:45:18.091198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:18.079433 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:45:18.091198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:18.080020 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:18.091198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:18.090787 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:45:18.103289 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:18.103258 5091 generic.go:356] "Generic (PLEG): container finished" podID="df2a21ed-1e2b-415b-b8c2-a04f82d7178e" containerID="0f352e1c7af0bf83bf1f7d828ff1da393847d4de4d7edf17c5cdcd6b6748e96e" exitCode=0 Oct 30 19:45:18.104094 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:18.103730 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" event={"ID":"df2a21ed-1e2b-415b-b8c2-a04f82d7178e","Type":"ContainerDied","Data":"0f352e1c7af0bf83bf1f7d828ff1da393847d4de4d7edf17c5cdcd6b6748e96e"} Oct 30 19:45:18.671089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:18.660388 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf"] Oct 30 19:45:18.990785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:18.990756 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" Oct 30 19:45:19.079212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.079174 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-maas-api-openshift-ca\") pod \"df2a21ed-1e2b-415b-b8c2-a04f82d7178e\" (UID: \"df2a21ed-1e2b-415b-b8c2-a04f82d7178e\") " Oct 30 19:45:19.079389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.079262 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-kube-api-access-98jh7\" (UniqueName: \"kubernetes.io/projected/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-kube-api-access-98jh7\") pod \"df2a21ed-1e2b-415b-b8c2-a04f82d7178e\" (UID: \"df2a21ed-1e2b-415b-b8c2-a04f82d7178e\") " Oct 30 19:45:19.079468 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.079450 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-maas-api-xks-ca\") pod \"df2a21ed-1e2b-415b-b8c2-a04f82d7178e\" (UID: \"df2a21ed-1e2b-415b-b8c2-a04f82d7178e\") " Oct 30 19:45:19.079639 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.079604 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-maas-api-openshift-ca" pod "df2a21ed-1e2b-415b-b8c2-a04f82d7178e" (UID: "df2a21ed-1e2b-415b-b8c2-a04f82d7178e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:19.080078 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.080044 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:19.112872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.112845 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" event={"ID":"7218f8cd-7572-479a-a136-a31fbcfab76c","Type":"ContainerStarted","Data":"1a731039fae26c9d4b709919998699ef1ed5882d39e36ac5f8917a0fd48a9c73"} Oct 30 19:45:19.114975 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.114949 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" event={"ID":"a716551d-0905-4fab-889a-f0aa0e6019b3","Type":"ContainerStarted","Data":"df4ffbd8d0ce09a8a3b8238ab712b6ee9d0b6d9a01d1ffbf58230ec666df0d0e"} Oct 30 19:45:19.116947 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.116923 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" event={"ID":"a932ddba-4214-4104-a02f-af5417aa345b","Type":"ContainerStarted","Data":"d899d5b2e4e41bb2859bc89442fc5486515cc17e8fa9c9ba74ad33092ec988ef"} Oct 30 19:45:19.118340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.118311 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" event={"ID":"df2a21ed-1e2b-415b-b8c2-a04f82d7178e","Type":"ContainerDied","Data":"a665418ed03b598951b1e011e5a4b6f87e4b7e90975b2ebc1740e8d4add4ee0d"} Oct 30 19:45:19.118340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.118329 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb" Oct 30 19:45:19.118481 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.118354 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a665418ed03b598951b1e011e5a4b6f87e4b7e90975b2ebc1740e8d4add4ee0d" Oct 30 19:45:19.146244 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.146211 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-kube-api-access-98jh7" pod "df2a21ed-1e2b-415b-b8c2-a04f82d7178e" (UID: "df2a21ed-1e2b-415b-b8c2-a04f82d7178e"). InnerVolumeSpecName "kube-api-access-98jh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:19.146728 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.146220 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-maas-api-xks-ca" pod "df2a21ed-1e2b-415b-b8c2-a04f82d7178e" (UID: "df2a21ed-1e2b-415b-b8c2-a04f82d7178e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:19.160569 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.160517 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" podStartSLOduration=19.160504551 podStartE2EDuration="19.160504551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:19.157319045 +0000 UTC m=+838.852527003" watchObservedRunningTime="2026-10-30 19:45:19.160504551 +0000 UTC m=+838.855712509" Oct 30 19:45:19.181282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.181260 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:19.181282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.181284 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98jh7\" (UniqueName: \"kubernetes.io/projected/df2a21ed-1e2b-415b-b8c2-a04f82d7178e-kube-api-access-98jh7\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:19.426082 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.425996 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:19.705782 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:19.705564 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:20.128055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:20.128015 5091 generic.go:356] "Generic (PLEG): container finished" podID="7218f8cd-7572-479a-a136-a31fbcfab76c" containerID="e90ab94e108c38259a5e6ec9c661d0d8092f387662e486cd9e5969b16fee629a" exitCode=0 Oct 30 19:45:20.128235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:20.128094 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" event={"ID":"7218f8cd-7572-479a-a136-a31fbcfab76c","Type":"ContainerDied","Data":"e90ab94e108c38259a5e6ec9c661d0d8092f387662e486cd9e5969b16fee629a"} Oct 30 19:45:20.130156 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:20.130117 5091 generic.go:356] "Generic (PLEG): container finished" podID="a716551d-0905-4fab-889a-f0aa0e6019b3" containerID="df4ffbd8d0ce09a8a3b8238ab712b6ee9d0b6d9a01d1ffbf58230ec666df0d0e" exitCode=60 Oct 30 19:45:20.130257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:20.130192 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" event={"ID":"a716551d-0905-4fab-889a-f0aa0e6019b3","Type":"ContainerDied","Data":"df4ffbd8d0ce09a8a3b8238ab712b6ee9d0b6d9a01d1ffbf58230ec666df0d0e"} Oct 30 19:45:20.130257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:20.130245 5091 scope.go:122] "RemoveContainer" containerID="ea7b90426ae368ccd9511c340e0649e45c54b7b958f43c18fd2b68d43337c676" Oct 30 19:45:20.130588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:20.130571 5091 scope.go:122] "RemoveContainer" containerID="df4ffbd8d0ce09a8a3b8238ab712b6ee9d0b6d9a01d1ffbf58230ec666df0d0e" Oct 30 19:45:20.130896 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:45:20.130877 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72_odh-ai-gateway-infra(a716551d-0905-4fab-889a-f0aa0e6019b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" podUID="a716551d-0905-4fab-889a-f0aa0e6019b3" Oct 30 19:45:20.132386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:20.132368 5091 generic.go:356] "Generic (PLEG): container finished" podID="a932ddba-4214-4104-a02f-af5417aa345b" containerID="d899d5b2e4e41bb2859bc89442fc5486515cc17e8fa9c9ba74ad33092ec988ef" exitCode=60 Oct 30 19:45:20.132471 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:20.132397 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" event={"ID":"a932ddba-4214-4104-a02f-af5417aa345b","Type":"ContainerDied","Data":"d899d5b2e4e41bb2859bc89442fc5486515cc17e8fa9c9ba74ad33092ec988ef"} Oct 30 19:45:20.132725 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:20.132712 5091 scope.go:122] "RemoveContainer" containerID="d899d5b2e4e41bb2859bc89442fc5486515cc17e8fa9c9ba74ad33092ec988ef" Oct 30 19:45:20.132985 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:45:20.132968 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk_odh-ai-gateway-infra(a932ddba-4214-4104-a02f-af5417aa345b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" podUID="a932ddba-4214-4104-a02f-af5417aa345b" Oct 30 19:45:20.143260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:20.143236 5091 scope.go:122] "RemoveContainer" containerID="ba401fcceda69bb6b3acdd55e683745c0045d55a3684148a4fa7ecab4d96db4e" Oct 30 19:45:20.148058 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:20.148033 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72"] Oct 30 19:45:20.177439 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:20.177413 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk"] Oct 30 19:45:20.658121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:20.658086 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:21.314779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.314746 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" Oct 30 19:45:21.410830 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.410803 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" Oct 30 19:45:21.413772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.413752 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" Oct 30 19:45:21.421286 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.421263 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a716551d-0905-4fab-889a-f0aa0e6019b3-kube-api-access-qmmqv\" (UniqueName: \"kubernetes.io/projected/a716551d-0905-4fab-889a-f0aa0e6019b3-kube-api-access-qmmqv\") pod \"a716551d-0905-4fab-889a-f0aa0e6019b3\" (UID: \"a716551d-0905-4fab-889a-f0aa0e6019b3\") " Oct 30 19:45:21.421369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.421335 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a716551d-0905-4fab-889a-f0aa0e6019b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a716551d-0905-4fab-889a-f0aa0e6019b3-maas-api-service-ca\") pod \"a716551d-0905-4fab-889a-f0aa0e6019b3\" (UID: \"a716551d-0905-4fab-889a-f0aa0e6019b3\") " Oct 30 19:45:21.421702 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.421679 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a716551d-0905-4fab-889a-f0aa0e6019b3-maas-api-service-ca" pod "a716551d-0905-4fab-889a-f0aa0e6019b3" (UID: "a716551d-0905-4fab-889a-f0aa0e6019b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:21.421801 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.421786 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a716551d-0905-4fab-889a-f0aa0e6019b3-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.423479 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.423454 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a716551d-0905-4fab-889a-f0aa0e6019b3-kube-api-access-qmmqv" pod "a716551d-0905-4fab-889a-f0aa0e6019b3" (UID: "a716551d-0905-4fab-889a-f0aa0e6019b3"). InnerVolumeSpecName "kube-api-access-qmmqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:21.522830 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.522798 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a932ddba-4214-4104-a02f-af5417aa345b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a932ddba-4214-4104-a02f-af5417aa345b-maas-api-service-ca\") pod \"a932ddba-4214-4104-a02f-af5417aa345b\" (UID: \"a932ddba-4214-4104-a02f-af5417aa345b\") " Oct 30 19:45:21.522999 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.522854 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7218f8cd-7572-479a-a136-a31fbcfab76c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7218f8cd-7572-479a-a136-a31fbcfab76c-maas-api-xks-ca\") pod \"7218f8cd-7572-479a-a136-a31fbcfab76c\" (UID: \"7218f8cd-7572-479a-a136-a31fbcfab76c\") " Oct 30 19:45:21.522999 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.522897 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7218f8cd-7572-479a-a136-a31fbcfab76c-kube-api-access-b6g76\" (UniqueName: \"kubernetes.io/projected/7218f8cd-7572-479a-a136-a31fbcfab76c-kube-api-access-b6g76\") pod \"7218f8cd-7572-479a-a136-a31fbcfab76c\" (UID: \"7218f8cd-7572-479a-a136-a31fbcfab76c\") " Oct 30 19:45:21.522999 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.522962 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7218f8cd-7572-479a-a136-a31fbcfab76c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7218f8cd-7572-479a-a136-a31fbcfab76c-maas-api-openshift-ca\") pod \"7218f8cd-7572-479a-a136-a31fbcfab76c\" (UID: \"7218f8cd-7572-479a-a136-a31fbcfab76c\") " Oct 30 19:45:21.523109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.523061 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a932ddba-4214-4104-a02f-af5417aa345b-kube-api-access-wggzk\" (UniqueName: \"kubernetes.io/projected/a932ddba-4214-4104-a02f-af5417aa345b-kube-api-access-wggzk\") pod \"a932ddba-4214-4104-a02f-af5417aa345b\" (UID: \"a932ddba-4214-4104-a02f-af5417aa345b\") " Oct 30 19:45:21.523267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.523239 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a932ddba-4214-4104-a02f-af5417aa345b-maas-api-service-ca" pod "a932ddba-4214-4104-a02f-af5417aa345b" (UID: "a932ddba-4214-4104-a02f-af5417aa345b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:21.523452 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.523422 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7218f8cd-7572-479a-a136-a31fbcfab76c-maas-api-openshift-ca" pod "7218f8cd-7572-479a-a136-a31fbcfab76c" (UID: "7218f8cd-7572-479a-a136-a31fbcfab76c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:21.523816 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.523795 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7218f8cd-7572-479a-a136-a31fbcfab76c-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.523816 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.523817 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmmqv\" (UniqueName: \"kubernetes.io/projected/a716551d-0905-4fab-889a-f0aa0e6019b3-kube-api-access-qmmqv\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.523960 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.523827 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a932ddba-4214-4104-a02f-af5417aa345b-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.525230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.525204 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7218f8cd-7572-479a-a136-a31fbcfab76c-maas-api-xks-ca" pod "7218f8cd-7572-479a-a136-a31fbcfab76c" (UID: "7218f8cd-7572-479a-a136-a31fbcfab76c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:21.525515 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.525494 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7218f8cd-7572-479a-a136-a31fbcfab76c-kube-api-access-b6g76" pod "7218f8cd-7572-479a-a136-a31fbcfab76c" (UID: "7218f8cd-7572-479a-a136-a31fbcfab76c"). InnerVolumeSpecName "kube-api-access-b6g76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:21.525586 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.525527 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a932ddba-4214-4104-a02f-af5417aa345b-kube-api-access-wggzk" pod "a932ddba-4214-4104-a02f-af5417aa345b" (UID: "a932ddba-4214-4104-a02f-af5417aa345b"). InnerVolumeSpecName "kube-api-access-wggzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:21.625580 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.625542 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wggzk\" (UniqueName: \"kubernetes.io/projected/a932ddba-4214-4104-a02f-af5417aa345b-kube-api-access-wggzk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.625580 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.625576 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7218f8cd-7572-479a-a136-a31fbcfab76c-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.625580 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.625586 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6g76\" (UniqueName: \"kubernetes.io/projected/7218f8cd-7572-479a-a136-a31fbcfab76c-kube-api-access-b6g76\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:21.687598 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:21.686061 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:22.153235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.153204 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" Oct 30 19:45:22.153421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.153204 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf" event={"ID":"7218f8cd-7572-479a-a136-a31fbcfab76c","Type":"ContainerDied","Data":"1a731039fae26c9d4b709919998699ef1ed5882d39e36ac5f8917a0fd48a9c73"} Oct 30 19:45:22.153421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.153358 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a731039fae26c9d4b709919998699ef1ed5882d39e36ac5f8917a0fd48a9c73" Oct 30 19:45:22.154812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.154792 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" Oct 30 19:45:22.154947 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.154809 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72" event={"ID":"a716551d-0905-4fab-889a-f0aa0e6019b3","Type":"ContainerDied","Data":"6b1b3d6ce3bde1c6b4cc9d67b3395723de6be02e3b9d87bd54d1c3007fd1305c"} Oct 30 19:45:22.154947 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.154883 5091 scope.go:122] "RemoveContainer" containerID="df4ffbd8d0ce09a8a3b8238ab712b6ee9d0b6d9a01d1ffbf58230ec666df0d0e" Oct 30 19:45:22.163370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.163345 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" event={"ID":"a932ddba-4214-4104-a02f-af5417aa345b","Type":"ContainerDied","Data":"212822af0c360b9655d3691ba57fc1ae6f3af27e2f65eef6f48c6e3fbf5c0621"} Oct 30 19:45:22.163457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.163433 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk" Oct 30 19:45:22.167002 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.166981 5091 scope.go:122] "RemoveContainer" containerID="d899d5b2e4e41bb2859bc89442fc5486515cc17e8fa9c9ba74ad33092ec988ef" Oct 30 19:45:22.215507 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.215480 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72"] Oct 30 19:45:22.224858 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.224832 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889825-g7n72"] Oct 30 19:45:22.242744 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.242715 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk"] Oct 30 19:45:22.250122 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.250078 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-8836a8-29889825-8gnkk"] Oct 30 19:45:22.442644 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.442564 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a716551d-0905-4fab-889a-f0aa0e6019b3" path="/var/lib/kubelet/pods/a716551d-0905-4fab-889a-f0aa0e6019b3/volumes" Oct 30 19:45:22.443114 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.442958 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a932ddba-4214-4104-a02f-af5417aa345b" path="/var/lib/kubelet/pods/a932ddba-4214-4104-a02f-af5417aa345b/volumes" Oct 30 19:45:22.580100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:22.580067 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:23.679185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.675320 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:23.906697 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.906661 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw"] Oct 30 19:45:23.907260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907245 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a716551d-0905-4fab-889a-f0aa0e6019b3" containerName="cleanup" Oct 30 19:45:23.907318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907260 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a716551d-0905-4fab-889a-f0aa0e6019b3" containerName="cleanup" Oct 30 19:45:23.907318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907278 5091 cpu_manager.go:414] "containerMap: removing container" podUID="df2a21ed-1e2b-415b-b8c2-a04f82d7178e" containerName="revoke-keys" Oct 30 19:45:23.907318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907283 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df2a21ed-1e2b-415b-b8c2-a04f82d7178e" containerName="revoke-keys" Oct 30 19:45:23.907318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907291 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a932ddba-4214-4104-a02f-af5417aa345b" containerName="cleanup" Oct 30 19:45:23.907318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907295 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a932ddba-4214-4104-a02f-af5417aa345b" containerName="cleanup" Oct 30 19:45:23.907318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907317 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a716551d-0905-4fab-889a-f0aa0e6019b3" containerName="cleanup" Oct 30 19:45:23.907604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907321 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a716551d-0905-4fab-889a-f0aa0e6019b3" containerName="cleanup" Oct 30 19:45:23.907604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907327 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a932ddba-4214-4104-a02f-af5417aa345b" containerName="cleanup" Oct 30 19:45:23.907604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907331 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a932ddba-4214-4104-a02f-af5417aa345b" containerName="cleanup" Oct 30 19:45:23.907604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907344 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a716551d-0905-4fab-889a-f0aa0e6019b3" containerName="cleanup" Oct 30 19:45:23.907604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907348 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a716551d-0905-4fab-889a-f0aa0e6019b3" containerName="cleanup" Oct 30 19:45:23.907604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907357 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7218f8cd-7572-479a-a136-a31fbcfab76c" containerName="revoke-keys" Oct 30 19:45:23.907604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907361 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7218f8cd-7572-479a-a136-a31fbcfab76c" containerName="revoke-keys" Oct 30 19:45:23.907604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907457 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a716551d-0905-4fab-889a-f0aa0e6019b3" containerName="cleanup" Oct 30 19:45:23.907604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907464 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a932ddba-4214-4104-a02f-af5417aa345b" containerName="cleanup" Oct 30 19:45:23.907604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907471 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a716551d-0905-4fab-889a-f0aa0e6019b3" containerName="cleanup" Oct 30 19:45:23.907604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907476 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7218f8cd-7572-479a-a136-a31fbcfab76c" containerName="revoke-keys" Oct 30 19:45:23.907604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907483 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a932ddba-4214-4104-a02f-af5417aa345b" containerName="cleanup" Oct 30 19:45:23.907604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.907491 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="df2a21ed-1e2b-415b-b8c2-a04f82d7178e" containerName="revoke-keys" Oct 30 19:45:23.912543 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.912518 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:23.923741 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.923713 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2a3e1b-openshift-default-dockercfg-gv7kl\"" Oct 30 19:45:23.927463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:23.927437 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw"] Oct 30 19:45:24.062914 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.062875 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-podinfo\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.063087 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.062925 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-token\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.063087 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.063058 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-workload-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.063218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.063196 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-credential-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.063277 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.063236 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-data\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.063277 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.063255 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr7p8\" (UniqueName: \"kubernetes.io/projected/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-kube-api-access-mr7p8\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.063394 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.063377 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istiod-ca-cert\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.063449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.063420 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-envoy\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.063510 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.063496 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-workload-certs\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.164627 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.164593 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-workload-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.164794 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.164681 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-credential-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.164794 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.164707 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-data\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.164794 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.164723 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr7p8\" (UniqueName: \"kubernetes.io/projected/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-kube-api-access-mr7p8\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.164794 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.164761 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istiod-ca-cert\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.165032 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.164797 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-envoy\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.165032 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.164833 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-workload-certs\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.165032 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.164902 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-podinfo\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.165032 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.164939 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-token\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.165246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.165116 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-workload-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.165597 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.165560 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-data\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.165714 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.165611 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-credential-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.165714 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.165663 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-workload-certs\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.165714 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.165676 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istiod-ca-cert\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.167398 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.167370 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-envoy\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.167602 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.167581 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-podinfo\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.178967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.178798 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-token\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.180038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.180014 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr7p8\" (UniqueName: \"kubernetes.io/projected/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-kube-api-access-mr7p8\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.231723 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.231698 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:24.528121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.528092 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw"] Oct 30 19:45:24.528668 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:45:24.528643 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e73eeb_fd82_4fd3_86ab_ca818b7961a8.slice/crio-d5386632bc57e5f838cee39a9f4e2dd7096ede616080f05da477f86cac629751 WatchSource:0}: Error finding container d5386632bc57e5f838cee39a9f4e2dd7096ede616080f05da477f86cac629751: Status 404 returned error can't find the container with id d5386632bc57e5f838cee39a9f4e2dd7096ede616080f05da477f86cac629751 Oct 30 19:45:24.531790 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.531758 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:45:24.531877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.531823 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:45:24.531877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.531852 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:45:24.588189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:24.585991 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:25.191285 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:25.191251 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" event={"ID":"56e73eeb-fd82-4fd3-86ab-ca818b7961a8","Type":"ContainerStarted","Data":"19f7d3dd2b17ccc22d8f28aa42604da8cc0469b9989e1e2778f9458d23974969"} Oct 30 19:45:25.191691 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:25.191315 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" event={"ID":"56e73eeb-fd82-4fd3-86ab-ca818b7961a8","Type":"ContainerStarted","Data":"d5386632bc57e5f838cee39a9f4e2dd7096ede616080f05da477f86cac629751"} Oct 30 19:45:25.223235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:25.223187 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podStartSLOduration=2.223171726 podStartE2EDuration="2.223171726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:25.219656331 +0000 UTC m=+844.914864314" watchObservedRunningTime="2026-10-30 19:45:25.223171726 +0000 UTC m=+844.918379684" Oct 30 19:45:25.232855 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:25.232823 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:25.234497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:25.234466 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:45:25.234626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:25.234519 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:45:25.540024 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:25.537617 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:26.232678 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:26.232650 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:45:26.233048 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:26.232699 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:45:26.502563 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:26.502533 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:27.232177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:27.232118 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:45:27.232333 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:27.232218 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:45:27.430200 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:27.423854 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:27.589736 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:27.589704 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:27.887993 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:27.887895 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:28.035774 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.035735 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:28.232092 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.232014 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:45:28.232092 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.232077 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:45:28.521238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.521189 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:28.583133 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.583103 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n"] Oct 30 19:45:28.583757 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.583742 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a932ddba-4214-4104-a02f-af5417aa345b" containerName="cleanup" Oct 30 19:45:28.583757 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.583758 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a932ddba-4214-4104-a02f-af5417aa345b" containerName="cleanup" Oct 30 19:45:28.583855 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.583843 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a716551d-0905-4fab-889a-f0aa0e6019b3" containerName="cleanup" Oct 30 19:45:28.583855 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.583853 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a932ddba-4214-4104-a02f-af5417aa345b" containerName="cleanup" Oct 30 19:45:28.587601 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.587582 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" Oct 30 19:45:28.630174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.623948 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:45:28.630174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.623993 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n"] Oct 30 19:45:28.630174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.624091 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:45:28.690181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.678984 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvzvw\" (UniqueName: \"kubernetes.io/projected/35570a64-0a60-408f-97de-897f2b97132d-kube-api-access-vvzvw\") pod \"maas-api-revoke-sub-1ac7309d-4xf7n\" (UID: \"35570a64-0a60-408f-97de-897f2b97132d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" Oct 30 19:45:28.710331 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.710293 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35570a64-0a60-408f-97de-897f2b97132d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ac7309d-4xf7n\" (UID: \"35570a64-0a60-408f-97de-897f2b97132d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" Oct 30 19:45:28.710662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.710641 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35570a64-0a60-408f-97de-897f2b97132d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ac7309d-4xf7n\" (UID: \"35570a64-0a60-408f-97de-897f2b97132d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" Oct 30 19:45:28.812562 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.812470 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35570a64-0a60-408f-97de-897f2b97132d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ac7309d-4xf7n\" (UID: \"35570a64-0a60-408f-97de-897f2b97132d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" Oct 30 19:45:28.812562 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.812561 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35570a64-0a60-408f-97de-897f2b97132d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ac7309d-4xf7n\" (UID: \"35570a64-0a60-408f-97de-897f2b97132d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" Oct 30 19:45:28.812751 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.812694 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvzvw\" (UniqueName: \"kubernetes.io/projected/35570a64-0a60-408f-97de-897f2b97132d-kube-api-access-vvzvw\") pod \"maas-api-revoke-sub-1ac7309d-4xf7n\" (UID: \"35570a64-0a60-408f-97de-897f2b97132d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" Oct 30 19:45:28.813184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.813126 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35570a64-0a60-408f-97de-897f2b97132d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1ac7309d-4xf7n\" (UID: \"35570a64-0a60-408f-97de-897f2b97132d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" Oct 30 19:45:28.814833 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.814808 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35570a64-0a60-408f-97de-897f2b97132d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1ac7309d-4xf7n\" (UID: \"35570a64-0a60-408f-97de-897f2b97132d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" Oct 30 19:45:28.826946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.826925 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvzvw\" (UniqueName: \"kubernetes.io/projected/35570a64-0a60-408f-97de-897f2b97132d-kube-api-access-vvzvw\") pod \"maas-api-revoke-sub-1ac7309d-4xf7n\" (UID: \"35570a64-0a60-408f-97de-897f2b97132d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" Oct 30 19:45:28.952169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:28.945223 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" Oct 30 19:45:29.232772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:29.232691 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:45:29.232772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:29.232752 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:45:29.284475 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:29.284023 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n"] Oct 30 19:45:29.286601 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:45:29.286564 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35570a64_0a60_408f_97de_897f2b97132d.slice/crio-4b53159c01bcd5f30899c5bcf8fcd6768c46c56eeb02abbe748c19400b65df2a WatchSource:0}: Error finding container 4b53159c01bcd5f30899c5bcf8fcd6768c46c56eeb02abbe748c19400b65df2a: Status 404 returned error can't find the container with id 4b53159c01bcd5f30899c5bcf8fcd6768c46c56eeb02abbe748c19400b65df2a Oct 30 19:45:29.477710 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:29.477673 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:29.955219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:29.954774 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:30.236895 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:30.236811 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:45:30.236895 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:30.236867 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:45:30.293555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:30.293372 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:30.389598 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:30.389559 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" event={"ID":"35570a64-0a60-408f-97de-897f2b97132d","Type":"ContainerStarted","Data":"047571ec2427200680926b3e3e1a0f4855a5356ea0383234ce731ff70debb2d4"} Oct 30 19:45:30.390026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:30.389988 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" event={"ID":"35570a64-0a60-408f-97de-897f2b97132d","Type":"ContainerStarted","Data":"4b53159c01bcd5f30899c5bcf8fcd6768c46c56eeb02abbe748c19400b65df2a"} Oct 30 19:45:30.508564 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:30.508448 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:30.545658 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:30.530618 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" podStartSLOduration=2.530589265 podStartE2EDuration="2.530589265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:30.513405502 +0000 UTC m=+850.208613460" watchObservedRunningTime="2026-10-30 19:45:30.530589265 +0000 UTC m=+850.225797227" Oct 30 19:45:30.667458 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:30.667417 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:31.232836 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:31.232796 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Oct 30 19:45:31.233329 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:31.232859 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Oct 30 19:45:31.457798 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:31.457323 5091 generic.go:356] "Generic (PLEG): container finished" podID="35570a64-0a60-408f-97de-897f2b97132d" containerID="047571ec2427200680926b3e3e1a0f4855a5356ea0383234ce731ff70debb2d4" exitCode=0 Oct 30 19:45:31.457798 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:31.457382 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" event={"ID":"35570a64-0a60-408f-97de-897f2b97132d","Type":"ContainerDied","Data":"047571ec2427200680926b3e3e1a0f4855a5356ea0383234ce731ff70debb2d4"} Oct 30 19:45:31.599492 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:31.599458 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:32.805240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:32.803247 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:32.872279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:32.872234 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded" Oct 30 19:45:33.098218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:33.053716 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:33.268022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:33.267246 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:33.268022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:33.267332 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:33.376337 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:33.376233 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:33.898171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:33.875298 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:33.898171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:33.875646 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:45:33.898171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:33.876312 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:45:34.404454 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:34.404409 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Oct 30 19:45:34.475651 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:34.405292 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:34.476698 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:34.476663 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:34.489726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:34.407508 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 19:45:34.618291 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:34.604967 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:45:34.893329 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:34.893285 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 19:45:34.893589 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:34.893564 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:45:34.894408 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:34.894384 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:45:34.921077 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:34.911337 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" Oct 30 19:45:35.217226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:35.192040 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 19:45:35.330625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:35.330572 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35570a64-0a60-408f-97de-897f2b97132d-kube-api-access-vvzvw\" (UniqueName: \"kubernetes.io/projected/35570a64-0a60-408f-97de-897f2b97132d-kube-api-access-vvzvw\") pod \"35570a64-0a60-408f-97de-897f2b97132d\" (UID: \"35570a64-0a60-408f-97de-897f2b97132d\") " Oct 30 19:45:35.330951 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:35.330928 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35570a64-0a60-408f-97de-897f2b97132d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35570a64-0a60-408f-97de-897f2b97132d-maas-api-openshift-ca\") pod \"35570a64-0a60-408f-97de-897f2b97132d\" (UID: \"35570a64-0a60-408f-97de-897f2b97132d\") " Oct 30 19:45:35.331116 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:35.331096 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35570a64-0a60-408f-97de-897f2b97132d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35570a64-0a60-408f-97de-897f2b97132d-maas-api-xks-ca\") pod \"35570a64-0a60-408f-97de-897f2b97132d\" (UID: \"35570a64-0a60-408f-97de-897f2b97132d\") " Oct 30 19:45:35.477478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:35.477271 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35570a64-0a60-408f-97de-897f2b97132d-maas-api-openshift-ca" pod "35570a64-0a60-408f-97de-897f2b97132d" (UID: "35570a64-0a60-408f-97de-897f2b97132d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:35.615008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:35.614971 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:45:35.677200 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:35.668585 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35570a64-0a60-408f-97de-897f2b97132d-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:35.677200 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:35.670113 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:35.677200 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:35.670195 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:35.839171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:35.829728 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35570a64-0a60-408f-97de-897f2b97132d-maas-api-xks-ca" pod "35570a64-0a60-408f-97de-897f2b97132d" (UID: "35570a64-0a60-408f-97de-897f2b97132d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:36.086172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:35.982921 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:36.110443 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.083267 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:36.111413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.111374 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35570a64-0a60-408f-97de-897f2b97132d-kube-api-access-vvzvw" pod "35570a64-0a60-408f-97de-897f2b97132d" (UID: "35570a64-0a60-408f-97de-897f2b97132d"). InnerVolumeSpecName "kube-api-access-vvzvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:36.196170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.134454 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35570a64-0a60-408f-97de-897f2b97132d-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:36.196170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.185414 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" Oct 30 19:45:36.203398 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.203264 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n" event={"ID":"35570a64-0a60-408f-97de-897f2b97132d","Type":"ContainerDied","Data":"4b53159c01bcd5f30899c5bcf8fcd6768c46c56eeb02abbe748c19400b65df2a"} Oct 30 19:45:36.203672 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.203638 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b53159c01bcd5f30899c5bcf8fcd6768c46c56eeb02abbe748c19400b65df2a" Oct 30 19:45:36.227274 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.227225 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:36.347217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.228380 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:36.347217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.259019 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvzvw\" (UniqueName: \"kubernetes.io/projected/35570a64-0a60-408f-97de-897f2b97132d-kube-api-access-vvzvw\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:36.372222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.291248 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:45:36.646202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.646082 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:36.763264 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.763213 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:36.776230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.768380 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:36.804235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.804189 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:36.846261 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.846218 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" podUID="a95d6ded-89c1-4aee-b4e2-da19030afca8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:36.854172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.806109 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:36.873796 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.872344 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:36.873796 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.872412 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" podUID="69602ea5-4de9-4b3a-8e3b-a3045d919c82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:36.922493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.922317 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:36.922493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:36.922374 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" podUID="1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:37.038675 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:37.038639 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:37.111270 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:37.111228 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:45:37.120197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:37.115784 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:37.300514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:37.300390 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:37.371709 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:37.371665 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:37.668539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:37.668375 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:37.669959 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:37.669010 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:45:37.856356 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:37.856264 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:37.856671 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:37.856632 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:37.981726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:37.959227 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded" Oct 30 19:45:38.025740 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:38.025245 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:38.025740 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:38.025313 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:38.122171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:38.118026 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:38.136917 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:38.135290 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:38.310503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:38.310457 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:38.702425 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:38.637972 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:38.807357 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:38.714381 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 19:45:38.936173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:38.909255 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:38.936173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:38.909319 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:39.110879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:39.110841 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:39.161699 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:39.161651 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:39.252187 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:39.185076 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:39.328165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:39.254590 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:39.328165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:39.326279 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 19:45:39.381224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:39.381117 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:39.727229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:39.724611 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:39.787538 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:39.787489 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:45:39.802739 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:39.802669 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:39.825264 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:39.825239 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:45:39.939173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:39.936482 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:45:39.965879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:39.961562 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:39.983321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:39.976259 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:45:39.983321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:39.976318 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:40.550753 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.550048 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:40.613287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.613253 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:45:40.613287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.613303 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g"] Oct 30 19:45:40.614241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.614213 5091 cpu_manager.go:414] "containerMap: removing container" podUID="35570a64-0a60-408f-97de-897f2b97132d" containerName="revoke-keys" Oct 30 19:45:40.614241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.614240 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35570a64-0a60-408f-97de-897f2b97132d" containerName="revoke-keys" Oct 30 19:45:40.614468 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.614450 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="35570a64-0a60-408f-97de-897f2b97132d" containerName="revoke-keys" Oct 30 19:45:40.621475 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.621431 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g"] Oct 30 19:45:40.621599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.621541 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.650314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.631457 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-477fd9c0-openshift-default-dockercfg-s2cnd\"" Oct 30 19:45:40.676711 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.676683 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:40.683222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.683187 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-workload-socket\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.683493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.683454 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52104831-cd64-4eea-b4ed-a389c06a4c30-istiod-ca-cert\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.687213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.683648 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-data\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.687438 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.687416 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-token\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.687580 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.687564 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmszr\" (UniqueName: \"kubernetes.io/projected/52104831-cd64-4eea-b4ed-a389c06a4c30-kube-api-access-lmszr\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.687742 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.687725 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-workload-certs\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.688553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.688530 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-envoy\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.688831 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.688811 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-podinfo\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.691815 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.691793 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-credential-socket\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.694647 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.694620 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:45:40.793706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.793674 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-podinfo\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.793886 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.793735 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-credential-socket\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.793886 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.793796 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-workload-socket\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.793886 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.793815 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52104831-cd64-4eea-b4ed-a389c06a4c30-istiod-ca-cert\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.793886 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.793858 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-data\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.794101 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.793901 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-token\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.794101 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.793929 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmszr\" (UniqueName: \"kubernetes.io/projected/52104831-cd64-4eea-b4ed-a389c06a4c30-kube-api-access-lmszr\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.794101 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.793969 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-workload-certs\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.794101 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.793985 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-envoy\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.794748 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.794720 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-credential-socket\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.795107 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.795002 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-workload-socket\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.795596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.795314 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-data\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.795596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.795554 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-workload-certs\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.795849 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.795774 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52104831-cd64-4eea-b4ed-a389c06a4c30-istiod-ca-cert\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.797338 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.797315 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-podinfo\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.804303 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.804235 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-envoy\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.808703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.808679 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-token\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.811309 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.811285 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmszr\" (UniqueName: \"kubernetes.io/projected/52104831-cd64-4eea-b4ed-a389c06a4c30-kube-api-access-lmszr\") pod \"e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:40.811397 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.811315 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:40.991953 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:40.991923 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:41.192182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:41.191010 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:41.285989 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:45:41.285955 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52104831_cd64_4eea_b4ed_a389c06a4c30.slice/crio-d829a08db98820c7958bcd94b69e0e2872bf4b151b4a025cf3dc475ffd320578 WatchSource:0}: Error finding container d829a08db98820c7958bcd94b69e0e2872bf4b151b4a025cf3dc475ffd320578: Status 404 returned error can't find the container with id d829a08db98820c7958bcd94b69e0e2872bf4b151b4a025cf3dc475ffd320578 Oct 30 19:45:41.286555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:41.286532 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g"] Oct 30 19:45:41.288463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:41.288429 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:45:41.288558 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:41.288513 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:45:41.288596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:41.288559 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:45:41.622946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:41.622854 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" event={"ID":"52104831-cd64-4eea-b4ed-a389c06a4c30","Type":"ContainerStarted","Data":"687c8174c37d688c6081946d8b60ec6cddd9486ff6ea25c5cfe7dac623e7df9e"} Oct 30 19:45:41.622946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:41.622925 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" event={"ID":"52104831-cd64-4eea-b4ed-a389c06a4c30","Type":"ContainerStarted","Data":"d829a08db98820c7958bcd94b69e0e2872bf4b151b4a025cf3dc475ffd320578"} Oct 30 19:45:41.672732 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:41.672665 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" podStartSLOduration=1.672646706 podStartE2EDuration="1.672646706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:41.670872621 +0000 UTC m=+861.366080579" watchObservedRunningTime="2026-10-30 19:45:41.672646706 +0000 UTC m=+861.367854667" Oct 30 19:45:41.992214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:41.992119 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:42.324180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:42.320413 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:42.998790 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:42.998759 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:43.550086 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:43.546899 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:43.647058 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:43.647029 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:43.648014 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:43.647992 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:45:43.924875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:43.924752 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:44.089225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:44.089192 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:44.608120 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:44.608082 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:44.787929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:44.787900 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:45.206276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:45.206239 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:45.392834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:45.392778 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:45.829341 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:45.829304 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:45.977589 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:45:45.977549 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:45:46.784637 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:46.775328 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:47.849831 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:47.849796 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:48.707474 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:48.707443 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:48.859172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:48.858875 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:49.004801 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:49.004770 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:49.832578 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:49.832545 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:50.163268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:50.161505 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:50.324274 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:50.324235 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:51.046820 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.046778 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:51.204663 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.204616 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:51.423948 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.423858 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:51.631718 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.631681 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d"] Oct 30 19:45:51.639724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.639694 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" Oct 30 19:45:51.646308 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.646280 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d"] Oct 30 19:45:51.651488 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.651464 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:45:51.651723 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.651708 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:45:51.745226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.744960 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df1aca0-a819-4417-b85a-f9d2fda67997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77c25d0d-fn72d\" (UID: \"2df1aca0-a819-4417-b85a-f9d2fda67997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" Oct 30 19:45:51.745226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.745037 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df1aca0-a819-4417-b85a-f9d2fda67997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77c25d0d-fn72d\" (UID: \"2df1aca0-a819-4417-b85a-f9d2fda67997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" Oct 30 19:45:51.745469 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.745391 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmp4c\" (UniqueName: \"kubernetes.io/projected/2df1aca0-a819-4417-b85a-f9d2fda67997-kube-api-access-lmp4c\") pod \"maas-api-revoke-sub-77c25d0d-fn72d\" (UID: \"2df1aca0-a819-4417-b85a-f9d2fda67997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" Oct 30 19:45:51.847087 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.847054 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df1aca0-a819-4417-b85a-f9d2fda67997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77c25d0d-fn72d\" (UID: \"2df1aca0-a819-4417-b85a-f9d2fda67997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" Oct 30 19:45:51.847317 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.847102 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df1aca0-a819-4417-b85a-f9d2fda67997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77c25d0d-fn72d\" (UID: \"2df1aca0-a819-4417-b85a-f9d2fda67997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" Oct 30 19:45:51.847317 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.847299 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmp4c\" (UniqueName: \"kubernetes.io/projected/2df1aca0-a819-4417-b85a-f9d2fda67997-kube-api-access-lmp4c\") pod \"maas-api-revoke-sub-77c25d0d-fn72d\" (UID: \"2df1aca0-a819-4417-b85a-f9d2fda67997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" Oct 30 19:45:51.847766 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.847746 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df1aca0-a819-4417-b85a-f9d2fda67997-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77c25d0d-fn72d\" (UID: \"2df1aca0-a819-4417-b85a-f9d2fda67997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" Oct 30 19:45:51.849526 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.849501 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df1aca0-a819-4417-b85a-f9d2fda67997-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77c25d0d-fn72d\" (UID: \"2df1aca0-a819-4417-b85a-f9d2fda67997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" Oct 30 19:45:51.861466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:51.861440 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmp4c\" (UniqueName: \"kubernetes.io/projected/2df1aca0-a819-4417-b85a-f9d2fda67997-kube-api-access-lmp4c\") pod \"maas-api-revoke-sub-77c25d0d-fn72d\" (UID: \"2df1aca0-a819-4417-b85a-f9d2fda67997\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" Oct 30 19:45:52.010348 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:52.010317 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" Oct 30 19:45:52.135971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:52.121831 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:52.301895 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:52.301864 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d"] Oct 30 19:45:52.768914 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:52.768876 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" event={"ID":"2df1aca0-a819-4417-b85a-f9d2fda67997","Type":"ContainerStarted","Data":"887e56383e2b89311f336d4b2f7adfde157fa8ea945c44ffabe4d314cd589a76"} Oct 30 19:45:52.769095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:52.768942 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" event={"ID":"2df1aca0-a819-4417-b85a-f9d2fda67997","Type":"ContainerStarted","Data":"c9c5ec4df3e4b9c4b83de5e740e13730801534fe782fc501c0da5789f4216142"} Oct 30 19:45:52.796497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:52.796451 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" podStartSLOduration=1.7964377759999999 podStartE2EDuration="1.796437776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:52.795135437 +0000 UTC m=+872.490343396" watchObservedRunningTime="2026-10-30 19:45:52.796437776 +0000 UTC m=+872.491645734" Oct 30 19:45:53.118500 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:53.118462 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:53.833170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:53.832391 5091 generic.go:356] "Generic (PLEG): container finished" podID="2df1aca0-a819-4417-b85a-f9d2fda67997" containerID="887e56383e2b89311f336d4b2f7adfde157fa8ea945c44ffabe4d314cd589a76" exitCode=0 Oct 30 19:45:53.833170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:53.832561 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" event={"ID":"2df1aca0-a819-4417-b85a-f9d2fda67997","Type":"ContainerDied","Data":"887e56383e2b89311f336d4b2f7adfde157fa8ea945c44ffabe4d314cd589a76"} Oct 30 19:45:54.068170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.058790 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs"] Oct 30 19:45:54.172220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.171119 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs"] Oct 30 19:45:54.187167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.172617 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.332170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.331396 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs29v\" (UniqueName: \"kubernetes.io/projected/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-kube-api-access-rs29v\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.332170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.331454 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-token\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.332170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.331512 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-podinfo\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.332170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.331553 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-data\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.332170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.331602 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-credential-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.332170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.331717 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-workload-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.332170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.331742 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-workload-certs\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.332170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.331782 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istiod-ca-cert\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.332170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.331827 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-envoy\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.482217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.474772 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-workload-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.482217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.474817 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-workload-certs\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.482217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.474860 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istiod-ca-cert\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.482217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.474897 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-envoy\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.482217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.474984 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs29v\" (UniqueName: \"kubernetes.io/projected/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-kube-api-access-rs29v\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.482217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.475018 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-token\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.482217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.475074 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-podinfo\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.482217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.475118 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-data\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.482217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.475180 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-credential-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.482217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.475730 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-credential-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.482217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.476570 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-workload-socket\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.482217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.476835 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-workload-certs\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.499174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.496629 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istiod-ca-cert\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.510168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.507683 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-data\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.571521 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.535452 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-envoy\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.623169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.622782 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-podinfo\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.673824 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.656834 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs29v\" (UniqueName: \"kubernetes.io/projected/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-kube-api-access-rs29v\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.732178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.730682 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-token\") pod \"e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:54.842172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:54.839230 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:55.040200 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:55.030247 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:55.154330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:55.152331 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:55.154996 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:55.154974 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:55.332364 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:55.331224 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:55.513085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:55.512504 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:55.733197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:55.733164 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:56.256858 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.164414 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:45:56.303201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.244086 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:56.303201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.244256 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq"] Oct 30 19:45:56.327195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.327165 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:45:56.327772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.327755 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq"] Oct 30 19:45:56.362269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.327457 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.383599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.383568 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-6ac47d-openshift-default-dockercfg-bwf56\"" Oct 30 19:45:56.471172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.452074 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:56.513219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.477900 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-token\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.513219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.477956 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-credential-socket\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.513219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.478027 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-workload-certs\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.513219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.478085 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-podinfo\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.513219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.478111 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-data\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.513219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.478186 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-workload-socket\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.513219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.478241 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istiod-ca-cert\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.513219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.478283 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-envoy\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.513219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.478397 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbk8w\" (UniqueName: \"kubernetes.io/projected/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-kube-api-access-nbk8w\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.611166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.592221 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-token\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.611166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.592279 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-credential-socket\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.611166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.592364 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-workload-certs\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.611166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.592420 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-podinfo\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.611166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.592446 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-data\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.611166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.592510 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-workload-socket\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.611166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.592558 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istiod-ca-cert\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.611166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.592598 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-envoy\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.611166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.592697 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbk8w\" (UniqueName: \"kubernetes.io/projected/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-kube-api-access-nbk8w\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.611166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.593709 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-credential-socket\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.611166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.593979 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-workload-certs\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.611166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.606508 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-podinfo\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.611166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.606788 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-data\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.611166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.607024 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-workload-socket\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.611166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.607630 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istiod-ca-cert\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.612199 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.611283 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-envoy\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.618592 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.614497 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-token\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.621214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.621129 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbk8w\" (UniqueName: \"kubernetes.io/projected/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-kube-api-access-nbk8w\") pod \"e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.787645 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.787614 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:45:56.876170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.857325 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:56.975432 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.967056 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:45:56.975432 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.967242 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:45:56.975432 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:56.967346 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:45:57.084254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.079555 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs"] Oct 30 19:45:57.105536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.105498 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" event={"ID":"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a","Type":"ContainerStarted","Data":"8da519ea956fcfe1fe910da1dd4104a9f6c1b5130449f0a9f9780037ca8dea9d"} Oct 30 19:45:57.196166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.194328 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" Oct 30 19:45:57.290260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.288788 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2df1aca0-a819-4417-b85a-f9d2fda67997-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df1aca0-a819-4417-b85a-f9d2fda67997-maas-api-xks-ca\") pod \"2df1aca0-a819-4417-b85a-f9d2fda67997\" (UID: \"2df1aca0-a819-4417-b85a-f9d2fda67997\") " Oct 30 19:45:57.290260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.288932 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2df1aca0-a819-4417-b85a-f9d2fda67997-kube-api-access-lmp4c\" (UniqueName: \"kubernetes.io/projected/2df1aca0-a819-4417-b85a-f9d2fda67997-kube-api-access-lmp4c\") pod \"2df1aca0-a819-4417-b85a-f9d2fda67997\" (UID: \"2df1aca0-a819-4417-b85a-f9d2fda67997\") " Oct 30 19:45:57.290260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.289022 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2df1aca0-a819-4417-b85a-f9d2fda67997-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df1aca0-a819-4417-b85a-f9d2fda67997-maas-api-openshift-ca\") pod \"2df1aca0-a819-4417-b85a-f9d2fda67997\" (UID: \"2df1aca0-a819-4417-b85a-f9d2fda67997\") " Oct 30 19:45:57.292007 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.291947 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2df1aca0-a819-4417-b85a-f9d2fda67997-maas-api-openshift-ca" pod "2df1aca0-a819-4417-b85a-f9d2fda67997" (UID: "2df1aca0-a819-4417-b85a-f9d2fda67997"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:45:57.326470 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.323410 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df1aca0-a819-4417-b85a-f9d2fda67997-maas-api-xks-ca" pod "2df1aca0-a819-4417-b85a-f9d2fda67997" (UID: "2df1aca0-a819-4417-b85a-f9d2fda67997"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:45:57.326470 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.323549 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df1aca0-a819-4417-b85a-f9d2fda67997-kube-api-access-lmp4c" pod "2df1aca0-a819-4417-b85a-f9d2fda67997" (UID: "2df1aca0-a819-4417-b85a-f9d2fda67997"). InnerVolumeSpecName "kube-api-access-lmp4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:45:57.391102 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.391022 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:57.391299 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.391272 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmp4c\" (UniqueName: \"kubernetes.io/projected/2df1aca0-a819-4417-b85a-f9d2fda67997-kube-api-access-lmp4c\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:57.391407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.391306 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2df1aca0-a819-4417-b85a-f9d2fda67997-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:57.391407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.391322 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2df1aca0-a819-4417-b85a-f9d2fda67997-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:45:57.509243 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:45:57.509205 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f70f61_e4ea_4f14_bfdc_92a0ad0ac9da.slice/crio-104ae35098c5292b32909bb0b71b529764cc79a1bbaeb19fbd58e47eaceaafca WatchSource:0}: Error finding container 104ae35098c5292b32909bb0b71b529764cc79a1bbaeb19fbd58e47eaceaafca: Status 404 returned error can't find the container with id 104ae35098c5292b32909bb0b71b529764cc79a1bbaeb19fbd58e47eaceaafca Oct 30 19:45:57.511702 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.511674 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq"] Oct 30 19:45:57.512005 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.511973 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:45:57.512092 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.512055 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:45:57.512195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:57.512099 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:45:58.113793 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:58.113758 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" event={"ID":"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a","Type":"ContainerStarted","Data":"4e4fb733f02cefdb7a1371f4e73082082dced208f7e00bbdd37b1f4a07f108d5"} Oct 30 19:45:58.115637 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:58.115611 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" event={"ID":"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da","Type":"ContainerStarted","Data":"b4227d7e6667971d2ad18b74ad245a7e33fd49b72354362153b6d8908913cb70"} Oct 30 19:45:58.115806 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:58.115655 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" event={"ID":"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da","Type":"ContainerStarted","Data":"104ae35098c5292b32909bb0b71b529764cc79a1bbaeb19fbd58e47eaceaafca"} Oct 30 19:45:58.117047 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:58.117024 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" event={"ID":"2df1aca0-a819-4417-b85a-f9d2fda67997","Type":"ContainerDied","Data":"c9c5ec4df3e4b9c4b83de5e740e13730801534fe782fc501c0da5789f4216142"} Oct 30 19:45:58.117047 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:58.117042 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d" Oct 30 19:45:58.117217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:58.117062 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9c5ec4df3e4b9c4b83de5e740e13730801534fe782fc501c0da5789f4216142" Oct 30 19:45:58.144359 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:58.144310 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" podStartSLOduration=5.144294348 podStartE2EDuration="5.144294348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:58.142844066 +0000 UTC m=+877.838052057" watchObservedRunningTime="2026-10-30 19:45:58.144294348 +0000 UTC m=+877.839502307" Oct 30 19:45:58.189674 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:58.189624 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" podStartSLOduration=2.189608205 podStartE2EDuration="2.189608205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:45:58.186497212 +0000 UTC m=+877.881705172" watchObservedRunningTime="2026-10-30 19:45:58.189608205 +0000 UTC m=+877.884816162" Oct 30 19:45:58.298238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:58.298195 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:45:58.865475 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:58.865441 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:58.883861 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:58.883836 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:59.041222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:59.041188 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:45:59.042758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:59.042731 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:45:59.042892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:59.042788 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" podUID="27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 19:45:59.124863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:59.124791 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:59.125907 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:59.125890 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:45:59.282169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:45:59.279041 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:00.041200 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:00.041137 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:46:00.041673 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:00.041615 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" podUID="27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 19:46:00.309341 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:00.309252 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:00.459719 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:00.459669 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:00.633821 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:00.633740 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:00.872297 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:00.868498 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:01.040791 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:01.040756 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:46:01.040984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:01.040810 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" podUID="27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 19:46:01.672303 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:01.672207 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:01.930624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:01.930513 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:02.040785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:02.040741 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:46:02.041261 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:02.040796 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" podUID="27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 19:46:02.091674 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:02.091636 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:02.875965 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:02.875928 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:03.042313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:03.042277 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:46:03.042499 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:03.042473 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" podUID="27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 19:46:03.877274 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:03.877231 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:04.041565 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:04.041528 5091 patch_prober.go:29] interesting pod/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Oct 30 19:46:04.041733 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:04.041597 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" podUID="27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Oct 30 19:46:04.114009 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:04.113972 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:05.018562 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:05.018504 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:05.046439 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:05.046407 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:46:05.046970 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:05.046942 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:46:05.047976 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:05.047957 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:46:05.980512 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:05.980475 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:06.991794 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:06.991751 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:07.900896 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:07.900864 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:08.088812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:08.088760 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:08.605467 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:08.605427 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:09.154881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:09.154844 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:09.341179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:09.341035 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:09.600707 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:09.600667 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:10.271632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:10.271579 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:10.660446 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:10.660349 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:11.093016 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:11.092978 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:11.350989 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:11.350897 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:12.044071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:12.043857 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:12.898204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:12.895468 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:13.797921 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:13.797758 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:14.689314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:14.689215 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:15.210857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.210821 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff"] Oct 30 19:46:15.211782 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.211762 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2df1aca0-a819-4417-b85a-f9d2fda67997" containerName="revoke-keys" Oct 30 19:46:15.211855 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.211786 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2df1aca0-a819-4417-b85a-f9d2fda67997" containerName="revoke-keys" Oct 30 19:46:15.211973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.211962 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2df1aca0-a819-4417-b85a-f9d2fda67997" containerName="revoke-keys" Oct 30 19:46:15.216426 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.216405 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.228101 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.228069 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-7a745c-openshift-default-dockercfg-svrhl\"" Oct 30 19:46:15.231765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.231742 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff"] Oct 30 19:46:15.313720 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.313687 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.313889 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.313731 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.313889 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.313799 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582eb068-27ce-4484-8aa0-02d5a409e103-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.313967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.313900 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.314004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.313989 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582eb068-27ce-4484-8aa0-02d5a409e103-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.314095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.314082 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582eb068-27ce-4484-8aa0-02d5a409e103-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.314160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.314124 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.314214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.314142 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v45qc\" (UniqueName: \"kubernetes.io/projected/582eb068-27ce-4484-8aa0-02d5a409e103-kube-api-access-v45qc\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.314256 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.314217 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.415029 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.414991 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582eb068-27ce-4484-8aa0-02d5a409e103-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.415203 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.415068 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.415203 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.415119 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582eb068-27ce-4484-8aa0-02d5a409e103-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.415376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.415222 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582eb068-27ce-4484-8aa0-02d5a409e103-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.415376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.415243 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.415376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.415260 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v45qc\" (UniqueName: \"kubernetes.io/projected/582eb068-27ce-4484-8aa0-02d5a409e103-kube-api-access-v45qc\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.415376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.415338 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.415588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.415402 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.415588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.415447 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.415765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.415740 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.415826 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.415761 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.415996 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.415978 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.416055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.416007 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582eb068-27ce-4484-8aa0-02d5a409e103-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.416055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.416016 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.417451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.417429 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.417660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.417641 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582eb068-27ce-4484-8aa0-02d5a409e103-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.429482 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.429459 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582eb068-27ce-4484-8aa0-02d5a409e103-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.429734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.429716 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v45qc\" (UniqueName: \"kubernetes.io/projected/582eb068-27ce-4484-8aa0-02d5a409e103-kube-api-access-v45qc\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.533699 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.533672 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:15.721654 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.720366 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:15.832619 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:46:15.832590 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod582eb068_27ce_4484_8aa0_02d5a409e103.slice/crio-c8d40908d29717778b6cdd094e98f7322503294a8b20a7938c7493bba35db6e3 WatchSource:0}: Error finding container c8d40908d29717778b6cdd094e98f7322503294a8b20a7938c7493bba35db6e3: Status 404 returned error can't find the container with id c8d40908d29717778b6cdd094e98f7322503294a8b20a7938c7493bba35db6e3 Oct 30 19:46:15.834104 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.834074 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff"] Oct 30 19:46:15.836184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.836042 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:46:15.836184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.836131 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:46:15.836359 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:15.836208 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:46:15.985372 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:46:15.985326 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:46:16.367442 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:16.367394 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" event={"ID":"582eb068-27ce-4484-8aa0-02d5a409e103","Type":"ContainerStarted","Data":"e43fc0b8a7641c479ae06ac9de0200f914c1110e8e2edacfe21feae9918d0671"} Oct 30 19:46:16.367620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:16.367472 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" event={"ID":"582eb068-27ce-4484-8aa0-02d5a409e103","Type":"ContainerStarted","Data":"c8d40908d29717778b6cdd094e98f7322503294a8b20a7938c7493bba35db6e3"} Oct 30 19:46:16.401539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:16.401489 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" podStartSLOduration=1.401475163 podStartE2EDuration="1.401475163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:16.395799629 +0000 UTC m=+896.091007589" watchObservedRunningTime="2026-10-30 19:46:16.401475163 +0000 UTC m=+896.096683127" Oct 30 19:46:16.534429 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:16.534395 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:16.538849 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:16.535326 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:46:16.538849 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:16.535378 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" podUID="582eb068-27ce-4484-8aa0-02d5a409e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 19:46:16.590167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:16.589962 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:17.534662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:17.534627 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:46:17.535077 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:17.534679 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" podUID="582eb068-27ce-4484-8aa0-02d5a409e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 19:46:17.716615 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:17.716540 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:18.541652 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:18.541612 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:46:18.542303 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:18.542268 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" podUID="582eb068-27ce-4484-8aa0-02d5a409e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 19:46:18.882850 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:18.882755 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:19.534975 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:19.534943 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:46:19.535162 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:19.534991 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" podUID="582eb068-27ce-4484-8aa0-02d5a409e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 19:46:19.985365 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:19.985276 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:20.534077 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:20.534020 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:46:20.534311 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:20.534096 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" podUID="582eb068-27ce-4484-8aa0-02d5a409e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 19:46:20.599628 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:20.599594 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:46:20.602459 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:20.602437 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:46:20.606974 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:20.606939 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:21.097019 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:21.096921 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:21.537170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:21.536567 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:46:21.537170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:21.536654 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" podUID="582eb068-27ce-4484-8aa0-02d5a409e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 19:46:21.984225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:21.973375 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m"] Oct 30 19:46:21.993992 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:21.992957 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" Oct 30 19:46:22.000049 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:21.999986 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:46:22.005309 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:22.004131 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:46:22.014594 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:22.014568 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fd2a10-73f0-4652-a684-cc2b206e89a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-07292b-2lm4m\" (UID: \"61fd2a10-73f0-4652-a684-cc2b206e89a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" Oct 30 19:46:22.017322 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:22.017295 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fd2a10-73f0-4652-a684-cc2b206e89a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-07292b-2lm4m\" (UID: \"61fd2a10-73f0-4652-a684-cc2b206e89a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" Oct 30 19:46:22.017596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:22.017574 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn2nb\" (UniqueName: \"kubernetes.io/projected/61fd2a10-73f0-4652-a684-cc2b206e89a0-kube-api-access-zn2nb\") pod \"maas-api-revoke-keys-e2e-api-a-07292b-2lm4m\" (UID: \"61fd2a10-73f0-4652-a684-cc2b206e89a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" Oct 30 19:46:22.032381 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:22.029774 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m"] Oct 30 19:46:22.142197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:22.139070 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fd2a10-73f0-4652-a684-cc2b206e89a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-07292b-2lm4m\" (UID: \"61fd2a10-73f0-4652-a684-cc2b206e89a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" Oct 30 19:46:22.146165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:22.142793 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fd2a10-73f0-4652-a684-cc2b206e89a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-07292b-2lm4m\" (UID: \"61fd2a10-73f0-4652-a684-cc2b206e89a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" Oct 30 19:46:22.162414 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:22.146419 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn2nb\" (UniqueName: \"kubernetes.io/projected/61fd2a10-73f0-4652-a684-cc2b206e89a0-kube-api-access-zn2nb\") pod \"maas-api-revoke-keys-e2e-api-a-07292b-2lm4m\" (UID: \"61fd2a10-73f0-4652-a684-cc2b206e89a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" Oct 30 19:46:22.163094 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:22.144742 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fd2a10-73f0-4652-a684-cc2b206e89a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-07292b-2lm4m\" (UID: \"61fd2a10-73f0-4652-a684-cc2b206e89a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" Oct 30 19:46:22.163094 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:22.144357 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fd2a10-73f0-4652-a684-cc2b206e89a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-07292b-2lm4m\" (UID: \"61fd2a10-73f0-4652-a684-cc2b206e89a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" Oct 30 19:46:22.181201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:22.181114 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn2nb\" (UniqueName: \"kubernetes.io/projected/61fd2a10-73f0-4652-a684-cc2b206e89a0-kube-api-access-zn2nb\") pod \"maas-api-revoke-keys-e2e-api-a-07292b-2lm4m\" (UID: \"61fd2a10-73f0-4652-a684-cc2b206e89a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" Oct 30 19:46:22.345579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:22.345543 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" Oct 30 19:46:22.636220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:22.552075 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:46:22.636220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:22.552133 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" podUID="582eb068-27ce-4484-8aa0-02d5a409e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 19:46:22.636220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:22.625844 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:23.538476 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:23.537867 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m"] Oct 30 19:46:23.549186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:23.545973 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" start-of-body= Oct 30 19:46:23.549186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:23.546058 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" podUID="582eb068-27ce-4484-8aa0-02d5a409e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": dial tcp 10.132.0.110:15021: connect: connection refused" Oct 30 19:46:23.662529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:23.662491 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" event={"ID":"61fd2a10-73f0-4652-a684-cc2b206e89a0","Type":"ContainerStarted","Data":"1c618124a118a78f8fc0a6ff6b2632a853c522c791eaa06904657fdfef5d4fcb"} Oct 30 19:46:24.733133 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:24.733085 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:24.767259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:24.767210 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" event={"ID":"61fd2a10-73f0-4652-a684-cc2b206e89a0","Type":"ContainerStarted","Data":"75db0e52e84f3ab2394e163581ae569cb89baff19cda66c57afab04cc338a02b"} Oct 30 19:46:24.939188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:24.938990 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" podStartSLOduration=3.93896873 podStartE2EDuration="3.93896873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:24.818042865 +0000 UTC m=+904.513250826" watchObservedRunningTime="2026-10-30 19:46:24.93896873 +0000 UTC m=+904.634176689" Oct 30 19:46:25.551322 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:25.551274 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:46:25.551679 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:25.551641 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" podUID="582eb068-27ce-4484-8aa0-02d5a409e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:46:25.833236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:25.833019 5091 generic.go:356] "Generic (PLEG): container finished" podID="61fd2a10-73f0-4652-a684-cc2b206e89a0" containerID="75db0e52e84f3ab2394e163581ae569cb89baff19cda66c57afab04cc338a02b" exitCode=0 Oct 30 19:46:25.833236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:25.833214 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" event={"ID":"61fd2a10-73f0-4652-a684-cc2b206e89a0","Type":"ContainerDied","Data":"75db0e52e84f3ab2394e163581ae569cb89baff19cda66c57afab04cc338a02b"} Oct 30 19:46:25.875202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:25.872493 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.324172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:26.320291 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:46:26.523075 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:26.516232 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:46:26.613218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:26.609289 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:46:26.613218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:26.609352 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" podUID="582eb068-27ce-4484-8aa0-02d5a409e103" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.110:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:46:26.913203 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:26.911051 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:26.925686 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:26.924303 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:46:26.925686 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:26.924360 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" podUID="c63bc1ab-ca04-4273-ae53-2d0923e4900b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:46:27.233662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.231043 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:46:27.325055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.323908 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:46:27.325055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.324349 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:46:27.417934 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.417892 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:46:27.443785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.443498 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:46:27.501828 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.501554 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:27.503248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.503227 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:27.509279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.505983 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:46:27.623739 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.623704 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" Oct 30 19:46:27.705157 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.705101 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61fd2a10-73f0-4652-a684-cc2b206e89a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fd2a10-73f0-4652-a684-cc2b206e89a0-maas-api-openshift-ca\") pod \"61fd2a10-73f0-4652-a684-cc2b206e89a0\" (UID: \"61fd2a10-73f0-4652-a684-cc2b206e89a0\") " Oct 30 19:46:27.705395 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.705370 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61fd2a10-73f0-4652-a684-cc2b206e89a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fd2a10-73f0-4652-a684-cc2b206e89a0-maas-api-xks-ca\") pod \"61fd2a10-73f0-4652-a684-cc2b206e89a0\" (UID: \"61fd2a10-73f0-4652-a684-cc2b206e89a0\") " Oct 30 19:46:27.705467 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.705435 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61fd2a10-73f0-4652-a684-cc2b206e89a0-maas-api-openshift-ca" pod "61fd2a10-73f0-4652-a684-cc2b206e89a0" (UID: "61fd2a10-73f0-4652-a684-cc2b206e89a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:27.705899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.705874 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61fd2a10-73f0-4652-a684-cc2b206e89a0-kube-api-access-zn2nb\" (UniqueName: \"kubernetes.io/projected/61fd2a10-73f0-4652-a684-cc2b206e89a0-kube-api-access-zn2nb\") pod \"61fd2a10-73f0-4652-a684-cc2b206e89a0\" (UID: \"61fd2a10-73f0-4652-a684-cc2b206e89a0\") " Oct 30 19:46:27.706974 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.706948 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fd2a10-73f0-4652-a684-cc2b206e89a0-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:27.707951 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.707927 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fd2a10-73f0-4652-a684-cc2b206e89a0-kube-api-access-zn2nb" pod "61fd2a10-73f0-4652-a684-cc2b206e89a0" (UID: "61fd2a10-73f0-4652-a684-cc2b206e89a0"). InnerVolumeSpecName "kube-api-access-zn2nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:27.708393 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.708370 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61fd2a10-73f0-4652-a684-cc2b206e89a0-maas-api-xks-ca" pod "61fd2a10-73f0-4652-a684-cc2b206e89a0" (UID: "61fd2a10-73f0-4652-a684-cc2b206e89a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:27.792244 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.783786 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:27.823178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.821280 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn2nb\" (UniqueName: \"kubernetes.io/projected/61fd2a10-73f0-4652-a684-cc2b206e89a0-kube-api-access-zn2nb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:27.823178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.821319 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fd2a10-73f0-4652-a684-cc2b206e89a0-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:27.910130 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.910104 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" Oct 30 19:46:27.910325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.910139 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m" event={"ID":"61fd2a10-73f0-4652-a684-cc2b206e89a0","Type":"ContainerDied","Data":"1c618124a118a78f8fc0a6ff6b2632a853c522c791eaa06904657fdfef5d4fcb"} Oct 30 19:46:27.910325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:27.910227 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c618124a118a78f8fc0a6ff6b2632a853c522c791eaa06904657fdfef5d4fcb" Oct 30 19:46:28.670273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:28.670236 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:29.501888 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:29.501851 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:30.248958 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.248921 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk"] Oct 30 19:46:30.249928 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.249905 5091 cpu_manager.go:414] "containerMap: removing container" podUID="61fd2a10-73f0-4652-a684-cc2b206e89a0" containerName="revoke-keys" Oct 30 19:46:30.249973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.249931 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fd2a10-73f0-4652-a684-cc2b206e89a0" containerName="revoke-keys" Oct 30 19:46:30.250080 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.250069 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fd2a10-73f0-4652-a684-cc2b206e89a0" containerName="revoke-keys" Oct 30 19:46:30.255638 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.255609 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.283718 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.283681 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk"] Oct 30 19:46:30.358039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.358000 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9adda6-46b7-42fa-9230-eeb60648a305-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.358249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.358204 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.358377 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.358354 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.358434 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.358401 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.358510 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.358490 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.358568 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.358538 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.358643 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.358628 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwdwv\" (UniqueName: \"kubernetes.io/projected/0c9adda6-46b7-42fa-9230-eeb60648a305-kube-api-access-vwdwv\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.358742 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.358725 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.358781 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.358769 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.422506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.422442 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:30.466522 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.466480 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.466713 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.466576 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.466713 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.466606 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.466713 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.466651 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.466713 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.466688 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.466931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.466740 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwdwv\" (UniqueName: \"kubernetes.io/projected/0c9adda6-46b7-42fa-9230-eeb60648a305-kube-api-access-vwdwv\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.466931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.466817 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.466931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.466859 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.467083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.466973 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9adda6-46b7-42fa-9230-eeb60648a305-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.467751 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.467718 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9adda6-46b7-42fa-9230-eeb60648a305-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.468006 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.467985 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.469389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.468460 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.469389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.468496 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.469389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.468762 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.471678 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.471634 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.472410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.472387 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.495625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.495051 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwdwv\" (UniqueName: \"kubernetes.io/projected/0c9adda6-46b7-42fa-9230-eeb60648a305-kube-api-access-vwdwv\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.499336 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.499269 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.572498 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.572456 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:30.873552 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:46:30.873521 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9adda6_46b7_42fa_9230_eeb60648a305.slice/crio-8f4c09c89b1d307fe96a730540910c7fa88211b296f9aba7c5707b82765953f4 WatchSource:0}: Error finding container 8f4c09c89b1d307fe96a730540910c7fa88211b296f9aba7c5707b82765953f4: Status 404 returned error can't find the container with id 8f4c09c89b1d307fe96a730540910c7fa88211b296f9aba7c5707b82765953f4 Oct 30 19:46:30.876346 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.876319 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk"] Oct 30 19:46:30.876630 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.876593 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:46:30.876827 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.876680 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:46:30.876827 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.876729 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:46:30.934219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:30.934188 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" event={"ID":"0c9adda6-46b7-42fa-9230-eeb60648a305","Type":"ContainerStarted","Data":"8f4c09c89b1d307fe96a730540910c7fa88211b296f9aba7c5707b82765953f4"} Oct 30 19:46:31.536543 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:31.535818 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:31.943522 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:31.943449 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" event={"ID":"0c9adda6-46b7-42fa-9230-eeb60648a305","Type":"ContainerStarted","Data":"f827f1d08677fa255340d862901e564746d6a37b0a147f266f36236b8494933b"} Oct 30 19:46:31.995163 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:31.995066 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" podStartSLOduration=1.9950521079999999 podStartE2EDuration="1.995052108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:46:31.990573339 +0000 UTC m=+911.685781297" watchObservedRunningTime="2026-10-30 19:46:31.995052108 +0000 UTC m=+911.690260066" Oct 30 19:46:32.282831 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:32.282797 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:32.511203 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:32.510430 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:32.587816 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:32.579472 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:32.587816 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:32.581507 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:46:32.587816 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:32.581582 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" podUID="0c9adda6-46b7-42fa-9230-eeb60648a305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 19:46:32.691570 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:32.691206 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:33.546252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:33.544284 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:33.577169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:33.576576 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:46:33.577169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:33.576660 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" podUID="0c9adda6-46b7-42fa-9230-eeb60648a305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 19:46:33.986240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:33.986209 5091 generic.go:356] "Generic (PLEG): container finished" podID="52104831-cd64-4eea-b4ed-a389c06a4c30" containerID="687c8174c37d688c6081946d8b60ec6cddd9486ff6ea25c5cfe7dac623e7df9e" exitCode=0 Oct 30 19:46:33.986673 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:33.986279 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" event={"ID":"52104831-cd64-4eea-b4ed-a389c06a4c30","Type":"ContainerDied","Data":"687c8174c37d688c6081946d8b60ec6cddd9486ff6ea25c5cfe7dac623e7df9e"} Oct 30 19:46:33.986764 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:33.986748 5091 scope.go:122] "RemoveContainer" containerID="687c8174c37d688c6081946d8b60ec6cddd9486ff6ea25c5cfe7dac623e7df9e" Oct 30 19:46:33.993587 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:33.993550 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:46:33.993794 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:33.993641 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:46:33.993794 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:33.993687 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:46:34.465168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:34.461107 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:34.575168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:34.574511 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:46:34.575168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:34.574568 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" podUID="0c9adda6-46b7-42fa-9230-eeb60648a305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 19:46:35.025683 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:35.025643 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" event={"ID":"52104831-cd64-4eea-b4ed-a389c06a4c30","Type":"ContainerStarted","Data":"1cc7d720f9425bffbd5e170bfb0c7b656f9f78989c508edae8185e20dadfc9ad"} Oct 30 19:46:35.378164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:35.375366 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:35.579647 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:35.579605 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:46:35.579931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:35.579897 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" podUID="0c9adda6-46b7-42fa-9230-eeb60648a305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 19:46:35.995171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:35.992541 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:46:35.995171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:35.992670 5091 patch_prober.go:29] interesting pod/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:46:35.995171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:35.992716 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" podUID="52104831-cd64-4eea-b4ed-a389c06a4c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 19:46:36.173711 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:36.169105 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:36.583167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:36.582615 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:46:36.583167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:36.582691 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" podUID="0c9adda6-46b7-42fa-9230-eeb60648a305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 19:46:36.979287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:36.976408 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:37.000173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:36.993555 5091 patch_prober.go:29] interesting pod/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:46:37.000173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:36.993625 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" podUID="52104831-cd64-4eea-b4ed-a389c06a4c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 19:46:37.590169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:37.589570 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body= Oct 30 19:46:37.590169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:37.589632 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" podUID="0c9adda6-46b7-42fa-9230-eeb60648a305" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" Oct 30 19:46:37.762191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:37.759862 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:37.993133 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:37.993038 5091 patch_prober.go:29] interesting pod/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:46:37.993309 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:37.993133 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" podUID="52104831-cd64-4eea-b4ed-a389c06a4c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 19:46:38.713327 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:38.708807 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:38.964627 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:38.964537 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:38.965477 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:38.965449 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:38.973314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:38.973294 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:46:38.992900 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:38.992861 5091 patch_prober.go:29] interesting pod/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:46:38.993095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:38.993075 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" podUID="52104831-cd64-4eea-b4ed-a389c06a4c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 19:46:39.580169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:39.580106 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:39.993434 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:39.993353 5091 patch_prober.go:29] interesting pod/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body= Oct 30 19:46:39.993434 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:39.993403 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" podUID="52104831-cd64-4eea-b4ed-a389c06a4c30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" Oct 30 19:46:40.489111 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:40.489070 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:40.997520 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:40.997490 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:46:40.997960 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:40.997940 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:46:40.998622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:40.998599 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:46:41.372319 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:41.372181 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:42.190385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:42.190289 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:43.014188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:43.007475 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:43.789809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:43.789771 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:44.534554 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:44.534524 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:44.744288 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:44.744251 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:44.941750 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:44.941661 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m"] Oct 30 19:46:44.947624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:44.947594 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-07292b-2lm4m"] Oct 30 19:46:44.975306 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:44.975275 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:45.133493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:45.133459 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:45.474915 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:45.474882 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:45.992611 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:46:45.992570 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:46:46.194304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:46.194220 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:46.444109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:46.444073 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61fd2a10-73f0-4652-a684-cc2b206e89a0" path="/var/lib/kubelet/pods/61fd2a10-73f0-4652-a684-cc2b206e89a0/volumes" Oct 30 19:46:47.011644 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:47.011611 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:47.864491 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:47.864451 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:48.708172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:48.708109 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:49.640243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:49.640177 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:50.492442 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:50.492402 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:51.310595 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:51.310559 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:52.089053 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:52.089015 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:52.267656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:52.267623 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:52.455863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:52.455826 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:52.873418 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:52.873383 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw"] Oct 30 19:46:52.879225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:52.879182 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" Oct 30 19:46:52.887965 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:52.887938 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw"] Oct 30 19:46:52.961426 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:52.961400 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:46:52.961774 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:52.961754 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:46:53.027731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:53.027700 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw\" (UID: \"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" Oct 30 19:46:53.027876 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:53.027804 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw\" (UID: \"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" Oct 30 19:46:53.027876 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:53.027867 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmctn\" (UniqueName: \"kubernetes.io/projected/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-kube-api-access-mmctn\") pod \"maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw\" (UID: \"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" Oct 30 19:46:53.129596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:53.129518 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw\" (UID: \"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" Oct 30 19:46:53.129596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:53.129589 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmctn\" (UniqueName: \"kubernetes.io/projected/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-kube-api-access-mmctn\") pod \"maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw\" (UID: \"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" Oct 30 19:46:53.129966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:53.129725 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw\" (UID: \"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" Oct 30 19:46:53.130290 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:53.130271 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw\" (UID: \"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" Oct 30 19:46:53.132084 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:53.132055 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw\" (UID: \"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" Oct 30 19:46:53.143635 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:53.143612 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmctn\" (UniqueName: \"kubernetes.io/projected/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-kube-api-access-mmctn\") pod \"maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw\" (UID: \"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" Oct 30 19:46:53.200541 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:53.200505 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" Oct 30 19:46:53.268112 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:53.268027 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:53.485296 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:46:53.485263 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db9c1e0_6e59_458e_a2f3_8bbddc5e6851.slice/crio-a6cc1e58a5be516a279e3e785a5aae94e912527fc78db2e7365ed8cbcc6a492a WatchSource:0}: Error finding container a6cc1e58a5be516a279e3e785a5aae94e912527fc78db2e7365ed8cbcc6a492a: Status 404 returned error can't find the container with id a6cc1e58a5be516a279e3e785a5aae94e912527fc78db2e7365ed8cbcc6a492a Oct 30 19:46:53.487869 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:53.487844 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw"] Oct 30 19:46:54.283892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:54.279880 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:54.361343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:54.361313 5091 generic.go:356] "Generic (PLEG): container finished" podID="1db9c1e0-6e59-458e-a2f3-8bbddc5e6851" containerID="4707cb545a69b1f9d2ba2519dfeeb9ac64ce29d6df2e59f3e3eb331e68ab3066" exitCode=0 Oct 30 19:46:54.361495 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:54.361389 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" event={"ID":"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851","Type":"ContainerDied","Data":"4707cb545a69b1f9d2ba2519dfeeb9ac64ce29d6df2e59f3e3eb331e68ab3066"} Oct 30 19:46:54.361495 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:54.361441 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" event={"ID":"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851","Type":"ContainerStarted","Data":"a6cc1e58a5be516a279e3e785a5aae94e912527fc78db2e7365ed8cbcc6a492a"} Oct 30 19:46:55.243414 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:55.243369 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:55.546319 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:55.546295 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" Oct 30 19:46:55.674111 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:55.674079 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-maas-api-xks-ca\") pod \"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851\" (UID: \"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851\") " Oct 30 19:46:55.674325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:55.674214 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-kube-api-access-mmctn\" (UniqueName: \"kubernetes.io/projected/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-kube-api-access-mmctn\") pod \"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851\" (UID: \"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851\") " Oct 30 19:46:55.674325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:55.674242 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-maas-api-openshift-ca\") pod \"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851\" (UID: \"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851\") " Oct 30 19:46:55.674742 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:55.674711 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-maas-api-openshift-ca" pod "1db9c1e0-6e59-458e-a2f3-8bbddc5e6851" (UID: "1db9c1e0-6e59-458e-a2f3-8bbddc5e6851"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:46:55.674883 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:55.674850 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:55.676413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:55.676349 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-maas-api-xks-ca" pod "1db9c1e0-6e59-458e-a2f3-8bbddc5e6851" (UID: "1db9c1e0-6e59-458e-a2f3-8bbddc5e6851"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:46:55.676792 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:55.676767 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-kube-api-access-mmctn" pod "1db9c1e0-6e59-458e-a2f3-8bbddc5e6851" (UID: "1db9c1e0-6e59-458e-a2f3-8bbddc5e6851"). InnerVolumeSpecName "kube-api-access-mmctn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:46:55.776180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:55.776084 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:55.776180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:55.776124 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmctn\" (UniqueName: \"kubernetes.io/projected/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851-kube-api-access-mmctn\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:46:56.173848 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:56.173712 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:56.379327 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:56.379287 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" event={"ID":"1db9c1e0-6e59-458e-a2f3-8bbddc5e6851","Type":"ContainerDied","Data":"a6cc1e58a5be516a279e3e785a5aae94e912527fc78db2e7365ed8cbcc6a492a"} Oct 30 19:46:56.379327 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:56.379324 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw" Oct 30 19:46:56.379553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:56.379351 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6cc1e58a5be516a279e3e785a5aae94e912527fc78db2e7365ed8cbcc6a492a" Oct 30 19:46:57.165188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:57.157302 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:58.068199 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:58.064912 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:58.897846 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:58.897801 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:46:59.773548 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:46:59.773509 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:00.629739 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:00.629700 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:01.551600 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:01.551561 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:02.521588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:02.516110 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:03.138236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:03.138199 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:03.286686 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:03.286645 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:03.456743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:03.456659 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:04.385110 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:04.384637 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:05.369015 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:05.368976 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:06.194171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:06.193774 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:07.128560 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:07.128514 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:08.026689 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:08.026651 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:08.917745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:08.917711 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:09.848770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:09.848736 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:10.680810 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:10.680711 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:11.409780 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:11.409741 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:11.561184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:11.558547 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:12.531171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.530220 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:12.820179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.820090 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-nw4zh-vslhs"] Oct 30 19:47:12.820777 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.820761 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1db9c1e0-6e59-458e-a2f3-8bbddc5e6851" containerName="revoke-keys" Oct 30 19:47:12.820826 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.820778 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db9c1e0-6e59-458e-a2f3-8bbddc5e6851" containerName="revoke-keys" Oct 30 19:47:12.820885 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.820876 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db9c1e0-6e59-458e-a2f3-8bbddc5e6851" containerName="revoke-keys" Oct 30 19:47:12.824610 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.824589 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:47:12.835950 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.835924 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-service-ca-bundle\") pod \"periodic-gathering-nw4zh-vslhs\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:47:12.836129 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.836099 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b552j\" (UniqueName: \"kubernetes.io/projected/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-kube-api-access-b552j\") pod \"periodic-gathering-nw4zh-vslhs\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:47:12.836249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.836181 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-archives-path\") pod \"periodic-gathering-nw4zh-vslhs\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:47:12.836312 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.836261 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-trusted-ca-bundle\") pod \"periodic-gathering-nw4zh-vslhs\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:47:12.837976 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.837955 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-nw4zh-vslhs"] Oct 30 19:47:12.937049 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.937020 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-trusted-ca-bundle\") pod \"periodic-gathering-nw4zh-vslhs\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:47:12.937238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.937095 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-service-ca-bundle\") pod \"periodic-gathering-nw4zh-vslhs\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:47:12.937238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.937220 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b552j\" (UniqueName: \"kubernetes.io/projected/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-kube-api-access-b552j\") pod \"periodic-gathering-nw4zh-vslhs\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:47:12.937347 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.937259 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-archives-path\") pod \"periodic-gathering-nw4zh-vslhs\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:47:12.937579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.937560 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-archives-path\") pod \"periodic-gathering-nw4zh-vslhs\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:47:12.937981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.937957 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-service-ca-bundle\") pod \"periodic-gathering-nw4zh-vslhs\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:47:12.938025 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.937966 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-trusted-ca-bundle\") pod \"periodic-gathering-nw4zh-vslhs\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:47:12.956525 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:12.956494 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b552j\" (UniqueName: \"kubernetes.io/projected/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-kube-api-access-b552j\") pod \"periodic-gathering-nw4zh-vslhs\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:47:13.145549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:13.145467 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:47:13.410538 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:13.410366 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:13.439095 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:47:13.439062 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b2a4b1_f12b_43d4_a6d4_06898dca3b9d.slice/crio-a2f70e007ef1957772dd99f8f483f58323de26ac7f4805599bfc509c251bc128 WatchSource:0}: Error finding container a2f70e007ef1957772dd99f8f483f58323de26ac7f4805599bfc509c251bc128: Status 404 returned error can't find the container with id a2f70e007ef1957772dd99f8f483f58323de26ac7f4805599bfc509c251bc128 Oct 30 19:47:13.439785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:13.439763 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-nw4zh-vslhs"] Oct 30 19:47:13.613461 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:13.613434 5091 generic.go:356] "Generic (PLEG): container finished" podID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" containerID="b4227d7e6667971d2ad18b74ad245a7e33fd49b72354362153b6d8908913cb70" exitCode=0 Oct 30 19:47:13.613903 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:13.613465 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" event={"ID":"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da","Type":"ContainerDied","Data":"b4227d7e6667971d2ad18b74ad245a7e33fd49b72354362153b6d8908913cb70"} Oct 30 19:47:13.614051 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:13.614033 5091 scope.go:122] "RemoveContainer" containerID="b4227d7e6667971d2ad18b74ad245a7e33fd49b72354362153b6d8908913cb70" Oct 30 19:47:13.617216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:13.616034 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:47:13.617216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:13.616124 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:47:13.617216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:13.616194 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:47:13.619732 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:13.619705 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nw4zh-vslhs" event={"ID":"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d","Type":"ContainerStarted","Data":"b13042955905aaaab4ef4039572814a5399d4e6086dcad8ac37cede26fded48d"} Oct 30 19:47:13.619853 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:13.619757 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nw4zh-vslhs" event={"ID":"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d","Type":"ContainerStarted","Data":"a2f70e007ef1957772dd99f8f483f58323de26ac7f4805599bfc509c251bc128"} Oct 30 19:47:13.670247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:13.669864 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-nw4zh-vslhs" podStartSLOduration=1.669844099 podStartE2EDuration="1.669844099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:13.666813833 +0000 UTC m=+953.362021809" watchObservedRunningTime="2026-10-30 19:47:13.669844099 +0000 UTC m=+953.365052052" Oct 30 19:47:14.125684 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:14.125637 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:47:14.307168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:14.303470 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:14.629101 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:14.629022 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" event={"ID":"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da","Type":"ContainerStarted","Data":"4a07390af02149f833ac2f315ca60ca3f0addda1d998022ebdad93354be95238"} Oct 30 19:47:14.876172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:14.869398 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:47:14.876172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:14.869531 5091 patch_prober.go:29] interesting pod/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:47:14.876172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:14.869573 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" podUID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:47:15.337065 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.333878 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:15.435658 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.435625 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9"] Oct 30 19:47:15.440010 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.439994 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" Oct 30 19:47:15.474168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.472933 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:47:15.496198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.490748 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9"] Oct 30 19:47:15.496198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.491630 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:47:15.499929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.499894 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7t2w\" (UniqueName: \"kubernetes.io/projected/18d1899e-042e-4b0d-9815-fdd5d565d1e7-kube-api-access-x7t2w\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9\" (UID: \"18d1899e-042e-4b0d-9815-fdd5d565d1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" Oct 30 19:47:15.500197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.500172 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d1899e-042e-4b0d-9815-fdd5d565d1e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9\" (UID: \"18d1899e-042e-4b0d-9815-fdd5d565d1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" Oct 30 19:47:15.500450 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.500420 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d1899e-042e-4b0d-9815-fdd5d565d1e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9\" (UID: \"18d1899e-042e-4b0d-9815-fdd5d565d1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" Oct 30 19:47:15.601386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.601315 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7t2w\" (UniqueName: \"kubernetes.io/projected/18d1899e-042e-4b0d-9815-fdd5d565d1e7-kube-api-access-x7t2w\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9\" (UID: \"18d1899e-042e-4b0d-9815-fdd5d565d1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" Oct 30 19:47:15.601386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.601373 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d1899e-042e-4b0d-9815-fdd5d565d1e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9\" (UID: \"18d1899e-042e-4b0d-9815-fdd5d565d1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" Oct 30 19:47:15.601579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.601474 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d1899e-042e-4b0d-9815-fdd5d565d1e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9\" (UID: \"18d1899e-042e-4b0d-9815-fdd5d565d1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" Oct 30 19:47:15.602049 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.602023 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d1899e-042e-4b0d-9815-fdd5d565d1e7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9\" (UID: \"18d1899e-042e-4b0d-9815-fdd5d565d1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" Oct 30 19:47:15.603691 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.603668 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d1899e-042e-4b0d-9815-fdd5d565d1e7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9\" (UID: \"18d1899e-042e-4b0d-9815-fdd5d565d1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" Oct 30 19:47:15.616942 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.616920 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7t2w\" (UniqueName: \"kubernetes.io/projected/18d1899e-042e-4b0d-9815-fdd5d565d1e7-kube-api-access-x7t2w\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9\" (UID: \"18d1899e-042e-4b0d-9815-fdd5d565d1e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" Oct 30 19:47:15.805205 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.805173 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" Oct 30 19:47:15.868225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.867576 5091 patch_prober.go:29] interesting pod/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:47:15.868225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:15.867639 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" podUID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:47:16.020880 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:47:16.020838 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:47:16.127343 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:47:16.127309 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d1899e_042e_4b0d_9815_fdd5d565d1e7.slice/crio-e6421be7ca9a6f12463420b85c262e7a32fb5e39f54e72dba1144e3da8ae1d89 WatchSource:0}: Error finding container e6421be7ca9a6f12463420b85c262e7a32fb5e39f54e72dba1144e3da8ae1d89: Status 404 returned error can't find the container with id e6421be7ca9a6f12463420b85c262e7a32fb5e39f54e72dba1144e3da8ae1d89 Oct 30 19:47:16.128493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:16.128469 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9"] Oct 30 19:47:16.362510 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:16.362461 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:16.655217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:16.647073 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" event={"ID":"18d1899e-042e-4b0d-9815-fdd5d565d1e7","Type":"ContainerStarted","Data":"61dbd4c26185be9a20036a9ce58d702f1b554c9ec819693f03947678a2403409"} Oct 30 19:47:16.655217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:16.647129 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" event={"ID":"18d1899e-042e-4b0d-9815-fdd5d565d1e7","Type":"ContainerStarted","Data":"e6421be7ca9a6f12463420b85c262e7a32fb5e39f54e72dba1144e3da8ae1d89"} Oct 30 19:47:16.860483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:16.860447 5091 patch_prober.go:29] interesting pod/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:47:16.861073 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:16.861043 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" podUID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:47:17.182168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:17.182116 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:17.567213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:17.537922 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:17.596838 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:17.596764 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" podStartSLOduration=2.596742845 podStartE2EDuration="2.596742845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:16.682382537 +0000 UTC m=+956.377590495" watchObservedRunningTime="2026-10-30 19:47:17.596742845 +0000 UTC m=+957.291950805" Oct 30 19:47:17.597362 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:17.597333 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw"] Oct 30 19:47:17.602863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:17.602833 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-6ac47d-p2ccw"] Oct 30 19:47:17.757168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:17.751440 5091 generic.go:356] "Generic (PLEG): container finished" podID="18d1899e-042e-4b0d-9815-fdd5d565d1e7" containerID="61dbd4c26185be9a20036a9ce58d702f1b554c9ec819693f03947678a2403409" exitCode=0 Oct 30 19:47:17.757168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:17.751634 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" event={"ID":"18d1899e-042e-4b0d-9815-fdd5d565d1e7","Type":"ContainerDied","Data":"61dbd4c26185be9a20036a9ce58d702f1b554c9ec819693f03947678a2403409"} Oct 30 19:47:17.775170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:17.768467 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:17.862618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:17.862531 5091 patch_prober.go:29] interesting pod/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:47:17.863217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:17.863185 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" podUID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:47:18.293959 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:18.293921 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:18.442249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:18.442212 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db9c1e0-6e59-458e-a2f3-8bbddc5e6851" path="/var/lib/kubelet/pods/1db9c1e0-6e59-458e-a2f3-8bbddc5e6851/volumes" Oct 30 19:47:18.863046 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:18.862260 5091 patch_prober.go:29] interesting pod/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:47:18.863046 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:18.862318 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" podUID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:47:18.871663 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:18.871521 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:19.125451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.125424 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" Oct 30 19:47:19.226518 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.226487 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18d1899e-042e-4b0d-9815-fdd5d565d1e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d1899e-042e-4b0d-9815-fdd5d565d1e7-maas-api-xks-ca\") pod \"18d1899e-042e-4b0d-9815-fdd5d565d1e7\" (UID: \"18d1899e-042e-4b0d-9815-fdd5d565d1e7\") " Oct 30 19:47:19.226682 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.226577 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18d1899e-042e-4b0d-9815-fdd5d565d1e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d1899e-042e-4b0d-9815-fdd5d565d1e7-maas-api-openshift-ca\") pod \"18d1899e-042e-4b0d-9815-fdd5d565d1e7\" (UID: \"18d1899e-042e-4b0d-9815-fdd5d565d1e7\") " Oct 30 19:47:19.226783 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.226766 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18d1899e-042e-4b0d-9815-fdd5d565d1e7-kube-api-access-x7t2w\" (UniqueName: \"kubernetes.io/projected/18d1899e-042e-4b0d-9815-fdd5d565d1e7-kube-api-access-x7t2w\") pod \"18d1899e-042e-4b0d-9815-fdd5d565d1e7\" (UID: \"18d1899e-042e-4b0d-9815-fdd5d565d1e7\") " Oct 30 19:47:19.226981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.226961 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18d1899e-042e-4b0d-9815-fdd5d565d1e7-maas-api-openshift-ca" pod "18d1899e-042e-4b0d-9815-fdd5d565d1e7" (UID: "18d1899e-042e-4b0d-9815-fdd5d565d1e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:19.227273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.227248 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18d1899e-042e-4b0d-9815-fdd5d565d1e7-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:19.228921 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.228896 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18d1899e-042e-4b0d-9815-fdd5d565d1e7-maas-api-xks-ca" pod "18d1899e-042e-4b0d-9815-fdd5d565d1e7" (UID: "18d1899e-042e-4b0d-9815-fdd5d565d1e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:19.229015 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.228959 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18d1899e-042e-4b0d-9815-fdd5d565d1e7-kube-api-access-x7t2w" pod "18d1899e-042e-4b0d-9815-fdd5d565d1e7" (UID: "18d1899e-042e-4b0d-9815-fdd5d565d1e7"). InnerVolumeSpecName "kube-api-access-x7t2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:19.328162 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.328107 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7t2w\" (UniqueName: \"kubernetes.io/projected/18d1899e-042e-4b0d-9815-fdd5d565d1e7-kube-api-access-x7t2w\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:19.328162 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.328140 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18d1899e-042e-4b0d-9815-fdd5d565d1e7-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:19.797930 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.797901 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" Oct 30 19:47:19.798157 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.798112 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9" event={"ID":"18d1899e-042e-4b0d-9815-fdd5d565d1e7","Type":"ContainerDied","Data":"e6421be7ca9a6f12463420b85c262e7a32fb5e39f54e72dba1144e3da8ae1d89"} Oct 30 19:47:19.798261 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.798249 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6421be7ca9a6f12463420b85c262e7a32fb5e39f54e72dba1144e3da8ae1d89" Oct 30 19:47:19.865171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.860499 5091 patch_prober.go:29] interesting pod/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body= Oct 30 19:47:19.865171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.860572 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" podUID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" Oct 30 19:47:19.908134 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:19.908099 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:20.784898 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:20.784821 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:20.838502 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:20.838468 5091 scope.go:122] "RemoveContainer" containerID="f5621783477ffc2c474780bf33e695b6bbb35025506eab5410695fd0da603f3a" Oct 30 19:47:20.848696 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:20.848678 5091 scope.go:122] "RemoveContainer" containerID="f046910eac154f85263424987bdf5e29e45102cf28dc379b0f01f23a33d1b802" Oct 30 19:47:20.858134 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:20.858105 5091 scope.go:122] "RemoveContainer" containerID="58d105fb956464b97eaf5120956b753bc27d9f770c09ad72ddd9017a71b62e99" Oct 30 19:47:20.865463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:20.865440 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:47:20.865899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:20.865880 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:47:20.866588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:20.866569 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:47:20.868306 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:20.868291 5091 scope.go:122] "RemoveContainer" containerID="b4936a27ae9c0f07e3ade20449226484147072d19395053391297f1bd5c22007" Oct 30 19:47:20.878545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:20.878525 5091 scope.go:122] "RemoveContainer" containerID="4454abf83f27e674e129398ce97db53c9bd0ebc7eee229b0636b68550ec1ee39" Oct 30 19:47:22.111112 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:22.111075 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:23.065172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:23.061611 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:24.080339 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:24.079804 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:24.715383 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:24.715346 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:24.855428 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:24.855394 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:25.001248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:25.000330 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq"] Oct 30 19:47:25.001248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:25.000564 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" podUID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" containerName="istio-proxy" containerID="cri-o://4a07390af02149f833ac2f315ca60ca3f0addda1d998022ebdad93354be95238" gracePeriod=30 Oct 30 19:47:25.008890 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:25.008761 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:25.164167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:25.163432 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:25.696591 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:25.696555 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:25.948892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:25.948813 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:26.240980 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:26.240904 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:26.776391 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:26.776357 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f"] Oct 30 19:47:26.777027 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:26.777010 5091 cpu_manager.go:414] "containerMap: removing container" podUID="18d1899e-042e-4b0d-9815-fdd5d565d1e7" containerName="revoke-keys" Oct 30 19:47:26.777075 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:26.777030 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18d1899e-042e-4b0d-9815-fdd5d565d1e7" containerName="revoke-keys" Oct 30 19:47:26.777229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:26.777215 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="18d1899e-042e-4b0d-9815-fdd5d565d1e7" containerName="revoke-keys" Oct 30 19:47:26.782772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:26.782745 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" Oct 30 19:47:26.788730 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:26.788705 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:47:26.789097 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:26.789075 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f"] Oct 30 19:47:26.790823 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:26.790796 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:47:26.910843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:26.910809 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6939c6cb-m654f\" (UID: \"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" Oct 30 19:47:26.911024 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:26.910976 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6939c6cb-m654f\" (UID: \"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" Oct 30 19:47:26.911189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:26.911170 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtshr\" (UniqueName: \"kubernetes.io/projected/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-kube-api-access-vtshr\") pod \"maas-api-revoke-sub-6939c6cb-m654f\" (UID: \"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" Oct 30 19:47:27.012619 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:27.012587 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6939c6cb-m654f\" (UID: \"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" Oct 30 19:47:27.012814 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:27.012709 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtshr\" (UniqueName: \"kubernetes.io/projected/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-kube-api-access-vtshr\") pod \"maas-api-revoke-sub-6939c6cb-m654f\" (UID: \"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" Oct 30 19:47:27.012886 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:27.012830 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6939c6cb-m654f\" (UID: \"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" Oct 30 19:47:27.013624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:27.013593 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6939c6cb-m654f\" (UID: \"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" Oct 30 19:47:27.014944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:27.014918 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6939c6cb-m654f\" (UID: \"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" Oct 30 19:47:27.026946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:27.026865 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtshr\" (UniqueName: \"kubernetes.io/projected/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-kube-api-access-vtshr\") pod \"maas-api-revoke-sub-6939c6cb-m654f\" (UID: \"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" Oct 30 19:47:27.092170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:27.084711 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:27.108170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:27.107397 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" Oct 30 19:47:27.412572 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:47:27.412539 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64cb85bc_70a8_4e3a_b28b_1d2a79c825d1.slice/crio-a7771c3cd41e549d9112669545cef04353690100a0fa0582e121709e988e54cb WatchSource:0}: Error finding container a7771c3cd41e549d9112669545cef04353690100a0fa0582e121709e988e54cb: Status 404 returned error can't find the container with id a7771c3cd41e549d9112669545cef04353690100a0fa0582e121709e988e54cb Oct 30 19:47:27.412984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:27.412940 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f"] Oct 30 19:47:27.922505 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:27.922470 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" event={"ID":"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1","Type":"ContainerStarted","Data":"25e6f139031abf46a5ae4a6b0059f48cfbab225f02bb0ae47c6239881042f7e6"} Oct 30 19:47:27.922671 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:27.922536 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" event={"ID":"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1","Type":"ContainerStarted","Data":"a7771c3cd41e549d9112669545cef04353690100a0fa0582e121709e988e54cb"} Oct 30 19:47:27.961857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:27.951004 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" podStartSLOduration=1.9509842210000001 podStartE2EDuration="1.950984221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:27.945624047 +0000 UTC m=+967.640832006" watchObservedRunningTime="2026-10-30 19:47:27.950984221 +0000 UTC m=+967.646192179" Oct 30 19:47:27.985208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:27.982434 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:28.832169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:28.832108 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:28.931946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:28.931912 5091 generic.go:356] "Generic (PLEG): container finished" podID="64cb85bc-70a8-4e3a-b28b-1d2a79c825d1" containerID="25e6f139031abf46a5ae4a6b0059f48cfbab225f02bb0ae47c6239881042f7e6" exitCode=0 Oct 30 19:47:28.932104 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:28.931986 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" event={"ID":"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1","Type":"ContainerDied","Data":"25e6f139031abf46a5ae4a6b0059f48cfbab225f02bb0ae47c6239881042f7e6"} Oct 30 19:47:29.726169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:29.724341 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:30.138704 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.138677 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" Oct 30 19:47:30.265421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.265329 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-maas-api-openshift-ca\") pod \"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1\" (UID: \"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1\") " Oct 30 19:47:30.265421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.265417 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-maas-api-xks-ca\") pod \"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1\" (UID: \"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1\") " Oct 30 19:47:30.265723 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.265703 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-kube-api-access-vtshr\" (UniqueName: \"kubernetes.io/projected/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-kube-api-access-vtshr\") pod \"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1\" (UID: \"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1\") " Oct 30 19:47:30.266994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.266950 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-maas-api-openshift-ca" pod "64cb85bc-70a8-4e3a-b28b-1d2a79c825d1" (UID: "64cb85bc-70a8-4e3a-b28b-1d2a79c825d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:30.269361 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.269328 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-kube-api-access-vtshr" pod "64cb85bc-70a8-4e3a-b28b-1d2a79c825d1" (UID: "64cb85bc-70a8-4e3a-b28b-1d2a79c825d1"). InnerVolumeSpecName "kube-api-access-vtshr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:30.272608 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.272581 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-maas-api-xks-ca" pod "64cb85bc-70a8-4e3a-b28b-1d2a79c825d1" (UID: "64cb85bc-70a8-4e3a-b28b-1d2a79c825d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:30.296954 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.296929 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:47:30.368008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.367967 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtshr\" (UniqueName: \"kubernetes.io/projected/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-kube-api-access-vtshr\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:30.368205 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.368012 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:30.368205 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.368030 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:30.469039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.469008 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-workload-certs\") pod \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " Oct 30 19:47:30.469221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.469077 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-podinfo\") pod \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " Oct 30 19:47:30.469221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.469111 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-credential-socket\") pod \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " Oct 30 19:47:30.469367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.469240 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-workload-socket\") pod \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " Oct 30 19:47:30.469367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.469316 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-kube-api-access-nbk8w\" (UniqueName: \"kubernetes.io/projected/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-kube-api-access-nbk8w\") pod \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " Oct 30 19:47:30.469367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.469326 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-workload-certs" pod "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" (UID: "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:30.469524 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.469412 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-credential-socket" pod "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" (UID: "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:30.469524 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.469504 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-data\") pod \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " Oct 30 19:47:30.469524 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.469513 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-workload-socket" pod "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" (UID: "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:30.469687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.469564 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-token\" (UniqueName: \"kubernetes.io/projected/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-token\") pod \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " Oct 30 19:47:30.469687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.469612 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istiod-ca-cert\") pod \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " Oct 30 19:47:30.469687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.469662 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-envoy\") pod \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\" (UID: \"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da\") " Oct 30 19:47:30.469859 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.469749 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-data" pod "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" (UID: "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:30.470449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.470055 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istiod-ca-cert" pod "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" (UID: "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:30.470449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.470355 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:30.470449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.470376 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:30.470449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.470390 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:30.470449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.470405 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:30.470449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.470420 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:30.471474 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.471454 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-podinfo" pod "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" (UID: "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:47:30.472112 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.472084 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-token" pod "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" (UID: "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:30.472227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.472175 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-envoy" pod "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" (UID: "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:47:30.472282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.472211 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-kube-api-access-nbk8w" pod "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" (UID: "e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da"). InnerVolumeSpecName "kube-api-access-nbk8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:30.481647 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.481570 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:30.571807 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.571684 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbk8w\" (UniqueName: \"kubernetes.io/projected/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-kube-api-access-nbk8w\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:30.571807 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.571716 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:30.571807 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.571726 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:30.571807 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.571736 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:30.771043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.771006 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:30.947371 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.947279 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" Oct 30 19:47:30.947371 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.947302 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f" event={"ID":"64cb85bc-70a8-4e3a-b28b-1d2a79c825d1","Type":"ContainerDied","Data":"a7771c3cd41e549d9112669545cef04353690100a0fa0582e121709e988e54cb"} Oct 30 19:47:30.947371 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.947362 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7771c3cd41e549d9112669545cef04353690100a0fa0582e121709e988e54cb" Oct 30 19:47:30.949324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.949292 5091 generic.go:356] "Generic (PLEG): container finished" podID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" containerID="4a07390af02149f833ac2f315ca60ca3f0addda1d998022ebdad93354be95238" exitCode=0 Oct 30 19:47:30.949472 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.949341 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" event={"ID":"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da","Type":"ContainerDied","Data":"4a07390af02149f833ac2f315ca60ca3f0addda1d998022ebdad93354be95238"} Oct 30 19:47:30.949472 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.949353 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" Oct 30 19:47:30.949472 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.949403 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq" event={"ID":"e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da","Type":"ContainerDied","Data":"104ae35098c5292b32909bb0b71b529764cc79a1bbaeb19fbd58e47eaceaafca"} Oct 30 19:47:30.949472 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.949431 5091 scope.go:122] "RemoveContainer" containerID="4a07390af02149f833ac2f315ca60ca3f0addda1d998022ebdad93354be95238" Oct 30 19:47:30.967252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.967226 5091 scope.go:122] "RemoveContainer" containerID="b4227d7e6667971d2ad18b74ad245a7e33fd49b72354362153b6d8908913cb70" Oct 30 19:47:30.977847 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.977822 5091 scope.go:122] "RemoveContainer" containerID="4a07390af02149f833ac2f315ca60ca3f0addda1d998022ebdad93354be95238" Oct 30 19:47:30.978197 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:47:30.978173 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a07390af02149f833ac2f315ca60ca3f0addda1d998022ebdad93354be95238\": container with ID starting with 4a07390af02149f833ac2f315ca60ca3f0addda1d998022ebdad93354be95238 not found: ID does not exist" containerID="4a07390af02149f833ac2f315ca60ca3f0addda1d998022ebdad93354be95238" Oct 30 19:47:30.978290 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.978207 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a07390af02149f833ac2f315ca60ca3f0addda1d998022ebdad93354be95238"} err="failed to get container status \"4a07390af02149f833ac2f315ca60ca3f0addda1d998022ebdad93354be95238\": rpc error: code = NotFound desc = could not find container \"4a07390af02149f833ac2f315ca60ca3f0addda1d998022ebdad93354be95238\": container with ID starting with 4a07390af02149f833ac2f315ca60ca3f0addda1d998022ebdad93354be95238 not found: ID does not exist" Oct 30 19:47:30.978290 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.978233 5091 scope.go:122] "RemoveContainer" containerID="b4227d7e6667971d2ad18b74ad245a7e33fd49b72354362153b6d8908913cb70" Oct 30 19:47:30.978519 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:47:30.978491 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4227d7e6667971d2ad18b74ad245a7e33fd49b72354362153b6d8908913cb70\": container with ID starting with b4227d7e6667971d2ad18b74ad245a7e33fd49b72354362153b6d8908913cb70 not found: ID does not exist" containerID="b4227d7e6667971d2ad18b74ad245a7e33fd49b72354362153b6d8908913cb70" Oct 30 19:47:30.978581 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.978530 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4227d7e6667971d2ad18b74ad245a7e33fd49b72354362153b6d8908913cb70"} err="failed to get container status \"b4227d7e6667971d2ad18b74ad245a7e33fd49b72354362153b6d8908913cb70\": rpc error: code = NotFound desc = could not find container \"b4227d7e6667971d2ad18b74ad245a7e33fd49b72354362153b6d8908913cb70\": container with ID starting with b4227d7e6667971d2ad18b74ad245a7e33fd49b72354362153b6d8908913cb70 not found: ID does not exist" Oct 30 19:47:30.989183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.989115 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq"] Oct 30 19:47:30.996514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:30.996488 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-6ac47d-openshift-default-7f95bfc97-4w6bq"] Oct 30 19:47:31.284184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:31.280187 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:31.430920 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:31.430800 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:31.640949 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:31.640804 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:32.442048 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.442015 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" path="/var/lib/kubelet/pods/e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da/volumes" Oct 30 19:47:32.503610 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.503577 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw"] Oct 30 19:47:32.504182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.504169 5091 cpu_manager.go:414] "containerMap: removing container" podUID="64cb85bc-70a8-4e3a-b28b-1d2a79c825d1" containerName="revoke-keys" Oct 30 19:47:32.504243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.504183 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64cb85bc-70a8-4e3a-b28b-1d2a79c825d1" containerName="revoke-keys" Oct 30 19:47:32.504243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.504212 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" containerName="istio-proxy" Oct 30 19:47:32.504243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.504218 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" containerName="istio-proxy" Oct 30 19:47:32.504243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.504243 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" containerName="istio-proxy" Oct 30 19:47:32.504365 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.504249 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" containerName="istio-proxy" Oct 30 19:47:32.504365 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.504337 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" containerName="istio-proxy" Oct 30 19:47:32.504365 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.504350 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7f70f61-e4ea-4f14-bfdc-92a0ad0ac9da" containerName="istio-proxy" Oct 30 19:47:32.504365 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.504360 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="64cb85bc-70a8-4e3a-b28b-1d2a79c825d1" containerName="revoke-keys" Oct 30 19:47:32.509660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.509637 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" Oct 30 19:47:32.515581 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.515559 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:47:32.515666 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.515605 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:47:32.517556 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.517535 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw"] Oct 30 19:47:32.578821 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.578132 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:32.616131 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.616087 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e3138a2-sjcrw\" (UID: \"48b8d321-8d7b-469f-994b-12f8a3b4f4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" Oct 30 19:47:32.616342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.616175 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q569q\" (UniqueName: \"kubernetes.io/projected/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-kube-api-access-q569q\") pod \"maas-api-revoke-sub-0e3138a2-sjcrw\" (UID: \"48b8d321-8d7b-469f-994b-12f8a3b4f4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" Oct 30 19:47:32.616342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.616209 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e3138a2-sjcrw\" (UID: \"48b8d321-8d7b-469f-994b-12f8a3b4f4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" Oct 30 19:47:32.718514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.718428 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e3138a2-sjcrw\" (UID: \"48b8d321-8d7b-469f-994b-12f8a3b4f4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" Oct 30 19:47:32.718514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.718486 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q569q\" (UniqueName: \"kubernetes.io/projected/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-kube-api-access-q569q\") pod \"maas-api-revoke-sub-0e3138a2-sjcrw\" (UID: \"48b8d321-8d7b-469f-994b-12f8a3b4f4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" Oct 30 19:47:32.718738 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.718514 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e3138a2-sjcrw\" (UID: \"48b8d321-8d7b-469f-994b-12f8a3b4f4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" Oct 30 19:47:32.719267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.719247 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e3138a2-sjcrw\" (UID: \"48b8d321-8d7b-469f-994b-12f8a3b4f4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" Oct 30 19:47:32.720857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.720839 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e3138a2-sjcrw\" (UID: \"48b8d321-8d7b-469f-994b-12f8a3b4f4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" Oct 30 19:47:32.742548 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.742522 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q569q\" (UniqueName: \"kubernetes.io/projected/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-kube-api-access-q569q\") pod \"maas-api-revoke-sub-0e3138a2-sjcrw\" (UID: \"48b8d321-8d7b-469f-994b-12f8a3b4f4b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" Oct 30 19:47:32.829878 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:32.829848 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" Oct 30 19:47:33.132828 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:47:33.132799 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b8d321_8d7b_469f_994b_12f8a3b4f4b4.slice/crio-4715c3fcbe5efc255b7d3b994e988322ae45159fe42b4e70a79ebfac23c776db WatchSource:0}: Error finding container 4715c3fcbe5efc255b7d3b994e988322ae45159fe42b4e70a79ebfac23c776db: Status 404 returned error can't find the container with id 4715c3fcbe5efc255b7d3b994e988322ae45159fe42b4e70a79ebfac23c776db Oct 30 19:47:33.133865 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:33.133841 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw"] Oct 30 19:47:33.597734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:33.597697 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:33.983287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:33.983246 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" event={"ID":"48b8d321-8d7b-469f-994b-12f8a3b4f4b4","Type":"ContainerStarted","Data":"c5290dbe8c17452884db4c94fc0d0be5603268d8cb190c74ea8e5a438c5f75b3"} Oct 30 19:47:33.983488 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:33.983316 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" event={"ID":"48b8d321-8d7b-469f-994b-12f8a3b4f4b4","Type":"ContainerStarted","Data":"4715c3fcbe5efc255b7d3b994e988322ae45159fe42b4e70a79ebfac23c776db"} Oct 30 19:47:34.007598 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:34.007550 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" podStartSLOduration=2.007537457 podStartE2EDuration="2.007537457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:34.007291348 +0000 UTC m=+973.702499316" watchObservedRunningTime="2026-10-30 19:47:34.007537457 +0000 UTC m=+973.702745414" Oct 30 19:47:34.301049 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:34.301013 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:34.450386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:34.450350 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:34.677384 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:34.677301 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:34.994291 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:34.994199 5091 generic.go:356] "Generic (PLEG): container finished" podID="48b8d321-8d7b-469f-994b-12f8a3b4f4b4" containerID="c5290dbe8c17452884db4c94fc0d0be5603268d8cb190c74ea8e5a438c5f75b3" exitCode=0 Oct 30 19:47:34.994291 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:34.994279 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" event={"ID":"48b8d321-8d7b-469f-994b-12f8a3b4f4b4","Type":"ContainerDied","Data":"c5290dbe8c17452884db4c94fc0d0be5603268d8cb190c74ea8e5a438c5f75b3"} Oct 30 19:47:35.657416 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:35.657386 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:35.875214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:35.875170 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:36.052658 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:36.052599 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:36.244361 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:36.244339 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" Oct 30 19:47:36.291473 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:36.291437 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-maas-api-openshift-ca\") pod \"48b8d321-8d7b-469f-994b-12f8a3b4f4b4\" (UID: \"48b8d321-8d7b-469f-994b-12f8a3b4f4b4\") " Oct 30 19:47:36.292315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:36.291495 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-maas-api-xks-ca\") pod \"48b8d321-8d7b-469f-994b-12f8a3b4f4b4\" (UID: \"48b8d321-8d7b-469f-994b-12f8a3b4f4b4\") " Oct 30 19:47:36.292315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:36.291543 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-kube-api-access-q569q\" (UniqueName: \"kubernetes.io/projected/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-kube-api-access-q569q\") pod \"48b8d321-8d7b-469f-994b-12f8a3b4f4b4\" (UID: \"48b8d321-8d7b-469f-994b-12f8a3b4f4b4\") " Oct 30 19:47:36.292315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:36.291900 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-maas-api-openshift-ca" pod "48b8d321-8d7b-469f-994b-12f8a3b4f4b4" (UID: "48b8d321-8d7b-469f-994b-12f8a3b4f4b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:47:36.292315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:36.292277 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:36.294860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:36.294821 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-kube-api-access-q569q" pod "48b8d321-8d7b-469f-994b-12f8a3b4f4b4" (UID: "48b8d321-8d7b-469f-994b-12f8a3b4f4b4"). InnerVolumeSpecName "kube-api-access-q569q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:47:36.295648 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:36.295614 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-maas-api-xks-ca" pod "48b8d321-8d7b-469f-994b-12f8a3b4f4b4" (UID: "48b8d321-8d7b-469f-994b-12f8a3b4f4b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:47:36.394291 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:36.394239 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:36.394291 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:36.394293 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q569q\" (UniqueName: \"kubernetes.io/projected/48b8d321-8d7b-469f-994b-12f8a3b4f4b4-kube-api-access-q569q\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:47:36.962194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:36.960825 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:37.059414 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:37.059387 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" Oct 30 19:47:37.059579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:37.059417 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw" event={"ID":"48b8d321-8d7b-469f-994b-12f8a3b4f4b4","Type":"ContainerDied","Data":"4715c3fcbe5efc255b7d3b994e988322ae45159fe42b4e70a79ebfac23c776db"} Oct 30 19:47:37.059579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:37.059467 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4715c3fcbe5efc255b7d3b994e988322ae45159fe42b4e70a79ebfac23c776db" Oct 30 19:47:37.922171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:37.922066 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:38.604956 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:38.604919 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:38.866284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:38.861848 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:39.000910 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:39.000878 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9"] Oct 30 19:47:39.004652 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:39.004627 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-xknf9"] Oct 30 19:47:39.033707 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:39.033677 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps"] Oct 30 19:47:39.044739 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:39.044706 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1e4bc83d-p4qps"] Oct 30 19:47:39.050239 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:39.050206 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:39.205276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:39.205166 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:40.228181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:40.224624 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:40.385590 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:40.385550 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:40.445273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:40.445239 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18d1899e-042e-4b0d-9815-fdd5d565d1e7" path="/var/lib/kubelet/pods/18d1899e-042e-4b0d-9815-fdd5d565d1e7/volumes" Oct 30 19:47:40.445905 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:40.445876 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c" path="/var/lib/kubelet/pods/5eb7bab1-6d8a-40ed-8c1a-2fca2afd4a8c/volumes" Oct 30 19:47:40.535491 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:40.535460 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:41.290025 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:41.289987 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:42.212426 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:42.212384 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:42.825540 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:42.825505 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:43.267756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:43.267662 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:43.456170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:43.452946 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:44.613008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:44.612975 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:45.433022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.432988 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc"] Oct 30 19:47:45.434018 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.433996 5091 cpu_manager.go:414] "containerMap: removing container" podUID="48b8d321-8d7b-469f-994b-12f8a3b4f4b4" containerName="revoke-keys" Oct 30 19:47:45.434086 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.434022 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b8d321-8d7b-469f-994b-12f8a3b4f4b4" containerName="revoke-keys" Oct 30 19:47:45.434242 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.434227 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b8d321-8d7b-469f-994b-12f8a3b4f4b4" containerName="revoke-keys" Oct 30 19:47:45.440230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.440206 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.454024 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.453574 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc"] Oct 30 19:47:45.559632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.559598 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.559847 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.559823 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.559964 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.559946 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.560078 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.560063 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.560158 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.560127 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.560226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.560173 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.560283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.560240 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.560340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.560326 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghgt7\" (UniqueName: \"kubernetes.io/projected/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-kube-api-access-ghgt7\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.560448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.560434 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.667167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.667099 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.667637 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.667243 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.667637 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.667329 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.667637 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.667373 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.667637 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.667401 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.667637 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.667455 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.667637 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.667532 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghgt7\" (UniqueName: \"kubernetes.io/projected/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-kube-api-access-ghgt7\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.667637 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.667620 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.668568 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.667670 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.713625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.668618 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istiod-ca-cert\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.713625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.669331 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-credential-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.713625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.669569 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-workload-certs\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.713625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.695229 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-data\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.713625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.670137 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-workload-socket\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.713625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.697415 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:45.713625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.713480 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-token\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.714415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.714359 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-envoy\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.717938 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.717428 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-podinfo\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.720912 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.720849 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghgt7\" (UniqueName: \"kubernetes.io/projected/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-kube-api-access-ghgt7\") pod \"e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.765407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:45.765379 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:45.976612 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:47:45.976529 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:47:46.061902 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:47:46.061865 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dcda2f6_599d_4eea_b8d7_766b9a95cc2e.slice/crio-cf856da271bc49512ee73cfac24ec3234c778f0514260900d309c86bba608ebc WatchSource:0}: Error finding container cf856da271bc49512ee73cfac24ec3234c778f0514260900d309c86bba608ebc: Status 404 returned error can't find the container with id cf856da271bc49512ee73cfac24ec3234c778f0514260900d309c86bba608ebc Oct 30 19:47:46.063261 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:46.063234 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc"] Oct 30 19:47:46.064499 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:46.064465 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:47:46.064620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:46.064537 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:47:46.064620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:46.064586 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:47:46.178190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:46.178122 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" event={"ID":"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e","Type":"ContainerStarted","Data":"3628ea82aea5ca2b1f72f994b74b3cb30e8f5397dace04bc7a3080a56dd28af4"} Oct 30 19:47:46.178353 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:46.178218 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" event={"ID":"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e","Type":"ContainerStarted","Data":"cf856da271bc49512ee73cfac24ec3234c778f0514260900d309c86bba608ebc"} Oct 30 19:47:46.208343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:46.208284 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" podStartSLOduration=1.2082644 podStartE2EDuration="1.2082644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:47:46.208173423 +0000 UTC m=+985.903381383" watchObservedRunningTime="2026-10-30 19:47:46.2082644 +0000 UTC m=+985.903472361" Oct 30 19:47:46.708178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:46.700653 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:46.771172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:46.770254 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:47.771172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:47.770624 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 19:47:47.771172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:47.770695 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" podUID="7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 19:47:47.844190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:47.844070 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:48.770167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:48.769462 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 19:47:48.770167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:48.769526 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" podUID="7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 19:47:49.161335 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:49.161241 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:49.776904 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:49.776366 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 19:47:49.776904 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:49.776444 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" podUID="7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 19:47:50.535166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:50.534675 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:50.776673 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:50.776492 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 19:47:50.776673 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:50.776576 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" podUID="7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 19:47:51.245086 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:51.245053 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:51.390090 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:51.390056 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:51.590503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:51.590464 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:51.766177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:51.765687 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:51.766177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:51.766129 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 19:47:51.766441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:51.766206 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" podUID="7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 19:47:51.930423 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:51.930388 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:52.766680 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:52.766637 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 19:47:52.767098 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:52.766759 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" podUID="7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 19:47:53.042233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:53.038624 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:53.816849 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:53.816808 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:53.817401 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:53.817242 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:53.819397 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:53.819370 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 19:47:54.306361 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:54.306312 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:54.527842 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:54.527810 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:55.159879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:55.159843 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:55.776182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:55.774809 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:56.009070 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:56.009033 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:56.244709 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:56.244630 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:57.335488 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:57.335436 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:58.885297 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:58.885257 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:47:59.969009 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:47:59.968978 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:00.116911 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:00.116875 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:00.615202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:00.615169 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:00.770631 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:00.770589 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:01.547118 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:01.547070 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:01.731645 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:01.731601 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:02.435676 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:02.435628 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:04.112195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:04.102816 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:05.237559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:05.237498 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:06.149332 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:06.149292 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:07.026449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:07.026313 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:07.827120 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:07.827067 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:08.772362 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:08.772331 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:08.936105 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:08.936068 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:09.912200 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:09.910254 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:11.054983 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:11.054948 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:11.458906 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:11.458814 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:12.013174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:12.009618 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:13.189190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:13.189139 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:14.280013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:14.279965 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:15.343043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:15.341675 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:15.978298 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:48:15.978261 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:48:16.274444 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:16.267958 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:17.311862 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:17.311826 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:18.472866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:18.472829 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:19.437182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:19.427423 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:19.898463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:19.898365 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:20.590166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:20.585291 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:21.169343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.169304 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg"] Oct 30 19:48:21.175180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.175129 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" Oct 30 19:48:21.193813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.193737 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:48:21.193941 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.193850 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:48:21.198437 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.198412 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg"] Oct 30 19:48:21.283622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.283584 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd4ss\" (UniqueName: \"kubernetes.io/projected/241d9027-0493-42af-a5d6-13ac4856cd7a-kube-api-access-nd4ss\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg\" (UID: \"241d9027-0493-42af-a5d6-13ac4856cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" Oct 30 19:48:21.283795 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.283759 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241d9027-0493-42af-a5d6-13ac4856cd7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg\" (UID: \"241d9027-0493-42af-a5d6-13ac4856cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" Oct 30 19:48:21.283879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.283864 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241d9027-0493-42af-a5d6-13ac4856cd7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg\" (UID: \"241d9027-0493-42af-a5d6-13ac4856cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" Oct 30 19:48:21.385552 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.385521 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241d9027-0493-42af-a5d6-13ac4856cd7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg\" (UID: \"241d9027-0493-42af-a5d6-13ac4856cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" Oct 30 19:48:21.385723 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.385614 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241d9027-0493-42af-a5d6-13ac4856cd7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg\" (UID: \"241d9027-0493-42af-a5d6-13ac4856cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" Oct 30 19:48:21.385780 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.385719 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd4ss\" (UniqueName: \"kubernetes.io/projected/241d9027-0493-42af-a5d6-13ac4856cd7a-kube-api-access-nd4ss\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg\" (UID: \"241d9027-0493-42af-a5d6-13ac4856cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" Oct 30 19:48:21.386222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.386199 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241d9027-0493-42af-a5d6-13ac4856cd7a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg\" (UID: \"241d9027-0493-42af-a5d6-13ac4856cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" Oct 30 19:48:21.387835 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.387814 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241d9027-0493-42af-a5d6-13ac4856cd7a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg\" (UID: \"241d9027-0493-42af-a5d6-13ac4856cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" Oct 30 19:48:21.401389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.401361 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd4ss\" (UniqueName: \"kubernetes.io/projected/241d9027-0493-42af-a5d6-13ac4856cd7a-kube-api-access-nd4ss\") pod \"maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg\" (UID: \"241d9027-0493-42af-a5d6-13ac4856cd7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" Oct 30 19:48:21.495626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.495538 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" Oct 30 19:48:21.822403 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:48:21.822374 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod241d9027_0493_42af_a5d6_13ac4856cd7a.slice/crio-bb9a4ee5ccdf47f211c9a8adf99045e18ea219f875c3255da235b79131742244 WatchSource:0}: Error finding container bb9a4ee5ccdf47f211c9a8adf99045e18ea219f875c3255da235b79131742244: Status 404 returned error can't find the container with id bb9a4ee5ccdf47f211c9a8adf99045e18ea219f875c3255da235b79131742244 Oct 30 19:48:21.822856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.822832 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg"] Oct 30 19:48:21.925623 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:21.925573 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:22.058535 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.058507 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd"] Oct 30 19:48:22.064427 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.064406 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" Oct 30 19:48:22.090309 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.089303 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd"] Oct 30 19:48:22.097771 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.097731 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k7r2\" (UniqueName: \"kubernetes.io/projected/2f33460c-3569-4427-97ec-86bbada99e74-kube-api-access-4k7r2\") pod \"maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd\" (UID: \"2f33460c-3569-4427-97ec-86bbada99e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" Oct 30 19:48:22.097946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.097890 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f33460c-3569-4427-97ec-86bbada99e74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd\" (UID: \"2f33460c-3569-4427-97ec-86bbada99e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" Oct 30 19:48:22.106224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.098028 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f33460c-3569-4427-97ec-86bbada99e74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd\" (UID: \"2f33460c-3569-4427-97ec-86bbada99e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" Oct 30 19:48:22.199618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.199580 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k7r2\" (UniqueName: \"kubernetes.io/projected/2f33460c-3569-4427-97ec-86bbada99e74-kube-api-access-4k7r2\") pod \"maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd\" (UID: \"2f33460c-3569-4427-97ec-86bbada99e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" Oct 30 19:48:22.200091 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.199748 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f33460c-3569-4427-97ec-86bbada99e74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd\" (UID: \"2f33460c-3569-4427-97ec-86bbada99e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" Oct 30 19:48:22.200091 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.199801 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f33460c-3569-4427-97ec-86bbada99e74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd\" (UID: \"2f33460c-3569-4427-97ec-86bbada99e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" Oct 30 19:48:22.200699 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.200679 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f33460c-3569-4427-97ec-86bbada99e74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd\" (UID: \"2f33460c-3569-4427-97ec-86bbada99e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" Oct 30 19:48:22.202087 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.202066 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f33460c-3569-4427-97ec-86bbada99e74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd\" (UID: \"2f33460c-3569-4427-97ec-86bbada99e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" Oct 30 19:48:22.213215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.213193 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k7r2\" (UniqueName: \"kubernetes.io/projected/2f33460c-3569-4427-97ec-86bbada99e74-kube-api-access-4k7r2\") pod \"maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd\" (UID: \"2f33460c-3569-4427-97ec-86bbada99e74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" Oct 30 19:48:22.377899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.377824 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" Oct 30 19:48:22.668911 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.668792 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" event={"ID":"241d9027-0493-42af-a5d6-13ac4856cd7a","Type":"ContainerStarted","Data":"691589cf34eae4d3bef60a1e983d8bb8d0aa2a9021e1e6b87d5bc485318cf98a"} Oct 30 19:48:22.669245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.669215 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" event={"ID":"241d9027-0493-42af-a5d6-13ac4856cd7a","Type":"ContainerStarted","Data":"bb9a4ee5ccdf47f211c9a8adf99045e18ea219f875c3255da235b79131742244"} Oct 30 19:48:22.674069 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:48:22.674037 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f33460c_3569_4427_97ec_86bbada99e74.slice/crio-10de57620b2ed3cd95a3a42a1edf2ffad502dcc7fac72999f750656213209f9f WatchSource:0}: Error finding container 10de57620b2ed3cd95a3a42a1edf2ffad502dcc7fac72999f750656213209f9f: Status 404 returned error can't find the container with id 10de57620b2ed3cd95a3a42a1edf2ffad502dcc7fac72999f750656213209f9f Oct 30 19:48:22.674278 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.674259 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd"] Oct 30 19:48:22.694913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:22.694865 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" podStartSLOduration=1.694850693 podStartE2EDuration="1.694850693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:22.693902254 +0000 UTC m=+1022.389110212" watchObservedRunningTime="2026-10-30 19:48:22.694850693 +0000 UTC m=+1022.390058651" Oct 30 19:48:23.307788 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:23.307751 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:23.678612 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:23.678575 5091 generic.go:356] "Generic (PLEG): container finished" podID="241d9027-0493-42af-a5d6-13ac4856cd7a" containerID="691589cf34eae4d3bef60a1e983d8bb8d0aa2a9021e1e6b87d5bc485318cf98a" exitCode=0 Oct 30 19:48:23.678794 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:23.678668 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" event={"ID":"241d9027-0493-42af-a5d6-13ac4856cd7a","Type":"ContainerDied","Data":"691589cf34eae4d3bef60a1e983d8bb8d0aa2a9021e1e6b87d5bc485318cf98a"} Oct 30 19:48:23.680560 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:23.680520 5091 generic.go:356] "Generic (PLEG): container finished" podID="2f33460c-3569-4427-97ec-86bbada99e74" containerID="535c3b687d4c08c0b3e284ddec83c2ad358ded2ade7903c142b152af1701fe3c" exitCode=0 Oct 30 19:48:23.680693 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:23.680612 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" event={"ID":"2f33460c-3569-4427-97ec-86bbada99e74","Type":"ContainerDied","Data":"535c3b687d4c08c0b3e284ddec83c2ad358ded2ade7903c142b152af1701fe3c"} Oct 30 19:48:23.680693 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:23.680660 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" event={"ID":"2f33460c-3569-4427-97ec-86bbada99e74","Type":"ContainerStarted","Data":"10de57620b2ed3cd95a3a42a1edf2ffad502dcc7fac72999f750656213209f9f"} Oct 30 19:48:24.198408 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:24.198375 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv"] Oct 30 19:48:24.202708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:24.202690 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" Oct 30 19:48:24.222990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:24.222952 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv"] Oct 30 19:48:24.227313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:24.227282 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jzw\" (UniqueName: \"kubernetes.io/projected/273c236a-1a5d-4313-a7dc-1836307afe35-kube-api-access-b7jzw\") pod \"maas-api-revoke-sub-0b00c018-hvtjv\" (UID: \"273c236a-1a5d-4313-a7dc-1836307afe35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" Oct 30 19:48:24.227490 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:24.227454 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273c236a-1a5d-4313-a7dc-1836307afe35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b00c018-hvtjv\" (UID: \"273c236a-1a5d-4313-a7dc-1836307afe35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" Oct 30 19:48:24.227593 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:24.227573 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273c236a-1a5d-4313-a7dc-1836307afe35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b00c018-hvtjv\" (UID: \"273c236a-1a5d-4313-a7dc-1836307afe35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" Oct 30 19:48:24.329310 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:24.329267 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jzw\" (UniqueName: \"kubernetes.io/projected/273c236a-1a5d-4313-a7dc-1836307afe35-kube-api-access-b7jzw\") pod \"maas-api-revoke-sub-0b00c018-hvtjv\" (UID: \"273c236a-1a5d-4313-a7dc-1836307afe35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" Oct 30 19:48:24.329775 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:24.329415 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273c236a-1a5d-4313-a7dc-1836307afe35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b00c018-hvtjv\" (UID: \"273c236a-1a5d-4313-a7dc-1836307afe35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" Oct 30 19:48:24.329775 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:24.329520 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273c236a-1a5d-4313-a7dc-1836307afe35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b00c018-hvtjv\" (UID: \"273c236a-1a5d-4313-a7dc-1836307afe35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" Oct 30 19:48:24.330459 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:24.330432 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273c236a-1a5d-4313-a7dc-1836307afe35-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b00c018-hvtjv\" (UID: \"273c236a-1a5d-4313-a7dc-1836307afe35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" Oct 30 19:48:24.331877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:24.331855 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273c236a-1a5d-4313-a7dc-1836307afe35-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b00c018-hvtjv\" (UID: \"273c236a-1a5d-4313-a7dc-1836307afe35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" Oct 30 19:48:24.343966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:24.343940 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jzw\" (UniqueName: \"kubernetes.io/projected/273c236a-1a5d-4313-a7dc-1836307afe35-kube-api-access-b7jzw\") pod \"maas-api-revoke-sub-0b00c018-hvtjv\" (UID: \"273c236a-1a5d-4313-a7dc-1836307afe35\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" Oct 30 19:48:24.509167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:24.507237 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:24.530166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:24.524419 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" Oct 30 19:48:24.849535 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:24.847965 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv"] Oct 30 19:48:24.852343 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:48:24.852313 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod273c236a_1a5d_4313_a7dc_1836307afe35.slice/crio-85a06b7251f3ed235236e00e1be7d9a5dc6b86b3ff2555326103fe26f4e80f58 WatchSource:0}: Error finding container 85a06b7251f3ed235236e00e1be7d9a5dc6b86b3ff2555326103fe26f4e80f58: Status 404 returned error can't find the container with id 85a06b7251f3ed235236e00e1be7d9a5dc6b86b3ff2555326103fe26f4e80f58 Oct 30 19:48:25.002447 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.002418 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" Oct 30 19:48:25.004773 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.004756 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" Oct 30 19:48:25.041771 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.041687 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/241d9027-0493-42af-a5d6-13ac4856cd7a-kube-api-access-nd4ss\" (UniqueName: \"kubernetes.io/projected/241d9027-0493-42af-a5d6-13ac4856cd7a-kube-api-access-nd4ss\") pod \"241d9027-0493-42af-a5d6-13ac4856cd7a\" (UID: \"241d9027-0493-42af-a5d6-13ac4856cd7a\") " Oct 30 19:48:25.041771 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.041737 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/241d9027-0493-42af-a5d6-13ac4856cd7a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241d9027-0493-42af-a5d6-13ac4856cd7a-maas-api-xks-ca\") pod \"241d9027-0493-42af-a5d6-13ac4856cd7a\" (UID: \"241d9027-0493-42af-a5d6-13ac4856cd7a\") " Oct 30 19:48:25.041981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.041813 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f33460c-3569-4427-97ec-86bbada99e74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f33460c-3569-4427-97ec-86bbada99e74-maas-api-xks-ca\") pod \"2f33460c-3569-4427-97ec-86bbada99e74\" (UID: \"2f33460c-3569-4427-97ec-86bbada99e74\") " Oct 30 19:48:25.041981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.041849 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f33460c-3569-4427-97ec-86bbada99e74-kube-api-access-4k7r2\" (UniqueName: \"kubernetes.io/projected/2f33460c-3569-4427-97ec-86bbada99e74-kube-api-access-4k7r2\") pod \"2f33460c-3569-4427-97ec-86bbada99e74\" (UID: \"2f33460c-3569-4427-97ec-86bbada99e74\") " Oct 30 19:48:25.041981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.041899 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f33460c-3569-4427-97ec-86bbada99e74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f33460c-3569-4427-97ec-86bbada99e74-maas-api-openshift-ca\") pod \"2f33460c-3569-4427-97ec-86bbada99e74\" (UID: \"2f33460c-3569-4427-97ec-86bbada99e74\") " Oct 30 19:48:25.042136 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.041978 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/241d9027-0493-42af-a5d6-13ac4856cd7a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241d9027-0493-42af-a5d6-13ac4856cd7a-maas-api-openshift-ca\") pod \"241d9027-0493-42af-a5d6-13ac4856cd7a\" (UID: \"241d9027-0493-42af-a5d6-13ac4856cd7a\") " Oct 30 19:48:25.042619 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.042522 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f33460c-3569-4427-97ec-86bbada99e74-maas-api-openshift-ca" pod "2f33460c-3569-4427-97ec-86bbada99e74" (UID: "2f33460c-3569-4427-97ec-86bbada99e74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:25.042619 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.042589 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/241d9027-0493-42af-a5d6-13ac4856cd7a-maas-api-openshift-ca" pod "241d9027-0493-42af-a5d6-13ac4856cd7a" (UID: "241d9027-0493-42af-a5d6-13ac4856cd7a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:25.042786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.042745 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f33460c-3569-4427-97ec-86bbada99e74-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:25.042786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.042768 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/241d9027-0493-42af-a5d6-13ac4856cd7a-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:25.044410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.044376 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f33460c-3569-4427-97ec-86bbada99e74-kube-api-access-4k7r2" pod "2f33460c-3569-4427-97ec-86bbada99e74" (UID: "2f33460c-3569-4427-97ec-86bbada99e74"). InnerVolumeSpecName "kube-api-access-4k7r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:25.044505 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.044449 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f33460c-3569-4427-97ec-86bbada99e74-maas-api-xks-ca" pod "2f33460c-3569-4427-97ec-86bbada99e74" (UID: "2f33460c-3569-4427-97ec-86bbada99e74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:25.044560 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.044538 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/241d9027-0493-42af-a5d6-13ac4856cd7a-kube-api-access-nd4ss" pod "241d9027-0493-42af-a5d6-13ac4856cd7a" (UID: "241d9027-0493-42af-a5d6-13ac4856cd7a"). InnerVolumeSpecName "kube-api-access-nd4ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:25.045498 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.045475 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/241d9027-0493-42af-a5d6-13ac4856cd7a-maas-api-xks-ca" pod "241d9027-0493-42af-a5d6-13ac4856cd7a" (UID: "241d9027-0493-42af-a5d6-13ac4856cd7a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:25.144221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.144142 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd4ss\" (UniqueName: \"kubernetes.io/projected/241d9027-0493-42af-a5d6-13ac4856cd7a-kube-api-access-nd4ss\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:25.144221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.144220 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/241d9027-0493-42af-a5d6-13ac4856cd7a-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:25.144221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.144233 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f33460c-3569-4427-97ec-86bbada99e74-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:25.144456 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.144243 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4k7r2\" (UniqueName: \"kubernetes.io/projected/2f33460c-3569-4427-97ec-86bbada99e74-kube-api-access-4k7r2\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:25.439566 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.439530 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:25.699496 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.699421 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" Oct 30 19:48:25.699496 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.699456 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg" event={"ID":"241d9027-0493-42af-a5d6-13ac4856cd7a","Type":"ContainerDied","Data":"bb9a4ee5ccdf47f211c9a8adf99045e18ea219f875c3255da235b79131742244"} Oct 30 19:48:25.699687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.699507 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb9a4ee5ccdf47f211c9a8adf99045e18ea219f875c3255da235b79131742244" Oct 30 19:48:25.700838 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.700819 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" Oct 30 19:48:25.700975 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.700855 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd" event={"ID":"2f33460c-3569-4427-97ec-86bbada99e74","Type":"ContainerDied","Data":"10de57620b2ed3cd95a3a42a1edf2ffad502dcc7fac72999f750656213209f9f"} Oct 30 19:48:25.700975 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.700900 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10de57620b2ed3cd95a3a42a1edf2ffad502dcc7fac72999f750656213209f9f" Oct 30 19:48:25.702672 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.702649 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" event={"ID":"273c236a-1a5d-4313-a7dc-1836307afe35","Type":"ContainerStarted","Data":"c7eadfefefdd941544d14c55acbade5ebf8185893f8a6531d49a66a2103f7dde"} Oct 30 19:48:25.702804 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.702687 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" event={"ID":"273c236a-1a5d-4313-a7dc-1836307afe35","Type":"ContainerStarted","Data":"85a06b7251f3ed235236e00e1be7d9a5dc6b86b3ff2555326103fe26f4e80f58"} Oct 30 19:48:25.734685 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.734624 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" podStartSLOduration=1.734612112 podStartE2EDuration="1.734612112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:25.734239076 +0000 UTC m=+1025.429447036" watchObservedRunningTime="2026-10-30 19:48:25.734612112 +0000 UTC m=+1025.429820069" Oct 30 19:48:25.998514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:25.998476 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:26.478544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.474600 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf"] Oct 30 19:48:26.478544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.475253 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2f33460c-3569-4427-97ec-86bbada99e74" containerName="revoke-keys" Oct 30 19:48:26.478544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.475267 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f33460c-3569-4427-97ec-86bbada99e74" containerName="revoke-keys" Oct 30 19:48:26.478544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.475318 5091 cpu_manager.go:414] "containerMap: removing container" podUID="241d9027-0493-42af-a5d6-13ac4856cd7a" containerName="revoke-keys" Oct 30 19:48:26.478544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.475324 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="241d9027-0493-42af-a5d6-13ac4856cd7a" containerName="revoke-keys" Oct 30 19:48:26.478544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.475414 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="241d9027-0493-42af-a5d6-13ac4856cd7a" containerName="revoke-keys" Oct 30 19:48:26.478544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.475423 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f33460c-3569-4427-97ec-86bbada99e74" containerName="revoke-keys" Oct 30 19:48:26.479339 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.479216 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.497348 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.497311 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-913351-openshift-default-dockercfg-wwrg9\"" Oct 30 19:48:26.503880 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.503851 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf"] Oct 30 19:48:26.581709 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.581679 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-workload-certs\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.581881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.581747 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3c46ef-f755-4462-9dae-250f052538b5-istio-token\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.581881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.581808 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-istio-envoy\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.581881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.581874 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-credential-socket\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.582053 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.581981 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-workload-socket\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.582053 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.582013 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3c46ef-f755-4462-9dae-250f052538b5-istiod-ca-cert\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.582053 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.582031 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3c46ef-f755-4462-9dae-250f052538b5-istio-podinfo\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.582228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.582168 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgrk8\" (UniqueName: \"kubernetes.io/projected/4b3c46ef-f755-4462-9dae-250f052538b5-kube-api-access-tgrk8\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.582281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.582224 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-istio-data\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.667171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.642783 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:26.695240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.694078 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-istio-data\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.695240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.694262 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-workload-certs\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.695240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.694329 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3c46ef-f755-4462-9dae-250f052538b5-istio-token\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.695240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.694361 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-istio-envoy\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.695240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.694432 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-credential-socket\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.695240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.694491 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-workload-socket\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.695240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.694541 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3c46ef-f755-4462-9dae-250f052538b5-istiod-ca-cert\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.695240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.694566 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3c46ef-f755-4462-9dae-250f052538b5-istio-podinfo\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.695240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.694653 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgrk8\" (UniqueName: \"kubernetes.io/projected/4b3c46ef-f755-4462-9dae-250f052538b5-kube-api-access-tgrk8\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.695870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.695489 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-istio-data\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.695870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.695727 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-workload-certs\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.706055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.700605 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-workload-socket\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.706055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.700923 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-credential-socket\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.706055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.701581 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3c46ef-f755-4462-9dae-250f052538b5-istiod-ca-cert\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.832516 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.809728 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-istio-envoy\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.832516 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.810640 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3c46ef-f755-4462-9dae-250f052538b5-istio-podinfo\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.857212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.835272 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgrk8\" (UniqueName: \"kubernetes.io/projected/4b3c46ef-f755-4462-9dae-250f052538b5-kube-api-access-tgrk8\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.857212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.843723 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:26.857212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.852893 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3c46ef-f755-4462-9dae-250f052538b5-istio-token\") pod \"e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:26.871237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.860287 5091 generic.go:356] "Generic (PLEG): container finished" podID="273c236a-1a5d-4313-a7dc-1836307afe35" containerID="c7eadfefefdd941544d14c55acbade5ebf8185893f8a6531d49a66a2103f7dde" exitCode=0 Oct 30 19:48:26.871237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:26.860466 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" event={"ID":"273c236a-1a5d-4313-a7dc-1836307afe35","Type":"ContainerDied","Data":"c7eadfefefdd941544d14c55acbade5ebf8185893f8a6531d49a66a2103f7dde"} Oct 30 19:48:27.096364 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:27.096270 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:27.392172 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:48:27.392114 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b3c46ef_f755_4462_9dae_250f052538b5.slice/crio-2e0efa3d75bb18cd67e617a23eae1250c366f185ee4214e919c8e14e32395387 WatchSource:0}: Error finding container 2e0efa3d75bb18cd67e617a23eae1250c366f185ee4214e919c8e14e32395387: Status 404 returned error can't find the container with id 2e0efa3d75bb18cd67e617a23eae1250c366f185ee4214e919c8e14e32395387 Oct 30 19:48:27.394894 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:27.394858 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:48:27.395008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:27.394932 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:48:27.395008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:27.394962 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:48:27.395113 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:27.395015 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf"] Oct 30 19:48:27.632139 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:27.632068 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:27.869334 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:27.869293 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" event={"ID":"4b3c46ef-f755-4462-9dae-250f052538b5","Type":"ContainerStarted","Data":"0a674f1877120eb81273084b00af32f9b0af4bd6298e08853ded33b74a2e5af4"} Oct 30 19:48:27.869491 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:27.869361 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" event={"ID":"4b3c46ef-f755-4462-9dae-250f052538b5","Type":"ContainerStarted","Data":"2e0efa3d75bb18cd67e617a23eae1250c366f185ee4214e919c8e14e32395387"} Oct 30 19:48:27.905440 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:27.905389 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" podStartSLOduration=1.9053700180000002 podStartE2EDuration="1.905370018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:27.902655763 +0000 UTC m=+1027.597863721" watchObservedRunningTime="2026-10-30 19:48:27.905370018 +0000 UTC m=+1027.600577977" Oct 30 19:48:28.100469 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.098657 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:28.100469 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.099829 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" Oct 30 19:48:28.117234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.117055 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:28.210221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.210117 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/273c236a-1a5d-4313-a7dc-1836307afe35-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273c236a-1a5d-4313-a7dc-1836307afe35-maas-api-xks-ca\") pod \"273c236a-1a5d-4313-a7dc-1836307afe35\" (UID: \"273c236a-1a5d-4313-a7dc-1836307afe35\") " Oct 30 19:48:28.210360 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.210250 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/273c236a-1a5d-4313-a7dc-1836307afe35-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273c236a-1a5d-4313-a7dc-1836307afe35-maas-api-openshift-ca\") pod \"273c236a-1a5d-4313-a7dc-1836307afe35\" (UID: \"273c236a-1a5d-4313-a7dc-1836307afe35\") " Oct 30 19:48:28.210403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.210368 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/273c236a-1a5d-4313-a7dc-1836307afe35-kube-api-access-b7jzw\" (UniqueName: \"kubernetes.io/projected/273c236a-1a5d-4313-a7dc-1836307afe35-kube-api-access-b7jzw\") pod \"273c236a-1a5d-4313-a7dc-1836307afe35\" (UID: \"273c236a-1a5d-4313-a7dc-1836307afe35\") " Oct 30 19:48:28.210607 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.210579 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/273c236a-1a5d-4313-a7dc-1836307afe35-maas-api-openshift-ca" pod "273c236a-1a5d-4313-a7dc-1836307afe35" (UID: "273c236a-1a5d-4313-a7dc-1836307afe35"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:28.211006 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.210983 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/273c236a-1a5d-4313-a7dc-1836307afe35-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:28.212196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.212170 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/273c236a-1a5d-4313-a7dc-1836307afe35-maas-api-xks-ca" pod "273c236a-1a5d-4313-a7dc-1836307afe35" (UID: "273c236a-1a5d-4313-a7dc-1836307afe35"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:28.212321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.212300 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/273c236a-1a5d-4313-a7dc-1836307afe35-kube-api-access-b7jzw" pod "273c236a-1a5d-4313-a7dc-1836307afe35" (UID: "273c236a-1a5d-4313-a7dc-1836307afe35"). InnerVolumeSpecName "kube-api-access-b7jzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:28.312734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.312698 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7jzw\" (UniqueName: \"kubernetes.io/projected/273c236a-1a5d-4313-a7dc-1836307afe35-kube-api-access-b7jzw\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:28.312734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.312728 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/273c236a-1a5d-4313-a7dc-1836307afe35-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:28.699894 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.699785 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:28.880939 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.880909 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" Oct 30 19:48:28.880939 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.880929 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv" event={"ID":"273c236a-1a5d-4313-a7dc-1836307afe35","Type":"ContainerDied","Data":"85a06b7251f3ed235236e00e1be7d9a5dc6b86b3ff2555326103fe26f4e80f58"} Oct 30 19:48:28.881251 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.880979 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85a06b7251f3ed235236e00e1be7d9a5dc6b86b3ff2555326103fe26f4e80f58" Oct 30 19:48:28.881636 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.881569 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:28.882797 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:28.882775 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:48:29.750171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:29.737781 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:30.198122 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:30.198022 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:30.355305 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:30.355265 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:30.898615 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:30.898140 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:31.542869 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:31.542832 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:31.796563 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:31.796465 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:32.229224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:32.222876 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:32.371763 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:32.371727 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:32.517181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:32.517116 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:32.755167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:32.730934 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:33.010236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:33.010201 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:33.437222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:33.417877 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:34.382854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:34.382806 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:35.312340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.312245 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:35.342574 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.342538 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85"] Oct 30 19:48:35.343984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.343953 5091 cpu_manager.go:414] "containerMap: removing container" podUID="273c236a-1a5d-4313-a7dc-1836307afe35" containerName="revoke-keys" Oct 30 19:48:35.344171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.344139 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="273c236a-1a5d-4313-a7dc-1836307afe35" containerName="revoke-keys" Oct 30 19:48:35.344603 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.344583 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="273c236a-1a5d-4313-a7dc-1836307afe35" containerName="revoke-keys" Oct 30 19:48:35.352564 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.352543 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" Oct 30 19:48:35.368313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.368195 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85"] Oct 30 19:48:35.373894 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.373850 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:48:35.376236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.376214 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:48:35.441519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.441486 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e9f9dd1-39a8-419f-bccc-38a670d0786b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36edca7e-6xv85\" (UID: \"3e9f9dd1-39a8-419f-bccc-38a670d0786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" Oct 30 19:48:35.441676 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.441525 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xnn\" (UniqueName: \"kubernetes.io/projected/3e9f9dd1-39a8-419f-bccc-38a670d0786b-kube-api-access-z8xnn\") pod \"maas-api-revoke-sub-36edca7e-6xv85\" (UID: \"3e9f9dd1-39a8-419f-bccc-38a670d0786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" Oct 30 19:48:35.441942 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.441914 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e9f9dd1-39a8-419f-bccc-38a670d0786b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36edca7e-6xv85\" (UID: \"3e9f9dd1-39a8-419f-bccc-38a670d0786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" Oct 30 19:48:35.543703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.543663 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e9f9dd1-39a8-419f-bccc-38a670d0786b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36edca7e-6xv85\" (UID: \"3e9f9dd1-39a8-419f-bccc-38a670d0786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" Oct 30 19:48:35.543874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.543771 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e9f9dd1-39a8-419f-bccc-38a670d0786b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36edca7e-6xv85\" (UID: \"3e9f9dd1-39a8-419f-bccc-38a670d0786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" Oct 30 19:48:35.543874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.543808 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xnn\" (UniqueName: \"kubernetes.io/projected/3e9f9dd1-39a8-419f-bccc-38a670d0786b-kube-api-access-z8xnn\") pod \"maas-api-revoke-sub-36edca7e-6xv85\" (UID: \"3e9f9dd1-39a8-419f-bccc-38a670d0786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" Oct 30 19:48:35.544509 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.544485 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e9f9dd1-39a8-419f-bccc-38a670d0786b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36edca7e-6xv85\" (UID: \"3e9f9dd1-39a8-419f-bccc-38a670d0786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" Oct 30 19:48:35.546090 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.546068 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e9f9dd1-39a8-419f-bccc-38a670d0786b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36edca7e-6xv85\" (UID: \"3e9f9dd1-39a8-419f-bccc-38a670d0786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" Oct 30 19:48:35.569821 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.569763 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xnn\" (UniqueName: \"kubernetes.io/projected/3e9f9dd1-39a8-419f-bccc-38a670d0786b-kube-api-access-z8xnn\") pod \"maas-api-revoke-sub-36edca7e-6xv85\" (UID: \"3e9f9dd1-39a8-419f-bccc-38a670d0786b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" Oct 30 19:48:35.675471 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.675444 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" Oct 30 19:48:35.992849 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:48:35.992820 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e9f9dd1_39a8_419f_bccc_38a670d0786b.slice/crio-e2deba2ba3efc8117b347e97b225379929ff4d524955ec4d511ba9e2b8470202 WatchSource:0}: Error finding container e2deba2ba3efc8117b347e97b225379929ff4d524955ec4d511ba9e2b8470202: Status 404 returned error can't find the container with id e2deba2ba3efc8117b347e97b225379929ff4d524955ec4d511ba9e2b8470202 Oct 30 19:48:35.994073 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:35.994051 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85"] Oct 30 19:48:36.080978 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:36.080943 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" event={"ID":"3e9f9dd1-39a8-419f-bccc-38a670d0786b","Type":"ContainerStarted","Data":"e2deba2ba3efc8117b347e97b225379929ff4d524955ec4d511ba9e2b8470202"} Oct 30 19:48:36.497433 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:36.497392 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:37.092695 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:37.092600 5091 generic.go:356] "Generic (PLEG): container finished" podID="3e9f9dd1-39a8-419f-bccc-38a670d0786b" containerID="c02f45b949dca6064862cb795f8ee59bf5800fe7526e2009f9014af0338a43df" exitCode=0 Oct 30 19:48:37.092904 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:37.092689 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" event={"ID":"3e9f9dd1-39a8-419f-bccc-38a670d0786b","Type":"ContainerDied","Data":"c02f45b949dca6064862cb795f8ee59bf5800fe7526e2009f9014af0338a43df"} Oct 30 19:48:37.606060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:37.606022 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:38.262864 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:38.262841 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" Oct 30 19:48:38.400201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:38.400082 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e9f9dd1-39a8-419f-bccc-38a670d0786b-kube-api-access-z8xnn\" (UniqueName: \"kubernetes.io/projected/3e9f9dd1-39a8-419f-bccc-38a670d0786b-kube-api-access-z8xnn\") pod \"3e9f9dd1-39a8-419f-bccc-38a670d0786b\" (UID: \"3e9f9dd1-39a8-419f-bccc-38a670d0786b\") " Oct 30 19:48:38.400375 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:38.400330 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e9f9dd1-39a8-419f-bccc-38a670d0786b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e9f9dd1-39a8-419f-bccc-38a670d0786b-maas-api-openshift-ca\") pod \"3e9f9dd1-39a8-419f-bccc-38a670d0786b\" (UID: \"3e9f9dd1-39a8-419f-bccc-38a670d0786b\") " Oct 30 19:48:38.400446 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:38.400404 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e9f9dd1-39a8-419f-bccc-38a670d0786b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e9f9dd1-39a8-419f-bccc-38a670d0786b-maas-api-xks-ca\") pod \"3e9f9dd1-39a8-419f-bccc-38a670d0786b\" (UID: \"3e9f9dd1-39a8-419f-bccc-38a670d0786b\") " Oct 30 19:48:38.400695 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:38.400669 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e9f9dd1-39a8-419f-bccc-38a670d0786b-maas-api-openshift-ca" pod "3e9f9dd1-39a8-419f-bccc-38a670d0786b" (UID: "3e9f9dd1-39a8-419f-bccc-38a670d0786b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:38.401009 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:38.400991 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e9f9dd1-39a8-419f-bccc-38a670d0786b-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:38.402417 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:38.402380 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e9f9dd1-39a8-419f-bccc-38a670d0786b-kube-api-access-z8xnn" pod "3e9f9dd1-39a8-419f-bccc-38a670d0786b" (UID: "3e9f9dd1-39a8-419f-bccc-38a670d0786b"). InnerVolumeSpecName "kube-api-access-z8xnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:38.402780 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:38.402757 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e9f9dd1-39a8-419f-bccc-38a670d0786b-maas-api-xks-ca" pod "3e9f9dd1-39a8-419f-bccc-38a670d0786b" (UID: "3e9f9dd1-39a8-419f-bccc-38a670d0786b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:48:38.502529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:38.502498 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e9f9dd1-39a8-419f-bccc-38a670d0786b-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:38.502656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:38.502542 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8xnn\" (UniqueName: \"kubernetes.io/projected/3e9f9dd1-39a8-419f-bccc-38a670d0786b-kube-api-access-z8xnn\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:38.699219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:38.698519 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:38.978846 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:38.978760 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:39.110273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:39.110247 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" Oct 30 19:48:39.110441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:39.110387 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85" event={"ID":"3e9f9dd1-39a8-419f-bccc-38a670d0786b","Type":"ContainerDied","Data":"e2deba2ba3efc8117b347e97b225379929ff4d524955ec4d511ba9e2b8470202"} Oct 30 19:48:39.110486 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:39.110459 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2deba2ba3efc8117b347e97b225379929ff4d524955ec4d511ba9e2b8470202" Oct 30 19:48:39.238410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:39.238327 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:39.892966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:39.892927 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:40.713342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:40.713303 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:40.991623 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:40.991534 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:41.175918 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:41.175846 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg"] Oct 30 19:48:41.184435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:41.184403 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-477fd9c0-lkgqg"] Oct 30 19:48:41.208606 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:41.208576 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:41.353508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:41.353472 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:41.744361 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:41.744272 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:42.442110 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:42.442074 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="241d9027-0493-42af-a5d6-13ac4856cd7a" path="/var/lib/kubelet/pods/241d9027-0493-42af-a5d6-13ac4856cd7a/volumes" Oct 30 19:48:42.571302 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:42.571262 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:43.481071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:43.481034 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:44.710852 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:44.710814 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:46.073177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:46.065414 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:46.238912 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:48:46.238291 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:48:46.636166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:46.635727 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:46.946954 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:46.946864 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:47.248225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:47.229618 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:47.476170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:47.462860 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:47.961292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:47.961257 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:48.016711 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:48.016674 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd"] Oct 30 19:48:48.021579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:48.021551 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-501ac8-gnvzd"] Oct 30 19:48:48.160788 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:48.160754 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:48.442197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:48.442114 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f33460c-3569-4427-97ec-86bbada99e74" path="/var/lib/kubelet/pods/2f33460c-3569-4427-97ec-86bbada99e74/volumes" Oct 30 19:48:48.456181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:48.456120 5091 generic.go:356] "Generic (PLEG): container finished" podID="c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d" containerID="b13042955905aaaab4ef4039572814a5399d4e6086dcad8ac37cede26fded48d" exitCode=0 Oct 30 19:48:48.456483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:48.456190 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nw4zh-vslhs" event={"ID":"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d","Type":"ContainerDied","Data":"b13042955905aaaab4ef4039572814a5399d4e6086dcad8ac37cede26fded48d"} Oct 30 19:48:48.618441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:48.618403 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:49.440752 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:49.440714 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:49.650190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:49.650162 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:48:49.684665 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:49.684632 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-archives-path\") pod \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " Oct 30 19:48:49.684843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:49.684687 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-trusted-ca-bundle\") pod \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " Oct 30 19:48:49.684843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:49.684826 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-kube-api-access-b552j\" (UniqueName: \"kubernetes.io/projected/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-kube-api-access-b552j\") pod \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " Oct 30 19:48:49.684977 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:49.684967 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-service-ca-bundle\") pod \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\" (UID: \"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d\") " Oct 30 19:48:49.685092 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:49.685017 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-archives-path" pod "c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d" (UID: "c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:48:49.685237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:49.685214 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-trusted-ca-bundle" pod "c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d" (UID: "c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:49.685434 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:49.685412 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-service-ca-bundle" pod "c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d" (UID: "c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:48:49.685771 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:49.685747 5091 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-archives-path\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:49.685851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:49.685777 5091 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-trusted-ca-bundle\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:49.685851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:49.685791 5091 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-service-ca-bundle\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:49.687521 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:49.687495 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-kube-api-access-b552j" pod "c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d" (UID: "c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d"). InnerVolumeSpecName "kube-api-access-b552j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:48:49.787546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:49.787505 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b552j\" (UniqueName: \"kubernetes.io/projected/c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d-kube-api-access-b552j\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:48:50.496135 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:50.496108 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-nw4zh-vslhs" Oct 30 19:48:50.496135 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:50.496158 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-nw4zh-vslhs" event={"ID":"c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d","Type":"ContainerDied","Data":"a2f70e007ef1957772dd99f8f483f58323de26ac7f4805599bfc509c251bc128"} Oct 30 19:48:50.496379 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:50.496210 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2f70e007ef1957772dd99f8f483f58323de26ac7f4805599bfc509c251bc128" Oct 30 19:48:50.625860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:50.625824 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:50.969527 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:50.969493 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:51.121812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:51.121771 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:51.691351 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:51.691308 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:51.904168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:51.897432 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:52.900422 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:52.900380 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:53.061255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:53.061213 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:53.872198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:53.869121 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:54.799138 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:54.793953 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:55.521275 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:55.521243 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:55.667587 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:55.667557 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:55.828266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:55.825589 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:56.570946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:56.570721 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:56.743828 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:56.743750 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:56.923132 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:56.923039 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:57.375581 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.375541 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj"] Oct 30 19:48:57.376525 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.376498 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d" containerName="insights-gathering" Oct 30 19:48:57.376623 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.376528 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d" containerName="insights-gathering" Oct 30 19:48:57.376623 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.376566 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3e9f9dd1-39a8-419f-bccc-38a670d0786b" containerName="revoke-keys" Oct 30 19:48:57.376623 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.376575 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e9f9dd1-39a8-419f-bccc-38a670d0786b" containerName="revoke-keys" Oct 30 19:48:57.376772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.376760 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e9f9dd1-39a8-419f-bccc-38a670d0786b" containerName="revoke-keys" Oct 30 19:48:57.376810 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.376787 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9b2a4b1-f12b-43d4-a6d4-06898dca3b9d" containerName="insights-gathering" Oct 30 19:48:57.382657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.382636 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" Oct 30 19:48:57.390330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.390304 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj"] Oct 30 19:48:57.393817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.393797 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:48:57.393915 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.393838 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:48:57.481776 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.481734 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2761c7-g78vj\" (UID: \"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" Oct 30 19:48:57.481971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.481862 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2761c7-g78vj\" (UID: \"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" Oct 30 19:48:57.481971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.481892 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmzbd\" (UniqueName: \"kubernetes.io/projected/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-kube-api-access-hmzbd\") pod \"maas-api-revoke-keys-e2e-api-b-2761c7-g78vj\" (UID: \"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" Oct 30 19:48:57.583512 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.583475 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2761c7-g78vj\" (UID: \"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" Oct 30 19:48:57.583715 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.583565 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2761c7-g78vj\" (UID: \"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" Oct 30 19:48:57.583715 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.583601 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmzbd\" (UniqueName: \"kubernetes.io/projected/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-kube-api-access-hmzbd\") pod \"maas-api-revoke-keys-e2e-api-b-2761c7-g78vj\" (UID: \"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" Oct 30 19:48:57.584252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.584217 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2761c7-g78vj\" (UID: \"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" Oct 30 19:48:57.586142 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.586114 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2761c7-g78vj\" (UID: \"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" Oct 30 19:48:57.602867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.602844 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmzbd\" (UniqueName: \"kubernetes.io/projected/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-kube-api-access-hmzbd\") pod \"maas-api-revoke-keys-e2e-api-b-2761c7-g78vj\" (UID: \"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" Oct 30 19:48:57.701780 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.701715 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" Oct 30 19:48:57.903977 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.903772 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:57.988012 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:48:57.987980 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2af1909_5f46_40e4_9efb_9e7e8a3c52f6.slice/crio-af953b41dfb9ccc04370a8ac960dbac52d895c6ef34dd9284887d806120b341c WatchSource:0}: Error finding container af953b41dfb9ccc04370a8ac960dbac52d895c6ef34dd9284887d806120b341c: Status 404 returned error can't find the container with id af953b41dfb9ccc04370a8ac960dbac52d895c6ef34dd9284887d806120b341c Oct 30 19:48:57.990935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:57.990901 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj"] Oct 30 19:48:58.530331 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:58.530250 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:58.564691 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:58.564653 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" event={"ID":"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6","Type":"ContainerStarted","Data":"707a911c1a24031c9b4b293dd84b75485a24c71a81879e125eee459b1394dc12"} Oct 30 19:48:58.564930 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:58.564907 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" event={"ID":"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6","Type":"ContainerStarted","Data":"af953b41dfb9ccc04370a8ac960dbac52d895c6ef34dd9284887d806120b341c"} Oct 30 19:48:58.614515 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:58.614463 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" podStartSLOduration=1.614449729 podStartE2EDuration="1.614449729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:48:58.609499558 +0000 UTC m=+1058.304707517" watchObservedRunningTime="2026-10-30 19:48:58.614449729 +0000 UTC m=+1058.309657686" Oct 30 19:48:58.754172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:58.750539 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:58.986384 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:58.986300 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667"] Oct 30 19:48:59.000320 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.000280 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667"] Oct 30 19:48:59.000826 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.000404 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" Oct 30 19:48:59.024358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.024320 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-913351-dockercfg-kgqrt\"" Oct 30 19:48:59.024658 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.024637 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-913351\"" Oct 30 19:48:59.128004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.127978 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncbwl\" (UniqueName: \"kubernetes.io/projected/92637c83-dfce-417e-99a2-2f433a1cfdf2-kube-api-access-ncbwl\") pod \"payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667\" (UID: \"92637c83-dfce-417e-99a2-2f433a1cfdf2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" Oct 30 19:48:59.128138 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.128109 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92637c83-dfce-417e-99a2-2f433a1cfdf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667\" (UID: \"92637c83-dfce-417e-99a2-2f433a1cfdf2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" Oct 30 19:48:59.229059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.229020 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncbwl\" (UniqueName: \"kubernetes.io/projected/92637c83-dfce-417e-99a2-2f433a1cfdf2-kube-api-access-ncbwl\") pod \"payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667\" (UID: \"92637c83-dfce-417e-99a2-2f433a1cfdf2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" Oct 30 19:48:59.229271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.229129 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92637c83-dfce-417e-99a2-2f433a1cfdf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667\" (UID: \"92637c83-dfce-417e-99a2-2f433a1cfdf2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" Oct 30 19:48:59.230049 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.230025 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92637c83-dfce-417e-99a2-2f433a1cfdf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667\" (UID: \"92637c83-dfce-417e-99a2-2f433a1cfdf2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" Oct 30 19:48:59.293791 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.292865 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncbwl\" (UniqueName: \"kubernetes.io/projected/92637c83-dfce-417e-99a2-2f433a1cfdf2-kube-api-access-ncbwl\") pod \"payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667\" (UID: \"92637c83-dfce-417e-99a2-2f433a1cfdf2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" Oct 30 19:48:59.316462 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.316429 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" Oct 30 19:48:59.395713 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.395589 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:59.569501 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.553395 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:48:59.613109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.613053 5091 generic.go:356] "Generic (PLEG): container finished" podID="d2af1909-5f46-40e4-9efb-9e7e8a3c52f6" containerID="707a911c1a24031c9b4b293dd84b75485a24c71a81879e125eee459b1394dc12" exitCode=0 Oct 30 19:48:59.613537 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.613512 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" event={"ID":"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6","Type":"ContainerDied","Data":"707a911c1a24031c9b4b293dd84b75485a24c71a81879e125eee459b1394dc12"} Oct 30 19:48:59.644951 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:48:59.644919 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92637c83_dfce_417e_99a2_2f433a1cfdf2.slice/crio-d1a5b69f837c4598d6014fc7e360c21384476b881e88b7938d74ceff546f2e0e WatchSource:0}: Error finding container d1a5b69f837c4598d6014fc7e360c21384476b881e88b7938d74ceff546f2e0e: Status 404 returned error can't find the container with id d1a5b69f837c4598d6014fc7e360c21384476b881e88b7938d74ceff546f2e0e Oct 30 19:48:59.647610 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.647588 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 19:48:59.648557 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:48:59.648532 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667"] Oct 30 19:49:00.659472 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:00.658723 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" event={"ID":"92637c83-dfce-417e-99a2-2f433a1cfdf2","Type":"ContainerStarted","Data":"d1a5b69f837c4598d6014fc7e360c21384476b881e88b7938d74ceff546f2e0e"} Oct 30 19:49:00.670390 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:00.667869 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:00.919529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:00.919462 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" Oct 30 19:49:01.034214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:01.033669 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-kube-api-access-hmzbd\" (UniqueName: \"kubernetes.io/projected/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-kube-api-access-hmzbd\") pod \"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6\" (UID: \"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6\") " Oct 30 19:49:01.034214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:01.033905 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-maas-api-openshift-ca\") pod \"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6\" (UID: \"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6\") " Oct 30 19:49:01.034214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:01.034053 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-maas-api-xks-ca\") pod \"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6\" (UID: \"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6\") " Oct 30 19:49:01.071180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:01.069797 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-maas-api-openshift-ca" pod "d2af1909-5f46-40e4-9efb-9e7e8a3c52f6" (UID: "d2af1909-5f46-40e4-9efb-9e7e8a3c52f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:01.092169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:01.080760 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-maas-api-xks-ca" pod "d2af1909-5f46-40e4-9efb-9e7e8a3c52f6" (UID: "d2af1909-5f46-40e4-9efb-9e7e8a3c52f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:49:01.134336 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:01.133829 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-kube-api-access-hmzbd" pod "d2af1909-5f46-40e4-9efb-9e7e8a3c52f6" (UID: "d2af1909-5f46-40e4-9efb-9e7e8a3c52f6"). InnerVolumeSpecName "kube-api-access-hmzbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:01.136235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:01.136163 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:01.136235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:01.136194 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:01.136235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:01.136210 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmzbd\" (UniqueName: \"kubernetes.io/projected/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6-kube-api-access-hmzbd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:01.691188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:01.690137 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" Oct 30 19:49:01.691188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:01.690243 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj" event={"ID":"d2af1909-5f46-40e4-9efb-9e7e8a3c52f6","Type":"ContainerDied","Data":"af953b41dfb9ccc04370a8ac960dbac52d895c6ef34dd9284887d806120b341c"} Oct 30 19:49:01.691188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:01.690292 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af953b41dfb9ccc04370a8ac960dbac52d895c6ef34dd9284887d806120b341c" Oct 30 19:49:02.116182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:02.100529 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:03.528241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:03.528128 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:04.539187 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:04.538183 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:04.736119 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:04.736073 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" event={"ID":"92637c83-dfce-417e-99a2-2f433a1cfdf2","Type":"ContainerStarted","Data":"35d8cb8dd97f042f0ca05a0f0f62a0c058b51d53537c0136166acaec3753daa6"} Oct 30 19:49:04.736321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:04.736264 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" Oct 30 19:49:04.763958 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:04.763898 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" podStartSLOduration=2.174464357 podStartE2EDuration="6.763879665s" totalImagesPullingTime="4.589415308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:48:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:04.758442566 +0000 UTC m=+1064.453650528" watchObservedRunningTime="2026-10-30 19:49:04.763879665 +0000 UTC m=+1064.459087624" Oct 30 19:49:05.542491 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:05.542452 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:06.565367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:06.565332 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:07.390249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:07.390209 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:07.698906 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:07.698828 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:07.840906 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:07.840874 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:08.490609 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:08.490571 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:09.673299 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:09.673250 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:10.580642 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:10.580609 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:11.044254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:11.044219 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw"] Oct 30 19:49:11.056692 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:11.056658 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-514fad51-kf5gw"] Oct 30 19:49:11.576840 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:11.576407 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:11.726055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:11.726015 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:11.908698 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:11.908612 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:12.442719 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:12.442685 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41d846aa-4c80-4702-ab6e-9a2216de0b5b" path="/var/lib/kubelet/pods/41d846aa-4c80-4702-ab6e-9a2216de0b5b/volumes" Oct 30 19:49:12.593823 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:12.593786 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:13.299970 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:13.299931 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:13.458735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:13.458696 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:13.893178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:13.892489 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:14.341455 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:14.341415 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:14.797613 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:14.797574 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:15.651687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:15.651585 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:15.765346 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:15.764365 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" Oct 30 19:49:15.980642 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:49:15.980565 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:49:16.685432 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:16.685320 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:17.293344 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:17.293307 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:17.863627 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:17.863587 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:18.063165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:18.062814 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:18.223378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:18.223271 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:18.370901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:18.370807 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:18.610701 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:18.603167 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:19.223762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.223728 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:19.411261 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.411225 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk"] Oct 30 19:49:19.412067 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.412045 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d2af1909-5f46-40e4-9efb-9e7e8a3c52f6" containerName="revoke-keys" Oct 30 19:49:19.412067 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.412066 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2af1909-5f46-40e4-9efb-9e7e8a3c52f6" containerName="revoke-keys" Oct 30 19:49:19.412309 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.412294 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2af1909-5f46-40e4-9efb-9e7e8a3c52f6" containerName="revoke-keys" Oct 30 19:49:19.418088 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.418061 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" Oct 30 19:49:19.436755 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.434757 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:19.439694 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.439474 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c9d69c\"" Oct 30 19:49:19.439694 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.439625 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c9d69c-dockercfg-gkrsq\"" Oct 30 19:49:19.469516 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.469440 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk"] Oct 30 19:49:19.495175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.495134 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q"] Oct 30 19:49:19.499008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.498978 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cn6c\" (UniqueName: \"kubernetes.io/projected/47f67a54-7bd4-4c38-b6d3-1e20d963aa1b-kube-api-access-5cn6c\") pod \"payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk\" (UID: \"47f67a54-7bd4-4c38-b6d3-1e20d963aa1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" Oct 30 19:49:19.499124 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.499092 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f67a54-7bd4-4c38-b6d3-1e20d963aa1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk\" (UID: \"47f67a54-7bd4-4c38-b6d3-1e20d963aa1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" Oct 30 19:49:19.504855 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.504835 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q"] Oct 30 19:49:19.504981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.504917 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" Oct 30 19:49:19.600740 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.600706 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f67a54-7bd4-4c38-b6d3-1e20d963aa1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk\" (UID: \"47f67a54-7bd4-4c38-b6d3-1e20d963aa1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" Oct 30 19:49:19.600911 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.600841 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzh8t\" (UniqueName: \"kubernetes.io/projected/8b6a1c99-8256-4957-bd95-dc3f18f1d147-kube-api-access-rzh8t\") pod \"payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q\" (UID: \"8b6a1c99-8256-4957-bd95-dc3f18f1d147\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" Oct 30 19:49:19.601050 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.601022 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b6a1c99-8256-4957-bd95-dc3f18f1d147-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q\" (UID: \"8b6a1c99-8256-4957-bd95-dc3f18f1d147\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" Oct 30 19:49:19.601208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.601094 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cn6c\" (UniqueName: \"kubernetes.io/projected/47f67a54-7bd4-4c38-b6d3-1e20d963aa1b-kube-api-access-5cn6c\") pod \"payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk\" (UID: \"47f67a54-7bd4-4c38-b6d3-1e20d963aa1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" Oct 30 19:49:19.602100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.602076 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f67a54-7bd4-4c38-b6d3-1e20d963aa1b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk\" (UID: \"47f67a54-7bd4-4c38-b6d3-1e20d963aa1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" Oct 30 19:49:19.602787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.602758 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:19.620496 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.620461 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cn6c\" (UniqueName: \"kubernetes.io/projected/47f67a54-7bd4-4c38-b6d3-1e20d963aa1b-kube-api-access-5cn6c\") pod \"payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk\" (UID: \"47f67a54-7bd4-4c38-b6d3-1e20d963aa1b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" Oct 30 19:49:19.717170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.716418 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b6a1c99-8256-4957-bd95-dc3f18f1d147-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q\" (UID: \"8b6a1c99-8256-4957-bd95-dc3f18f1d147\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" Oct 30 19:49:19.717170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.716657 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzh8t\" (UniqueName: \"kubernetes.io/projected/8b6a1c99-8256-4957-bd95-dc3f18f1d147-kube-api-access-rzh8t\") pod \"payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q\" (UID: \"8b6a1c99-8256-4957-bd95-dc3f18f1d147\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" Oct 30 19:49:19.738216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.717979 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b6a1c99-8256-4957-bd95-dc3f18f1d147-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q\" (UID: \"8b6a1c99-8256-4957-bd95-dc3f18f1d147\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" Oct 30 19:49:19.821453 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.792440 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" Oct 30 19:49:19.969080 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:19.968991 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzh8t\" (UniqueName: \"kubernetes.io/projected/8b6a1c99-8256-4957-bd95-dc3f18f1d147-kube-api-access-rzh8t\") pod \"payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q\" (UID: \"8b6a1c99-8256-4957-bd95-dc3f18f1d147\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" Oct 30 19:49:20.123183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:20.123138 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" Oct 30 19:49:20.176945 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:20.176833 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:20.269519 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:49:20.269487 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f67a54_7bd4_4c38_b6d3_1e20d963aa1b.slice/crio-bab91d27024f2d00d61810c0e222515045069d2e5aa6d58a21237ae3e72afb6f WatchSource:0}: Error finding container bab91d27024f2d00d61810c0e222515045069d2e5aa6d58a21237ae3e72afb6f: Status 404 returned error can't find the container with id bab91d27024f2d00d61810c0e222515045069d2e5aa6d58a21237ae3e72afb6f Oct 30 19:49:20.276422 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:20.275450 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk"] Oct 30 19:49:20.503452 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:49:20.503413 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b6a1c99_8256_4957_bd95_dc3f18f1d147.slice/crio-3255c1f96c0ab9c65ac3037845bfbfdea29326fcceb101d4299af6c191d306c7 WatchSource:0}: Error finding container 3255c1f96c0ab9c65ac3037845bfbfdea29326fcceb101d4299af6c191d306c7: Status 404 returned error can't find the container with id 3255c1f96c0ab9c65ac3037845bfbfdea29326fcceb101d4299af6c191d306c7 Oct 30 19:49:20.505795 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:20.505764 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q"] Oct 30 19:49:20.951010 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:20.950915 5091 scope.go:122] "RemoveContainer" containerID="34e5a6bef5345a8470f164411ff0084cfc0735ce9e6afacf959ce25af3517818" Oct 30 19:49:21.041096 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.041047 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" event={"ID":"8b6a1c99-8256-4957-bd95-dc3f18f1d147","Type":"ContainerStarted","Data":"f138aad2f627d681920365b8a724e05dc462c1b8763eac8ac7e42e1ee619cd0d"} Oct 30 19:49:21.041283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.041114 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" event={"ID":"8b6a1c99-8256-4957-bd95-dc3f18f1d147","Type":"ContainerStarted","Data":"3255c1f96c0ab9c65ac3037845bfbfdea29326fcceb101d4299af6c191d306c7"} Oct 30 19:49:21.041283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.041178 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" Oct 30 19:49:21.042850 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.042820 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" event={"ID":"47f67a54-7bd4-4c38-b6d3-1e20d963aa1b","Type":"ContainerStarted","Data":"7eed83054d70b439d876294e47479fc7b3d30d689ce1e4c3708ee9e724a62f12"} Oct 30 19:49:21.042990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.042871 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" event={"ID":"47f67a54-7bd4-4c38-b6d3-1e20d963aa1b","Type":"ContainerStarted","Data":"bab91d27024f2d00d61810c0e222515045069d2e5aa6d58a21237ae3e72afb6f"} Oct 30 19:49:21.042990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.042899 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" Oct 30 19:49:21.044636 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.044613 5091 generic.go:356] "Generic (PLEG): container finished" podID="4b3c46ef-f755-4462-9dae-250f052538b5" containerID="0a674f1877120eb81273084b00af32f9b0af4bd6298e08853ded33b74a2e5af4" exitCode=0 Oct 30 19:49:21.044739 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.044644 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" event={"ID":"4b3c46ef-f755-4462-9dae-250f052538b5","Type":"ContainerDied","Data":"0a674f1877120eb81273084b00af32f9b0af4bd6298e08853ded33b74a2e5af4"} Oct 30 19:49:21.044989 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.044975 5091 scope.go:122] "RemoveContainer" containerID="0a674f1877120eb81273084b00af32f9b0af4bd6298e08853ded33b74a2e5af4" Oct 30 19:49:21.046700 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.046666 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:49:21.046796 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.046750 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:49:21.046840 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.046806 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:49:21.076184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.074450 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" podStartSLOduration=2.074431548 podStartE2EDuration="2.074431548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:21.066494522 +0000 UTC m=+1080.761702480" watchObservedRunningTime="2026-10-30 19:49:21.074431548 +0000 UTC m=+1080.769639507" Oct 30 19:49:21.144876 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.132051 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" podStartSLOduration=2.132036565 podStartE2EDuration="2.132036565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:21.131453871 +0000 UTC m=+1080.826661829" watchObservedRunningTime="2026-10-30 19:49:21.132036565 +0000 UTC m=+1080.827244526" Oct 30 19:49:21.220621 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.220439 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:21.842581 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:21.842550 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:22.018102 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:22.018061 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:22.077167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:22.072433 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" event={"ID":"4b3c46ef-f755-4462-9dae-250f052538b5","Type":"ContainerStarted","Data":"9dc97d61a4bcdbf458a1bcb4cf618b3df493cef760db9c27b9d36f19a3470dc7"} Oct 30 19:49:22.096529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:22.096430 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:49:22.096710 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:22.096635 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:49:22.096710 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:22.096682 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" podUID="4b3c46ef-f755-4462-9dae-250f052538b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 19:49:22.394217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:22.389108 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:23.097563 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:23.097535 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:49:23.098042 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:23.098012 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" podUID="4b3c46ef-f755-4462-9dae-250f052538b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 19:49:23.125015 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:23.124987 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:23.294954 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:23.294914 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:23.475748 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:23.475658 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:24.097302 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:24.097268 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:49:24.097483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:24.097328 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" podUID="4b3c46ef-f755-4462-9dae-250f052538b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 19:49:24.430815 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:24.430731 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:25.097135 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:25.097102 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:49:25.097331 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:25.097173 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" podUID="4b3c46ef-f755-4462-9dae-250f052538b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 19:49:25.468613 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:25.468514 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:26.096731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:26.096696 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:49:26.096912 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:26.096750 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" podUID="4b3c46ef-f755-4462-9dae-250f052538b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 19:49:26.381813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:26.381698 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:26.570169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:26.566676 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:26.735219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:26.733917 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:26.982624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:26.982588 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj"] Oct 30 19:49:26.989582 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:26.989510 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2761c7-g78vj"] Oct 30 19:49:27.000230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:27.000193 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:27.097264 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:27.097230 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body= Oct 30 19:49:27.097437 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:27.097299 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" podUID="4b3c46ef-f755-4462-9dae-250f052538b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" Oct 30 19:49:27.169170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:27.166525 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:27.529274 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:27.529244 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:28.032456 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:28.032412 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld"] Oct 30 19:49:28.037914 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:28.037882 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd8196c7-j48ld"] Oct 30 19:49:28.133026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:28.132993 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:49:28.133578 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:28.133552 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:49:28.134468 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:28.134440 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:49:28.370137 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:28.370052 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:28.503432 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:28.499429 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72865e5d-d9f3-49f8-a3a5-beedd320d8e9" path="/var/lib/kubelet/pods/72865e5d-d9f3-49f8-a3a5-beedd320d8e9/volumes" Oct 30 19:49:28.503432 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:28.500363 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2af1909-5f46-40e4-9efb-9e7e8a3c52f6" path="/var/lib/kubelet/pods/d2af1909-5f46-40e4-9efb-9e7e8a3c52f6/volumes" Oct 30 19:49:28.629843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:28.629751 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:29.701226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:29.701138 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:31.092169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:31.091873 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:32.074336 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:32.074305 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" Oct 30 19:49:32.078860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:32.078830 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" Oct 30 19:49:32.318168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:32.294741 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:32.628508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:32.628475 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:33.400481 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.400292 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:33.553879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.553838 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:33.808721 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.808688 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t"] Oct 30 19:49:33.815220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.815189 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:33.826045 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.826018 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-040623-openshift-default-dockercfg-6n4pg\"" Oct 30 19:49:33.832159 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.828700 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t"] Oct 30 19:49:33.869878 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.869841 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-credential-socket\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:33.870078 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.869907 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-istio-data\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:33.870078 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.870064 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54f332bc-3668-4f09-a354-697e1610cd3d-istio-token\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:33.870183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.870128 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54f332bc-3668-4f09-a354-697e1610cd3d-istio-podinfo\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:33.870224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.870192 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54f332bc-3668-4f09-a354-697e1610cd3d-istiod-ca-cert\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:33.870288 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.870273 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-istio-envoy\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:33.870365 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.870349 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-workload-certs\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:33.870403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.870391 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-workload-socket\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:33.870471 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.870455 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qk9m\" (UniqueName: \"kubernetes.io/projected/54f332bc-3668-4f09-a354-697e1610cd3d-kube-api-access-6qk9m\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:33.926165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:33.925884 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:34.004169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.002116 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54f332bc-3668-4f09-a354-697e1610cd3d-istio-token\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.004169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.002220 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54f332bc-3668-4f09-a354-697e1610cd3d-istio-podinfo\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.004169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.002270 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54f332bc-3668-4f09-a354-697e1610cd3d-istiod-ca-cert\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.004169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.002355 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-istio-envoy\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.004169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.002431 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-workload-certs\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.004169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.002468 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-workload-socket\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.004169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.002537 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qk9m\" (UniqueName: \"kubernetes.io/projected/54f332bc-3668-4f09-a354-697e1610cd3d-kube-api-access-6qk9m\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.004169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.002621 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-credential-socket\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.004169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.002666 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-istio-data\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.004169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.003119 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-istio-data\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.004863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.004481 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-workload-certs\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.005170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.005114 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54f332bc-3668-4f09-a354-697e1610cd3d-istiod-ca-cert\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.026172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.025928 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-workload-socket\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.026361 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.026185 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-credential-socket\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.094213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.077405 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54f332bc-3668-4f09-a354-697e1610cd3d-istio-token\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.094213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.093767 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-istio-envoy\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.104169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.095709 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54f332bc-3668-4f09-a354-697e1610cd3d-istio-podinfo\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.116596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.106588 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qk9m\" (UniqueName: \"kubernetes.io/projected/54f332bc-3668-4f09-a354-697e1610cd3d-kube-api-access-6qk9m\") pod \"e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.134179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.128616 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:34.158169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.157894 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:34.610854 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:49:34.610826 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54f332bc_3668_4f09_a354_697e1610cd3d.slice/crio-e7ad7f669c1601e6647ad82b365d4f2df50b522969f2e6c3ed7103a51ed106d2 WatchSource:0}: Error finding container e7ad7f669c1601e6647ad82b365d4f2df50b522969f2e6c3ed7103a51ed106d2: Status 404 returned error can't find the container with id e7ad7f669c1601e6647ad82b365d4f2df50b522969f2e6c3ed7103a51ed106d2 Oct 30 19:49:34.611666 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.611642 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t"] Oct 30 19:49:34.613711 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.613680 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:49:34.613773 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.613748 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:49:34.613815 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.613779 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:49:34.652180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.637547 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:34.920877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.920550 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:34.974450 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.974411 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g"] Oct 30 19:49:34.975316 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:34.975276 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" podUID="52104831-cd64-4eea-b4ed-a389c06a4c30" containerName="istio-proxy" containerID="cri-o://1cc7d720f9425bffbd5e170bfb0c7b656f9f78989c508edae8185e20dadfc9ad" gracePeriod=30 Oct 30 19:49:35.119230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:35.116855 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:35.340223 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:35.340185 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" event={"ID":"54f332bc-3668-4f09-a354-697e1610cd3d","Type":"ContainerStarted","Data":"1a8ce22bc4efc48cdfbbde310e448600f6ed47f295b68d59fb5fbf7295f5bb3d"} Oct 30 19:49:35.340422 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:35.340248 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" event={"ID":"54f332bc-3668-4f09-a354-697e1610cd3d","Type":"ContainerStarted","Data":"e7ad7f669c1601e6647ad82b365d4f2df50b522969f2e6c3ed7103a51ed106d2"} Oct 30 19:49:35.371970 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:35.371910 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" podStartSLOduration=2.3718946929999998 podStartE2EDuration="2.371894693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:35.368415686 +0000 UTC m=+1095.063623646" watchObservedRunningTime="2026-10-30 19:49:35.371894693 +0000 UTC m=+1095.067102651" Oct 30 19:49:35.803578 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:35.803541 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:35.947169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:35.947077 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:36.136202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.136122 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:36.167004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.166972 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:36.173300 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.173274 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:36.258798 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.258763 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49"] Oct 30 19:49:36.263918 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.263899 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.296198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.293985 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49"] Oct 30 19:49:36.387283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.353356 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e907e4-openshift-default-dockercfg-lkqh5\"" Oct 30 19:49:36.495208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.389641 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-envoy\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.495208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.494241 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-data\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.495208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.494292 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-credential-socket\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.495208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.494393 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-podinfo\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.495208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.494469 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-workload-socket\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.495208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.494602 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-workload-certs\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.495208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.494627 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-token\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.571170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.549095 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2rk6\" (UniqueName: \"kubernetes.io/projected/b7ab432f-a929-40f9-8ed4-042d9528a20b-kube-api-access-t2rk6\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.571170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.549168 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ab432f-a929-40f9-8ed4-042d9528a20b-istiod-ca-cert\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.608311 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.600264 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:36.736207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.723897 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-envoy\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.738119 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.738096 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-data\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.738358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.737567 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-envoy\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.738720 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.738500 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-data\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.739081 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.738892 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-credential-socket\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.739221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.738305 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-credential-socket\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.739476 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.739460 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-podinfo\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.739625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.739613 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-workload-socket\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.739857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.739845 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-workload-certs\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.739952 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.739939 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-token\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.740513 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.740271 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2rk6\" (UniqueName: \"kubernetes.io/projected/b7ab432f-a929-40f9-8ed4-042d9528a20b-kube-api-access-t2rk6\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.740677 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.740661 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ab432f-a929-40f9-8ed4-042d9528a20b-istiod-ca-cert\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.741297 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.741273 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-workload-socket\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.741437 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.741413 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ab432f-a929-40f9-8ed4-042d9528a20b-istiod-ca-cert\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.742737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.742711 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-workload-certs\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.744940 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.744910 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:36.745034 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.745009 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:49:36.745466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.745443 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-podinfo\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.748561 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.748515 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:36.757046 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.757002 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-token\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.757191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.757131 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2rk6\" (UniqueName: \"kubernetes.io/projected/b7ab432f-a929-40f9-8ed4-042d9528a20b-kube-api-access-t2rk6\") pod \"e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:36.899180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:36.898169 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:37.197800 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:49:37.197765 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7ab432f_a929_40f9_8ed4_042d9528a20b.slice/crio-31657b3c81d6f28c2583b77517b4bebeeb195941e7a15d5733b02a83c3ece41a WatchSource:0}: Error finding container 31657b3c81d6f28c2583b77517b4bebeeb195941e7a15d5733b02a83c3ece41a: Status 404 returned error can't find the container with id 31657b3c81d6f28c2583b77517b4bebeeb195941e7a15d5733b02a83c3ece41a Oct 30 19:49:37.200325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:37.200291 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:49:37.200416 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:37.200360 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:49:37.200416 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:37.200391 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:49:37.200536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:37.200504 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49"] Oct 30 19:49:37.368214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:37.367896 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:37.751220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:37.751140 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" event={"ID":"b7ab432f-a929-40f9-8ed4-042d9528a20b","Type":"ContainerStarted","Data":"be1259bb71acb8835bd67056bd1b78f202e95c8543001a0cc4482a20efd6152a"} Oct 30 19:49:37.751415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:37.751251 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" event={"ID":"b7ab432f-a929-40f9-8ed4-042d9528a20b","Type":"ContainerStarted","Data":"31657b3c81d6f28c2583b77517b4bebeeb195941e7a15d5733b02a83c3ece41a"} Oct 30 19:49:37.800497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:37.800447 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" podStartSLOduration=1.8004319020000001 podStartE2EDuration="1.800431902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:37.799895167 +0000 UTC m=+1097.495103124" watchObservedRunningTime="2026-10-30 19:49:37.800431902 +0000 UTC m=+1097.495639937" Oct 30 19:49:37.862060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:37.862027 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:37.899851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:37.899820 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:38.137273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:38.132205 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5"] Oct 30 19:49:38.137273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:38.132954 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:38.211392 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:38.211350 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99fa903d-4mhr5"] Oct 30 19:49:38.315249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:38.315214 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:38.445856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:38.445771 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a048461-6326-4544-a77f-830a1378c7bb" path="/var/lib/kubelet/pods/6a048461-6326-4544-a77f-830a1378c7bb/volumes" Oct 30 19:49:38.771174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:38.769476 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:38.906044 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:38.906016 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:39.174252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:39.174103 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:39.471220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:39.471018 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:39.768263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:39.768224 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:39.769455 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:39.769430 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:49:40.340167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.339592 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:49:40.428352 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.428272 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-data\") pod \"52104831-cd64-4eea-b4ed-a389c06a4c30\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " Oct 30 19:49:40.428352 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.428327 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-token\" (UniqueName: \"kubernetes.io/projected/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-token\") pod \"52104831-cd64-4eea-b4ed-a389c06a4c30\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " Oct 30 19:49:40.428601 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.428362 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-workload-certs\") pod \"52104831-cd64-4eea-b4ed-a389c06a4c30\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " Oct 30 19:49:40.428601 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.428404 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-credential-socket\") pod \"52104831-cd64-4eea-b4ed-a389c06a4c30\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " Oct 30 19:49:40.428601 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.428471 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52104831-cd64-4eea-b4ed-a389c06a4c30-kube-api-access-lmszr\" (UniqueName: \"kubernetes.io/projected/52104831-cd64-4eea-b4ed-a389c06a4c30-kube-api-access-lmszr\") pod \"52104831-cd64-4eea-b4ed-a389c06a4c30\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " Oct 30 19:49:40.428601 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.428549 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52104831-cd64-4eea-b4ed-a389c06a4c30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52104831-cd64-4eea-b4ed-a389c06a4c30-istiod-ca-cert\") pod \"52104831-cd64-4eea-b4ed-a389c06a4c30\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " Oct 30 19:49:40.428601 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.428587 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-envoy\") pod \"52104831-cd64-4eea-b4ed-a389c06a4c30\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " Oct 30 19:49:40.428904 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.428613 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-data" pod "52104831-cd64-4eea-b4ed-a389c06a4c30" (UID: "52104831-cd64-4eea-b4ed-a389c06a4c30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:40.428904 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.428736 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-workload-socket\") pod \"52104831-cd64-4eea-b4ed-a389c06a4c30\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " Oct 30 19:49:40.428904 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.428803 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-podinfo\") pod \"52104831-cd64-4eea-b4ed-a389c06a4c30\" (UID: \"52104831-cd64-4eea-b4ed-a389c06a4c30\") " Oct 30 19:49:40.428904 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.428823 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-credential-socket" pod "52104831-cd64-4eea-b4ed-a389c06a4c30" (UID: "52104831-cd64-4eea-b4ed-a389c06a4c30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:40.429782 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.429756 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:40.429909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.429785 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:40.429909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.429891 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-workload-socket" pod "52104831-cd64-4eea-b4ed-a389c06a4c30" (UID: "52104831-cd64-4eea-b4ed-a389c06a4c30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:40.430010 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.429958 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-workload-certs" pod "52104831-cd64-4eea-b4ed-a389c06a4c30" (UID: "52104831-cd64-4eea-b4ed-a389c06a4c30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:40.430135 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.430111 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52104831-cd64-4eea-b4ed-a389c06a4c30-istiod-ca-cert" pod "52104831-cd64-4eea-b4ed-a389c06a4c30" (UID: "52104831-cd64-4eea-b4ed-a389c06a4c30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:49:40.432867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.431894 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-podinfo" pod "52104831-cd64-4eea-b4ed-a389c06a4c30" (UID: "52104831-cd64-4eea-b4ed-a389c06a4c30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:49:40.447334 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.447306 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52104831-cd64-4eea-b4ed-a389c06a4c30-kube-api-access-lmszr" pod "52104831-cd64-4eea-b4ed-a389c06a4c30" (UID: "52104831-cd64-4eea-b4ed-a389c06a4c30"). InnerVolumeSpecName "kube-api-access-lmszr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:40.447456 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.447418 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-envoy" pod "52104831-cd64-4eea-b4ed-a389c06a4c30" (UID: "52104831-cd64-4eea-b4ed-a389c06a4c30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:49:40.449817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.449791 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-token" pod "52104831-cd64-4eea-b4ed-a389c06a4c30" (UID: "52104831-cd64-4eea-b4ed-a389c06a4c30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:49:40.532001 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.531945 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:40.532001 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.531976 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:40.532001 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.531990 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:40.532001 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.532007 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:40.532347 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.532022 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmszr\" (UniqueName: \"kubernetes.io/projected/52104831-cd64-4eea-b4ed-a389c06a4c30-kube-api-access-lmszr\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:40.532347 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.532037 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52104831-cd64-4eea-b4ed-a389c06a4c30-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:40.532347 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.532051 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52104831-cd64-4eea-b4ed-a389c06a4c30-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:49:40.679175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.670734 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:40.877557 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.877520 5091 generic.go:356] "Generic (PLEG): container finished" podID="52104831-cd64-4eea-b4ed-a389c06a4c30" containerID="1cc7d720f9425bffbd5e170bfb0c7b656f9f78989c508edae8185e20dadfc9ad" exitCode=0 Oct 30 19:49:40.890604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.890568 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" Oct 30 19:49:40.894299 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.892231 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" event={"ID":"52104831-cd64-4eea-b4ed-a389c06a4c30","Type":"ContainerDied","Data":"1cc7d720f9425bffbd5e170bfb0c7b656f9f78989c508edae8185e20dadfc9ad"} Oct 30 19:49:40.894299 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.892306 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g" event={"ID":"52104831-cd64-4eea-b4ed-a389c06a4c30","Type":"ContainerDied","Data":"d829a08db98820c7958bcd94b69e0e2872bf4b151b4a025cf3dc475ffd320578"} Oct 30 19:49:40.894299 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.892340 5091 scope.go:122] "RemoveContainer" containerID="1cc7d720f9425bffbd5e170bfb0c7b656f9f78989c508edae8185e20dadfc9ad" Oct 30 19:49:40.916711 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.916428 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:40.925519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.925500 5091 scope.go:122] "RemoveContainer" containerID="687c8174c37d688c6081946d8b60ec6cddd9486ff6ea25c5cfe7dac623e7df9e" Oct 30 19:49:40.931082 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.931026 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g"] Oct 30 19:49:40.937622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.937605 5091 scope.go:122] "RemoveContainer" containerID="1cc7d720f9425bffbd5e170bfb0c7b656f9f78989c508edae8185e20dadfc9ad" Oct 30 19:49:40.937880 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:49:40.937860 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cc7d720f9425bffbd5e170bfb0c7b656f9f78989c508edae8185e20dadfc9ad\": container with ID starting with 1cc7d720f9425bffbd5e170bfb0c7b656f9f78989c508edae8185e20dadfc9ad not found: ID does not exist" containerID="1cc7d720f9425bffbd5e170bfb0c7b656f9f78989c508edae8185e20dadfc9ad" Oct 30 19:49:40.937963 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.937889 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cc7d720f9425bffbd5e170bfb0c7b656f9f78989c508edae8185e20dadfc9ad"} err="failed to get container status \"1cc7d720f9425bffbd5e170bfb0c7b656f9f78989c508edae8185e20dadfc9ad\": rpc error: code = NotFound desc = could not find container \"1cc7d720f9425bffbd5e170bfb0c7b656f9f78989c508edae8185e20dadfc9ad\": container with ID starting with 1cc7d720f9425bffbd5e170bfb0c7b656f9f78989c508edae8185e20dadfc9ad not found: ID does not exist" Oct 30 19:49:40.937963 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.937911 5091 scope.go:122] "RemoveContainer" containerID="687c8174c37d688c6081946d8b60ec6cddd9486ff6ea25c5cfe7dac623e7df9e" Oct 30 19:49:40.938314 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:49:40.938281 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"687c8174c37d688c6081946d8b60ec6cddd9486ff6ea25c5cfe7dac623e7df9e\": container with ID starting with 687c8174c37d688c6081946d8b60ec6cddd9486ff6ea25c5cfe7dac623e7df9e not found: ID does not exist" containerID="687c8174c37d688c6081946d8b60ec6cddd9486ff6ea25c5cfe7dac623e7df9e" Oct 30 19:49:40.938415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.938310 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"687c8174c37d688c6081946d8b60ec6cddd9486ff6ea25c5cfe7dac623e7df9e"} err="failed to get container status \"687c8174c37d688c6081946d8b60ec6cddd9486ff6ea25c5cfe7dac623e7df9e\": rpc error: code = NotFound desc = could not find container \"687c8174c37d688c6081946d8b60ec6cddd9486ff6ea25c5cfe7dac623e7df9e\": container with ID starting with 687c8174c37d688c6081946d8b60ec6cddd9486ff6ea25c5cfe7dac623e7df9e not found: ID does not exist" Oct 30 19:49:40.940274 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:40.940252 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-477fd9c0-openshift-default-647479cffc-8sl5g"] Oct 30 19:49:41.897175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:41.897115 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:41.932624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:41.932592 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w"] Oct 30 19:49:41.933653 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:41.933633 5091 cpu_manager.go:414] "containerMap: removing container" podUID="52104831-cd64-4eea-b4ed-a389c06a4c30" containerName="istio-proxy" Oct 30 19:49:41.933707 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:41.933657 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52104831-cd64-4eea-b4ed-a389c06a4c30" containerName="istio-proxy" Oct 30 19:49:41.933707 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:41.933685 5091 cpu_manager.go:414] "containerMap: removing container" podUID="52104831-cd64-4eea-b4ed-a389c06a4c30" containerName="istio-proxy" Oct 30 19:49:41.933707 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:41.933694 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52104831-cd64-4eea-b4ed-a389c06a4c30" containerName="istio-proxy" Oct 30 19:49:41.933886 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:41.933874 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="52104831-cd64-4eea-b4ed-a389c06a4c30" containerName="istio-proxy" Oct 30 19:49:41.933922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:41.933893 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="52104831-cd64-4eea-b4ed-a389c06a4c30" containerName="istio-proxy" Oct 30 19:49:41.939668 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:41.939650 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" Oct 30 19:49:41.977295 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:41.977258 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w"] Oct 30 19:49:41.982084 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:41.982022 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e907e4\"" Oct 30 19:49:41.989437 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:41.989407 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2"] Oct 30 19:49:41.995274 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:41.995253 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" Oct 30 19:49:42.003199 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.003176 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2"] Oct 30 19:49:42.023119 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.022911 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e907e4-dockercfg-ndrbk\"" Oct 30 19:49:42.224226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.117321 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5jsp\" (UniqueName: \"kubernetes.io/projected/85f732fb-5e96-4788-9136-70a9011d8265-kube-api-access-r5jsp\") pod \"payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w\" (UID: \"85f732fb-5e96-4788-9136-70a9011d8265\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" Oct 30 19:49:42.224226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.117514 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a56e07-b00d-4103-8e93-619e8af7abb5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e907e4-654594658b-fw8n2\" (UID: \"25a56e07-b00d-4103-8e93-619e8af7abb5\") " pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" Oct 30 19:49:42.224226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.117557 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f732fb-5e96-4788-9136-70a9011d8265-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w\" (UID: \"85f732fb-5e96-4788-9136-70a9011d8265\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" Oct 30 19:49:42.224226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.117764 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndtm2\" (UniqueName: \"kubernetes.io/projected/25a56e07-b00d-4103-8e93-619e8af7abb5-kube-api-access-ndtm2\") pod \"payload-processing-e2e-api-a-e907e4-654594658b-fw8n2\" (UID: \"25a56e07-b00d-4103-8e93-619e8af7abb5\") " pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" Oct 30 19:49:42.224226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.133333 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:42.254522 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.254477 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndtm2\" (UniqueName: \"kubernetes.io/projected/25a56e07-b00d-4103-8e93-619e8af7abb5-kube-api-access-ndtm2\") pod \"payload-processing-e2e-api-a-e907e4-654594658b-fw8n2\" (UID: \"25a56e07-b00d-4103-8e93-619e8af7abb5\") " pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" Oct 30 19:49:42.254940 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.254912 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5jsp\" (UniqueName: \"kubernetes.io/projected/85f732fb-5e96-4788-9136-70a9011d8265-kube-api-access-r5jsp\") pod \"payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w\" (UID: \"85f732fb-5e96-4788-9136-70a9011d8265\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" Oct 30 19:49:42.255242 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.255214 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a56e07-b00d-4103-8e93-619e8af7abb5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e907e4-654594658b-fw8n2\" (UID: \"25a56e07-b00d-4103-8e93-619e8af7abb5\") " pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" Oct 30 19:49:42.255400 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.255384 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f732fb-5e96-4788-9136-70a9011d8265-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w\" (UID: \"85f732fb-5e96-4788-9136-70a9011d8265\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" Oct 30 19:49:42.256899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.256876 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f732fb-5e96-4788-9136-70a9011d8265-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w\" (UID: \"85f732fb-5e96-4788-9136-70a9011d8265\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" Oct 30 19:49:42.291568 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.277269 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a56e07-b00d-4103-8e93-619e8af7abb5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e907e4-654594658b-fw8n2\" (UID: \"25a56e07-b00d-4103-8e93-619e8af7abb5\") " pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" Oct 30 19:49:42.354986 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.354953 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5jsp\" (UniqueName: \"kubernetes.io/projected/85f732fb-5e96-4788-9136-70a9011d8265-kube-api-access-r5jsp\") pod \"payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w\" (UID: \"85f732fb-5e96-4788-9136-70a9011d8265\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" Oct 30 19:49:42.359621 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.359558 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndtm2\" (UniqueName: \"kubernetes.io/projected/25a56e07-b00d-4103-8e93-619e8af7abb5-kube-api-access-ndtm2\") pod \"payload-processing-e2e-api-a-e907e4-654594658b-fw8n2\" (UID: \"25a56e07-b00d-4103-8e93-619e8af7abb5\") " pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" Oct 30 19:49:42.442569 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.442535 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52104831-cd64-4eea-b4ed-a389c06a4c30" path="/var/lib/kubelet/pods/52104831-cd64-4eea-b4ed-a389c06a4c30/volumes" Oct 30 19:49:42.541038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.541007 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:42.586508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.586480 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" Oct 30 19:49:42.623814 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.623470 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" Oct 30 19:49:42.722168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.721614 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:42.898527 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.898342 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq"] Oct 30 19:49:42.905013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.904953 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:42.908914 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.908886 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:42.923518 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.922518 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq"] Oct 30 19:49:42.923697 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:42.923600 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-4f2325-openshift-default-dockercfg-kc8tb\"" Oct 30 19:49:43.008443 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.008395 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-istio-envoy\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.008740 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.008715 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-workload-certs\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.009051 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.009028 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-workload-socket\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.009220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.009202 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111283d7-9f88-4151-beee-283607f0e13b-istio-token\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.009436 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.009418 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111283d7-9f88-4151-beee-283607f0e13b-istiod-ca-cert\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.009621 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.009602 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xjlx\" (UniqueName: \"kubernetes.io/projected/111283d7-9f88-4151-beee-283607f0e13b-kube-api-access-8xjlx\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.009745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.009730 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-credential-socket\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.009892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.009879 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111283d7-9f88-4151-beee-283607f0e13b-istio-podinfo\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.010105 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.010019 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-istio-data\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.151219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.150936 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-istio-envoy\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.151719 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.151691 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-workload-certs\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.151893 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.151868 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-workload-socket\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.152294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.152263 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111283d7-9f88-4151-beee-283607f0e13b-istio-token\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.153450 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.153426 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-workload-certs\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.153665 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.153622 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111283d7-9f88-4151-beee-283607f0e13b-istiod-ca-cert\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.154358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.154338 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xjlx\" (UniqueName: \"kubernetes.io/projected/111283d7-9f88-4151-beee-283607f0e13b-kube-api-access-8xjlx\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.164638 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.164616 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-credential-socket\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.165438 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.165419 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111283d7-9f88-4151-beee-283607f0e13b-istio-podinfo\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.165620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.165602 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-istio-data\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.165865 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.154181 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-workload-socket\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.165958 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.165334 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-credential-socket\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.166030 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.161665 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-istio-envoy\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.194110 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.164542 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111283d7-9f88-4151-beee-283607f0e13b-istiod-ca-cert\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.194110 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.178526 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-istio-data\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.194110 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.179908 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:43.194110 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.184232 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xjlx\" (UniqueName: \"kubernetes.io/projected/111283d7-9f88-4151-beee-283607f0e13b-kube-api-access-8xjlx\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.218082 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.217791 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w"] Oct 30 19:49:43.227034 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.224412 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111283d7-9f88-4151-beee-283607f0e13b-istio-podinfo\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.254276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.254230 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9"] Oct 30 19:49:43.280721 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.280687 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f79cb4a7-njft9"] Oct 30 19:49:43.280939 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.280924 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2"] Oct 30 19:49:43.298815 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:49:43.298778 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25a56e07_b00d_4103_8e93_619e8af7abb5.slice/crio-eb0bbedf2ef74a9ee9aa6ebd7080ad98a8fdcc85b0a102193ef4638713fa7ac6 WatchSource:0}: Error finding container eb0bbedf2ef74a9ee9aa6ebd7080ad98a8fdcc85b0a102193ef4638713fa7ac6: Status 404 returned error can't find the container with id eb0bbedf2ef74a9ee9aa6ebd7080ad98a8fdcc85b0a102193ef4638713fa7ac6 Oct 30 19:49:43.327226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.313788 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111283d7-9f88-4151-beee-283607f0e13b-istio-token\") pod \"e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.558200 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.553340 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:43.585414 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.585382 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:43.978318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.977723 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq"] Oct 30 19:49:43.988173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.985319 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" event={"ID":"85f732fb-5e96-4788-9136-70a9011d8265","Type":"ContainerStarted","Data":"4f88180b03eb8ebba57b1b17cbd9465fcb4f1a9049f6dea31f618de12649ee82"} Oct 30 19:49:43.988173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.985384 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" Oct 30 19:49:43.988173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.985407 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" event={"ID":"85f732fb-5e96-4788-9136-70a9011d8265","Type":"ContainerStarted","Data":"3d136f1f165c6cdeba530198cee3919231d799b4cedd6d7f527caa2ce6bbca87"} Oct 30 19:49:43.988173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.987646 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" event={"ID":"25a56e07-b00d-4103-8e93-619e8af7abb5","Type":"ContainerStarted","Data":"b420648d0f8cfb717edd21cfbb86f6c04f91b217490ed56cf607dbba1ea4cffc"} Oct 30 19:49:43.988173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.987688 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" event={"ID":"25a56e07-b00d-4103-8e93-619e8af7abb5","Type":"ContainerStarted","Data":"eb0bbedf2ef74a9ee9aa6ebd7080ad98a8fdcc85b0a102193ef4638713fa7ac6"} Oct 30 19:49:43.988548 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:43.988455 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" Oct 30 19:49:43.997873 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:49:43.997837 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111283d7_9f88_4151_beee_283607f0e13b.slice/crio-1221847e7f54fc546f72fd434fb8aef0e82b26f0ff7dca51c80a9e0d5c3f0866 WatchSource:0}: Error finding container 1221847e7f54fc546f72fd434fb8aef0e82b26f0ff7dca51c80a9e0d5c3f0866: Status 404 returned error can't find the container with id 1221847e7f54fc546f72fd434fb8aef0e82b26f0ff7dca51c80a9e0d5c3f0866 Oct 30 19:49:44.001376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:44.001343 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:49:44.001465 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:44.001431 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:49:44.001547 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:44.001463 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:49:44.024850 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:44.024809 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" podStartSLOduration=3.02479676 podStartE2EDuration="3.02479676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:44.01915946 +0000 UTC m=+1103.714367424" watchObservedRunningTime="2026-10-30 19:49:44.02479676 +0000 UTC m=+1103.720004715" Oct 30 19:49:44.042863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:44.042649 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" podStartSLOduration=3.042637025 podStartE2EDuration="3.042637025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:44.040984436 +0000 UTC m=+1103.736192394" watchObservedRunningTime="2026-10-30 19:49:44.042637025 +0000 UTC m=+1103.737844982" Oct 30 19:49:44.240241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:44.238984 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:44.442346 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:44.442307 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c069c460-436f-46d1-a508-6e094e66d9b2" path="/var/lib/kubelet/pods/c069c460-436f-46d1-a508-6e094e66d9b2/volumes" Oct 30 19:49:44.998830 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:44.998787 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" event={"ID":"111283d7-9f88-4151-beee-283607f0e13b","Type":"ContainerStarted","Data":"c6dc220bc9f42c5e2bac62efc5eda85423de21fecc13cb7c8d66fda483812b67"} Oct 30 19:49:44.999260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:44.998850 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" event={"ID":"111283d7-9f88-4151-beee-283607f0e13b","Type":"ContainerStarted","Data":"1221847e7f54fc546f72fd434fb8aef0e82b26f0ff7dca51c80a9e0d5c3f0866"} Oct 30 19:49:45.040421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:45.040375 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" podStartSLOduration=3.040361811 podStartE2EDuration="3.040361811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:45.037950732 +0000 UTC m=+1104.733158704" watchObservedRunningTime="2026-10-30 19:49:45.040361811 +0000 UTC m=+1104.735569768" Oct 30 19:49:45.553922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:45.553881 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:45.560703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:45.560667 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:45.607847 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:45.607816 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:45.978141 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:49:45.978054 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:49:46.012231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:46.012201 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:46.013002 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:46.012974 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:49:46.626410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:46.625878 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:47.674166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:47.656777 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:47.870072 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:47.870036 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:48.057447 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:48.057414 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:48.649680 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:48.649643 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:48.941731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:48.941634 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:49.210745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:49.210655 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:49.427419 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:49.427386 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:49.728891 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:49.728698 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:49.937298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:49.936994 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:50.518807 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:50.518772 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:50.683069 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:50.683031 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:51.041344 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:51.041313 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47"] Oct 30 19:49:51.047935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:51.047907 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a27159b9-95c47"] Oct 30 19:49:51.897098 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:51.876924 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:52.067194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:52.067155 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:52.325378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:52.325339 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:52.441901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:52.441865 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4da59d7-41f9-4a9d-bef6-1a5288bb88aa" path="/var/lib/kubelet/pods/a4da59d7-41f9-4a9d-bef6-1a5288bb88aa/volumes" Oct 30 19:49:52.474929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:52.474851 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:52.719843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:52.719729 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:53.330962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:53.330916 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:53.685969 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:53.685882 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:53.870655 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:53.870617 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:54.984110 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:54.984072 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:55.002716 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.002689 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" Oct 30 19:49:55.145312 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.145279 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l"] Oct 30 19:49:55.151600 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.151574 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" Oct 30 19:49:55.159044 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.159017 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2beb7b-dockercfg-hcxp6\"" Oct 30 19:49:55.160060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.159702 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:55.165371 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.165347 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l"] Oct 30 19:49:55.200914 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.200884 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0f7a23-6124-419b-9b33-11c6ac5f4276-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l\" (UID: \"fc0f7a23-6124-419b-9b33-11c6ac5f4276\") " pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" Oct 30 19:49:55.201049 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.201033 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdnqb\" (UniqueName: \"kubernetes.io/projected/fc0f7a23-6124-419b-9b33-11c6ac5f4276-kube-api-access-pdnqb\") pod \"payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l\" (UID: \"fc0f7a23-6124-419b-9b33-11c6ac5f4276\") " pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" Oct 30 19:49:55.219608 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.219571 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2beb7b\"" Oct 30 19:49:55.395173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.357478 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0f7a23-6124-419b-9b33-11c6ac5f4276-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l\" (UID: \"fc0f7a23-6124-419b-9b33-11c6ac5f4276\") " pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" Oct 30 19:49:55.395173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.359472 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0f7a23-6124-419b-9b33-11c6ac5f4276-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l\" (UID: \"fc0f7a23-6124-419b-9b33-11c6ac5f4276\") " pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" Oct 30 19:49:55.395173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.380865 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdnqb\" (UniqueName: \"kubernetes.io/projected/fc0f7a23-6124-419b-9b33-11c6ac5f4276-kube-api-access-pdnqb\") pod \"payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l\" (UID: \"fc0f7a23-6124-419b-9b33-11c6ac5f4276\") " pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" Oct 30 19:49:55.415171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.414602 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:55.452176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.440893 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdnqb\" (UniqueName: \"kubernetes.io/projected/fc0f7a23-6124-419b-9b33-11c6ac5f4276-kube-api-access-pdnqb\") pod \"payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l\" (UID: \"fc0f7a23-6124-419b-9b33-11c6ac5f4276\") " pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" Oct 30 19:49:55.525780 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.499263 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" Oct 30 19:49:55.591603 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.591565 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:55.837328 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:49:55.837298 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0f7a23_6124_419b_9b33_11c6ac5f4276.slice/crio-4086d0e1fb888a3fc106a1cea3921a831e8d8d7461f306ad439e0d286ee702a9 WatchSource:0}: Error finding container 4086d0e1fb888a3fc106a1cea3921a831e8d8d7461f306ad439e0d286ee702a9: Status 404 returned error can't find the container with id 4086d0e1fb888a3fc106a1cea3921a831e8d8d7461f306ad439e0d286ee702a9 Oct 30 19:49:55.840421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:55.840309 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l"] Oct 30 19:49:56.015339 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:56.015315 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" Oct 30 19:49:56.181948 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:56.181869 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:56.367628 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:56.366628 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:56.422273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:56.422240 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" event={"ID":"fc0f7a23-6124-419b-9b33-11c6ac5f4276","Type":"ContainerStarted","Data":"a552591f13f6d618212700cfe53b0bba297adcacef6fa185c09b89416bfd14a8"} Oct 30 19:49:56.422474 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:56.422289 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" event={"ID":"fc0f7a23-6124-419b-9b33-11c6ac5f4276","Type":"ContainerStarted","Data":"4086d0e1fb888a3fc106a1cea3921a831e8d8d7461f306ad439e0d286ee702a9"} Oct 30 19:49:56.422474 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:56.422320 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" Oct 30 19:49:56.447478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:56.447368 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" podStartSLOduration=1.4473549239999999 podStartE2EDuration="1.447354924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:49:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:49:56.443722973 +0000 UTC m=+1116.138930931" watchObservedRunningTime="2026-10-30 19:49:56.447354924 +0000 UTC m=+1116.142562921" Oct 30 19:49:57.030995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:57.030958 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc"] Oct 30 19:49:57.046320 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:57.046293 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87bd8f73-fp7mc"] Oct 30 19:49:57.301216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:57.296602 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:58.442490 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:58.442452 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10adce2f-b9d8-47e7-9546-a1b45813dc5b" path="/var/lib/kubelet/pods/10adce2f-b9d8-47e7-9546-a1b45813dc5b/volumes" Oct 30 19:49:58.540007 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:58.539946 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:49:59.583170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:49:59.577773 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:00.817168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:00.813880 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:01.833000 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:01.832967 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:02.029609 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:02.029580 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:03.027735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:03.027702 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh"] Oct 30 19:50:03.035905 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:03.035875 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb22e2c1-77qkh"] Oct 30 19:50:03.075157 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:03.075106 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:03.895226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:03.895188 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:04.090124 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:04.090092 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:04.473255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:04.472836 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8459a593-be25-4760-900e-bc08897d8294" path="/var/lib/kubelet/pods/8459a593-be25-4760-900e-bc08897d8294/volumes" Oct 30 19:50:05.026231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:05.026193 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:05.059370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:05.059334 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288"] Oct 30 19:50:05.063470 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:05.063447 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06cdca52-kr288"] Oct 30 19:50:06.020876 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:06.020844 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:06.445026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:06.444945 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7" path="/var/lib/kubelet/pods/8c8ad84f-80cb-4a36-8d65-f0eb2cef68c7/volumes" Oct 30 19:50:07.087170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:07.077069 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:07.435924 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:07.435854 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" Oct 30 19:50:08.134328 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:08.131003 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:09.100235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:09.100196 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq"] Oct 30 19:50:09.138493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:09.138096 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4e77306-zmxbq"] Oct 30 19:50:09.431931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:09.431848 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:09.644441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:09.644406 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:09.813127 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:09.813097 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:10.445622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:10.445584 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86" path="/var/lib/kubelet/pods/2f6b1d5a-a7c4-4d91-9c16-0330a02d4c86/volumes" Oct 30 19:50:10.962753 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:10.962716 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:11.652512 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:11.652471 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:12.015987 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:12.015941 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:13.224186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:13.224116 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:13.560026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:13.559975 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:13.755367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:13.755312 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:13.950346 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:13.950255 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:14.938977 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:14.938944 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:16.065727 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:50:16.065652 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:50:16.294020 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:16.292822 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:17.406621 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:17.406575 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:17.659238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:17.655237 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:18.433977 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:18.433946 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:19.038721 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.038684 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:19.052433 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.052384 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj"] Oct 30 19:50:19.129010 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.128981 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.242605 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.197766 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f61e4b-openshift-default-dockercfg-4p6zw\"" Oct 30 19:50:19.355634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.341816 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj"] Oct 30 19:50:19.355901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.355511 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:19.452169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.445820 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-data\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.452169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.445985 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-envoy\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.452169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.446021 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cd91a7-7d69-46a4-b213-b36fed53e50b-istiod-ca-cert\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.452169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.446046 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-podinfo\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.452169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.446124 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-token\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.452169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.446189 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-workload-certs\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.452169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.446238 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4sdf\" (UniqueName: \"kubernetes.io/projected/79cd91a7-7d69-46a4-b213-b36fed53e50b-kube-api-access-v4sdf\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.452169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.446279 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-workload-socket\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.452169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.446333 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-credential-socket\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.577170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.562166 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4sdf\" (UniqueName: \"kubernetes.io/projected/79cd91a7-7d69-46a4-b213-b36fed53e50b-kube-api-access-v4sdf\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.577170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.562233 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-workload-socket\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.577170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.562295 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-credential-socket\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.577170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.562369 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-data\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.577170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.562563 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-envoy\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.577170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.562608 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cd91a7-7d69-46a4-b213-b36fed53e50b-istiod-ca-cert\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.577170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.562636 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-podinfo\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.577170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.562720 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-token\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.577170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.562775 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-workload-certs\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.577170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.563330 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-workload-certs\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.577170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.563962 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-workload-socket\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.577170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.564266 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-credential-socket\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.577170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.564501 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-data\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.577170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.573076 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cd91a7-7d69-46a4-b213-b36fed53e50b-istiod-ca-cert\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.635483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.605193 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-envoy\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.635483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.606043 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-podinfo\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.635483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.617927 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-token\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.635483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.618778 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4sdf\" (UniqueName: \"kubernetes.io/projected/79cd91a7-7d69-46a4-b213-b36fed53e50b-kube-api-access-v4sdf\") pod \"e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:19.772430 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:19.772374 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:20.187326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:20.187191 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb"] Oct 30 19:50:20.244239 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:20.232378 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-521aca49-xpkxb"] Oct 30 19:50:20.814731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:20.813737 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df2a21ed-1e2b-415b-b8c2-a04f82d7178e" path="/var/lib/kubelet/pods/df2a21ed-1e2b-415b-b8c2-a04f82d7178e/volumes" Oct 30 19:50:20.898919 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:20.898890 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj"] Oct 30 19:50:20.905525 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:20.905494 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:20.911099 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:20.911067 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:50:20.911196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:20.911132 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:50:20.911196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:20.911184 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:50:20.992169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:20.991888 5091 scope.go:122] "RemoveContainer" containerID="cf3a6e2d4fb22549006e7e9f954659e271270eba0d65ad3f15b76982eb1c9126" Oct 30 19:50:21.653213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:21.630918 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:21.765532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:21.765434 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" event={"ID":"79cd91a7-7d69-46a4-b213-b36fed53e50b","Type":"ContainerStarted","Data":"b573b8cfcb3098eec6c22107f9d9a1bb8d6b8a941996207387539fde58fe554d"} Oct 30 19:50:21.765532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:21.765497 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" event={"ID":"79cd91a7-7d69-46a4-b213-b36fed53e50b","Type":"ContainerStarted","Data":"881218d92faff6a3bd86e343355a669d8f850c574eba4f3c64c70f37d32f7180"} Oct 30 19:50:21.773618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:21.773590 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:21.801264 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:21.801207 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" podStartSLOduration=3.8011937810000003 podStartE2EDuration="3.801193781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:21.795956202 +0000 UTC m=+1141.491164174" watchObservedRunningTime="2026-10-30 19:50:21.801193781 +0000 UTC m=+1141.496401739" Oct 30 19:50:22.054807 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.054720 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf"] Oct 30 19:50:22.071674 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.071638 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c359af9c-rk6sf"] Oct 30 19:50:22.120851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.120752 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:22.450669 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.450561 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7218f8cd-7572-479a-a136-a31fbcfab76c" path="/var/lib/kubelet/pods/7218f8cd-7572-479a-a136-a31fbcfab76c/volumes" Oct 30 19:50:22.612201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.607304 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7"] Oct 30 19:50:22.621480 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.621446 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.622175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.622123 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7"] Oct 30 19:50:22.638587 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.638558 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-913351\"/\"kube-root-ca.crt\"" Oct 30 19:50:22.644983 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.644951 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-913351\"/\"openshift-service-ca.crt\"" Oct 30 19:50:22.645111 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.645096 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-913351\"/\"default-dockercfg-nqqgz\"" Oct 30 19:50:22.645241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.645225 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-913351\"/\"ipp-route-913351-kserve-self-signed-certs\"" Oct 30 19:50:22.702901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.702808 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:22.703856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.703831 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faa3220-d981-43af-a6d2-19d841026802-tls-certs\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.703988 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.703879 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-dshm\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.704044 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.704000 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-tmp-dir\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.704089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.704048 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8tqx\" (UniqueName: \"kubernetes.io/projected/5faa3220-d981-43af-a6d2-19d841026802-kube-api-access-j8tqx\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.704159 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.704084 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-model-cache\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.704311 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.704291 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-home\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.806591 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.806245 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:50:22.806591 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.806307 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:22.847175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.815063 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faa3220-d981-43af-a6d2-19d841026802-tls-certs\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.847175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.815125 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-dshm\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.847175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.815311 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-tmp-dir\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.847175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.815366 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8tqx\" (UniqueName: \"kubernetes.io/projected/5faa3220-d981-43af-a6d2-19d841026802-kube-api-access-j8tqx\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.847175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.815405 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-model-cache\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.847175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.815520 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-home\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.847175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.817610 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-model-cache\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.847175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.818650 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-home\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.847175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.818938 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-tmp-dir\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.911041 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.910424 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8tqx\" (UniqueName: \"kubernetes.io/projected/5faa3220-d981-43af-a6d2-19d841026802-kube-api-access-j8tqx\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.911041 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.910738 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:22.946206 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.938559 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faa3220-d981-43af-a6d2-19d841026802-tls-certs\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:22.966652 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:22.947717 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-dshm\") pod \"ipp-route-913351-kserve-66b7cb8c7-w9fg7\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:23.188178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:23.188127 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:23.251881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:23.251620 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:23.368978 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:23.368929 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:23.675656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:23.672206 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:23.844296 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:23.844212 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:23.850295 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:23.850268 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:50:24.022787 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:50:24.022174 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5faa3220_d981_43af_a6d2_19d841026802.slice/crio-ea950d6fa54054303b7bd6e6b7d7959c3638a4d736f8a5f4a82da173fa8f0b2a WatchSource:0}: Error finding container ea950d6fa54054303b7bd6e6b7d7959c3638a4d736f8a5f4a82da173fa8f0b2a: Status 404 returned error can't find the container with id ea950d6fa54054303b7bd6e6b7d7959c3638a4d736f8a5f4a82da173fa8f0b2a Oct 30 19:50:24.022787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:24.022464 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7"] Oct 30 19:50:24.636619 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:24.636580 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:24.788487 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:24.788439 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:24.856441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:24.856397 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" event={"ID":"5faa3220-d981-43af-a6d2-19d841026802","Type":"ContainerStarted","Data":"e11344a9458299258f6751fbb4bb79a1e8700fc25ff755dafbf588c7818384d8"} Oct 30 19:50:24.856670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:24.856453 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" event={"ID":"5faa3220-d981-43af-a6d2-19d841026802","Type":"ContainerStarted","Data":"ea950d6fa54054303b7bd6e6b7d7959c3638a4d736f8a5f4a82da173fa8f0b2a"} Oct 30 19:50:24.892442 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:24.892114 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" podStartSLOduration=2.892094792 podStartE2EDuration="2.892094792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:24.889893723 +0000 UTC m=+1144.585101686" watchObservedRunningTime="2026-10-30 19:50:24.892094792 +0000 UTC m=+1144.587302754" Oct 30 19:50:24.987068 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:24.986980 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:25.252924 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:25.252890 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:25.810641 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:25.810284 5091 patch_prober.go:29] interesting pod/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:50:25.810641 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:25.810351 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:26.019522 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:26.018567 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:26.248581 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:26.243443 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:26.469677 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:26.468422 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:27.517492 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:27.517448 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:27.929841 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:27.894261 5091 generic.go:356] "Generic (PLEG): container finished" podID="111283d7-9f88-4151-beee-283607f0e13b" containerID="c6dc220bc9f42c5e2bac62efc5eda85423de21fecc13cb7c8d66fda483812b67" exitCode=0 Oct 30 19:50:27.929841 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:27.894327 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" event={"ID":"111283d7-9f88-4151-beee-283607f0e13b","Type":"ContainerDied","Data":"c6dc220bc9f42c5e2bac62efc5eda85423de21fecc13cb7c8d66fda483812b67"} Oct 30 19:50:27.929841 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:27.894790 5091 scope.go:122] "RemoveContainer" containerID="c6dc220bc9f42c5e2bac62efc5eda85423de21fecc13cb7c8d66fda483812b67" Oct 30 19:50:27.929841 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:27.896501 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:50:27.929841 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:27.896569 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:50:27.929841 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:27.896599 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:50:28.359233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:28.345540 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:28.508102 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:28.508062 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:28.935009 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:28.934971 5091 generic.go:356] "Generic (PLEG): container finished" podID="54f332bc-3668-4f09-a354-697e1610cd3d" containerID="1a8ce22bc4efc48cdfbbde310e448600f6ed47f295b68d59fb5fbf7295f5bb3d" exitCode=0 Oct 30 19:50:28.935659 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:28.935230 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" event={"ID":"54f332bc-3668-4f09-a354-697e1610cd3d","Type":"ContainerDied","Data":"1a8ce22bc4efc48cdfbbde310e448600f6ed47f295b68d59fb5fbf7295f5bb3d"} Oct 30 19:50:28.938187 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:28.938124 5091 scope.go:122] "RemoveContainer" containerID="1a8ce22bc4efc48cdfbbde310e448600f6ed47f295b68d59fb5fbf7295f5bb3d" Oct 30 19:50:28.940336 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:28.940295 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:50:28.940567 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:28.940528 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:50:28.940706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:28.940690 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:50:28.941554 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:28.941533 5091 generic.go:356] "Generic (PLEG): container finished" podID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerID="be1259bb71acb8835bd67056bd1b78f202e95c8543001a0cc4482a20efd6152a" exitCode=0 Oct 30 19:50:28.941876 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:28.941854 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" event={"ID":"b7ab432f-a929-40f9-8ed4-042d9528a20b","Type":"ContainerDied","Data":"be1259bb71acb8835bd67056bd1b78f202e95c8543001a0cc4482a20efd6152a"} Oct 30 19:50:28.942462 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:28.942441 5091 scope.go:122] "RemoveContainer" containerID="be1259bb71acb8835bd67056bd1b78f202e95c8543001a0cc4482a20efd6152a" Oct 30 19:50:28.946479 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:28.946456 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" event={"ID":"111283d7-9f88-4151-beee-283607f0e13b","Type":"ContainerStarted","Data":"a40e8721d7da58ff2741d9694c332d0ff9ee53afbd898303adeee76d76ea9d7e"} Oct 30 19:50:28.952861 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:28.952826 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:50:28.953076 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:28.953055 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:50:28.954393 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:28.954369 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:50:29.301230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:29.294781 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:29.577172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:29.575224 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:50:29.586171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:29.579267 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:50:29.586171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:29.579322 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" podUID="111283d7-9f88-4151-beee-283607f0e13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 19:50:29.998854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:29.998817 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" event={"ID":"54f332bc-3668-4f09-a354-697e1610cd3d","Type":"ContainerStarted","Data":"3a0da0bf34e7cf2a671266fe0ee9d48905a98a9a75cac2795be5c94fbf332a6f"} Oct 30 19:50:30.001951 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:30.001914 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" event={"ID":"b7ab432f-a929-40f9-8ed4-042d9528a20b","Type":"ContainerStarted","Data":"c4c1f5093aeb7f6937e21519ac309e0620da01bad97fe0fd393ff0a5f0c61dca"} Oct 30 19:50:30.173070 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:30.172629 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:50:30.173070 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:30.172764 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 19:50:30.173070 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:30.172804 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 19:50:30.531669 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:30.531633 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:30.568331 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:30.556274 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:50:30.568331 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:30.556333 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" podUID="111283d7-9f88-4151-beee-283607f0e13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 19:50:30.903227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:30.899984 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:50:30.934209 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:30.934124 5091 patch_prober.go:29] interesting pod/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 19:50:30.934425 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:30.934212 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 19:50:31.023207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:31.018255 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:50:31.175901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:31.175791 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 19:50:31.175901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:31.175885 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 19:50:31.487123 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:31.486982 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:31.609169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:31.608631 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:50:31.609169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:31.608697 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" podUID="111283d7-9f88-4151-beee-283607f0e13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 19:50:31.914643 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:31.914602 5091 patch_prober.go:29] interesting pod/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 19:50:31.914937 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:31.914907 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 19:50:32.184173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:32.184065 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 19:50:32.184784 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:32.184737 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 19:50:32.389795 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:32.389752 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:32.558183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:32.557269 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:32.583172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:32.577850 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:50:32.583172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:32.577930 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" podUID="111283d7-9f88-4151-beee-283607f0e13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 19:50:32.902659 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:32.902568 5091 patch_prober.go:29] interesting pod/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 19:50:32.902918 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:32.902886 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 19:50:33.172236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:33.171827 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 19:50:33.172236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:33.171889 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 19:50:33.258181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:33.256897 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:33.282173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:33.277278 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:33.282173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:33.281120 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:33.543033 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:33.542998 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:33.571168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:33.566006 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:50:33.571168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:33.566073 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" podUID="111283d7-9f88-4151-beee-283607f0e13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 19:50:33.571168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:33.566523 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:50:33.571168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:33.566874 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:50:33.911216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:33.907510 5091 patch_prober.go:29] interesting pod/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 19:50:33.911216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:33.907571 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 19:50:34.177800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:34.177700 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 19:50:34.178083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:34.178035 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 19:50:34.257204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:34.257107 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:34.286882 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:34.286838 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:50:34.288233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:34.288204 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:50:34.526305 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:34.516233 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:34.565169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:34.565117 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:50:34.590171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:34.565432 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" podUID="111283d7-9f88-4151-beee-283607f0e13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 19:50:34.590171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:34.567069 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:50:34.742774 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:34.742740 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:50:34.742978 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:34.742809 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:50:34.927402 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:34.927311 5091 patch_prober.go:29] interesting pod/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 19:50:34.927669 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:34.927635 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 19:50:35.093169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.029319 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:50:35.168369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.168318 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 19:50:35.168655 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.168625 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 19:50:35.544171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.541042 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr"] Oct 30 19:50:35.614206 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.609810 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:35.650171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.635460 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" Oct 30 19:50:35.698172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.636309 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Oct 30 19:50:35.698172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.671087 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" podUID="111283d7-9f88-4151-beee-283607f0e13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Oct 30 19:50:35.698172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.693301 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:50:35.698172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.693548 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:50:35.706168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.699324 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr"] Oct 30 19:50:35.805758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.805381 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr\" (UID: \"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" Oct 30 19:50:35.805758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.805609 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ftd2\" (UniqueName: \"kubernetes.io/projected/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-kube-api-access-8ftd2\") pod \"maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr\" (UID: \"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" Oct 30 19:50:35.805990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.805787 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr\" (UID: \"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" Oct 30 19:50:35.928177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.913106 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr\" (UID: \"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" Oct 30 19:50:35.946230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.932255 5091 patch_prober.go:29] interesting pod/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 19:50:35.946230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.940447 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 19:50:35.946230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.940305 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr\" (UID: \"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" Oct 30 19:50:35.946681 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.946658 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr\" (UID: \"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" Oct 30 19:50:35.966465 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:35.966421 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ftd2\" (UniqueName: \"kubernetes.io/projected/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-kube-api-access-8ftd2\") pod \"maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr\" (UID: \"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" Oct 30 19:50:36.043699 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:36.043664 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ftd2\" (UniqueName: \"kubernetes.io/projected/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-kube-api-access-8ftd2\") pod \"maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr\" (UID: \"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" Oct 30 19:50:36.044365 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:36.044338 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr\" (UID: \"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" Oct 30 19:50:36.125330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:36.125229 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n"] Oct 30 19:50:36.132945 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:36.132908 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" Oct 30 19:50:36.137165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:36.136892 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1ac7309d-4xf7n"] Oct 30 19:50:36.168512 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:36.168479 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 19:50:36.168795 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:36.168761 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 19:50:36.529128 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:36.529091 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35570a64-0a60-408f-97de-897f2b97132d" path="/var/lib/kubelet/pods/35570a64-0a60-408f-97de-897f2b97132d/volumes" Oct 30 19:50:36.626201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:36.607589 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:50:36.626201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:36.625168 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr"] Oct 30 19:50:36.628301 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:36.628195 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:50:36.740651 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:36.740618 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:50:36.915206 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:36.908566 5091 patch_prober.go:29] interesting pod/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Oct 30 19:50:36.915206 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:36.908624 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Oct 30 19:50:36.923022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:36.922423 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:37.178170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:37.177579 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body= Oct 30 19:50:37.178170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:37.177636 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" Oct 30 19:50:37.558745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:37.558238 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" event={"ID":"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f","Type":"ContainerStarted","Data":"04e995e6dfbf9ca2835804ac9d7aaa59daff860111b95a9b05bfaeffce314786"} Oct 30 19:50:37.558952 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:37.558876 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" event={"ID":"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f","Type":"ContainerStarted","Data":"a1c9dc49ed127a9fbfa5c031186fda2c84e33baf17a9b74678ac0585d907cb1a"} Oct 30 19:50:37.585617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:37.585554 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" podStartSLOduration=2.585535274 podStartE2EDuration="2.585535274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:37.58028802 +0000 UTC m=+1157.275495974" watchObservedRunningTime="2026-10-30 19:50:37.585535274 +0000 UTC m=+1157.280743232" Oct 30 19:50:37.904926 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:37.904846 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:50:37.905321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:37.905238 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:50:37.905892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:37.905877 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:50:38.038157 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:38.037962 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:38.172433 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:38.172346 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:50:38.173692 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:38.173665 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:50:38.568697 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:38.568660 5091 generic.go:356] "Generic (PLEG): container finished" podID="bf3be4c0-8f20-4d09-992d-c9b52ca72e7f" containerID="04e995e6dfbf9ca2835804ac9d7aaa59daff860111b95a9b05bfaeffce314786" exitCode=0 Oct 30 19:50:38.568927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:38.568701 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" event={"ID":"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f","Type":"ContainerDied","Data":"04e995e6dfbf9ca2835804ac9d7aaa59daff860111b95a9b05bfaeffce314786"} Oct 30 19:50:38.631432 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:38.631389 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:38.784989 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:38.784903 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j"] Oct 30 19:50:38.792903 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:38.792861 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" Oct 30 19:50:38.795797 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:38.795212 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:38.797060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:38.797038 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j"] Oct 30 19:50:38.802584 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:38.802563 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4f2325-dockercfg-dff6p\"" Oct 30 19:50:38.905227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:38.869628 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4f2325\"" Oct 30 19:50:38.952190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:38.907578 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba6caa8-6ed3-40a3-bf9d-43d4008e9339-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j\" (UID: \"7ba6caa8-6ed3-40a3-bf9d-43d4008e9339\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" Oct 30 19:50:38.952573 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:38.952537 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnklc\" (UniqueName: \"kubernetes.io/projected/7ba6caa8-6ed3-40a3-bf9d-43d4008e9339-kube-api-access-qnklc\") pod \"payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j\" (UID: \"7ba6caa8-6ed3-40a3-bf9d-43d4008e9339\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" Oct 30 19:50:39.058713 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.058677 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba6caa8-6ed3-40a3-bf9d-43d4008e9339-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j\" (UID: \"7ba6caa8-6ed3-40a3-bf9d-43d4008e9339\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" Oct 30 19:50:39.058965 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.058943 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnklc\" (UniqueName: \"kubernetes.io/projected/7ba6caa8-6ed3-40a3-bf9d-43d4008e9339-kube-api-access-qnklc\") pod \"payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j\" (UID: \"7ba6caa8-6ed3-40a3-bf9d-43d4008e9339\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" Oct 30 19:50:39.060273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.060248 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba6caa8-6ed3-40a3-bf9d-43d4008e9339-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j\" (UID: \"7ba6caa8-6ed3-40a3-bf9d-43d4008e9339\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" Oct 30 19:50:39.129165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.125394 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnklc\" (UniqueName: \"kubernetes.io/projected/7ba6caa8-6ed3-40a3-bf9d-43d4008e9339-kube-api-access-qnklc\") pod \"payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j\" (UID: \"7ba6caa8-6ed3-40a3-bf9d-43d4008e9339\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" Oct 30 19:50:39.383184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.383123 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:39.414217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.414172 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" Oct 30 19:50:39.735654 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:50:39.735623 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ba6caa8_6ed3_40a3_bf9d_43d4008e9339.slice/crio-c6753cec7ecb405f55f707565ae4d823a7937b0a3dfc03e227b8ef84aa9b7a55 WatchSource:0}: Error finding container c6753cec7ecb405f55f707565ae4d823a7937b0a3dfc03e227b8ef84aa9b7a55: Status 404 returned error can't find the container with id c6753cec7ecb405f55f707565ae4d823a7937b0a3dfc03e227b8ef84aa9b7a55 Oct 30 19:50:39.735796 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.735686 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j"] Oct 30 19:50:39.762092 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.762068 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" Oct 30 19:50:39.876103 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.876068 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-maas-api-xks-ca\") pod \"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f\" (UID: \"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f\") " Oct 30 19:50:39.876280 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.876211 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-kube-api-access-8ftd2\" (UniqueName: \"kubernetes.io/projected/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-kube-api-access-8ftd2\") pod \"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f\" (UID: \"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f\") " Oct 30 19:50:39.876280 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.876272 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-maas-api-openshift-ca\") pod \"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f\" (UID: \"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f\") " Oct 30 19:50:39.877219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.877170 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-maas-api-openshift-ca" pod "bf3be4c0-8f20-4d09-992d-c9b52ca72e7f" (UID: "bf3be4c0-8f20-4d09-992d-c9b52ca72e7f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:39.879384 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.879363 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-kube-api-access-8ftd2" pod "bf3be4c0-8f20-4d09-992d-c9b52ca72e7f" (UID: "bf3be4c0-8f20-4d09-992d-c9b52ca72e7f"). InnerVolumeSpecName "kube-api-access-8ftd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:39.882755 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.882732 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-maas-api-xks-ca" pod "bf3be4c0-8f20-4d09-992d-c9b52ca72e7f" (UID: "bf3be4c0-8f20-4d09-992d-c9b52ca72e7f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:39.977983 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.977907 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:39.977983 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.977936 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ftd2\" (UniqueName: \"kubernetes.io/projected/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-kube-api-access-8ftd2\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:39.977983 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:39.977946 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:40.611058 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:40.611025 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" Oct 30 19:50:40.611265 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:40.611060 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr" event={"ID":"bf3be4c0-8f20-4d09-992d-c9b52ca72e7f","Type":"ContainerDied","Data":"a1c9dc49ed127a9fbfa5c031186fda2c84e33baf17a9b74678ac0585d907cb1a"} Oct 30 19:50:40.611265 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:40.611118 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1c9dc49ed127a9fbfa5c031186fda2c84e33baf17a9b74678ac0585d907cb1a" Oct 30 19:50:40.613059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:40.613025 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" event={"ID":"7ba6caa8-6ed3-40a3-bf9d-43d4008e9339","Type":"ContainerStarted","Data":"fd44830725da38a8c72890eb44fd49871605b67dd7753abc02aeb2a616486b5d"} Oct 30 19:50:40.613257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:40.613089 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" Oct 30 19:50:40.613257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:40.613114 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" event={"ID":"7ba6caa8-6ed3-40a3-bf9d-43d4008e9339","Type":"ContainerStarted","Data":"c6753cec7ecb405f55f707565ae4d823a7937b0a3dfc03e227b8ef84aa9b7a55"} Oct 30 19:50:40.641974 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:40.641932 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" podStartSLOduration=2.641921944 podStartE2EDuration="2.641921944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:40.64070977 +0000 UTC m=+1160.335917728" watchObservedRunningTime="2026-10-30 19:50:40.641921944 +0000 UTC m=+1160.337129918" Oct 30 19:50:40.647992 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:40.647964 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:40.837488 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:40.835242 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:41.873195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:41.873086 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:42.678172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:42.672892 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:43.488837 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:43.488803 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:44.295547 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:44.295497 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:45.468682 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:45.468651 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:45.846278 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:45.846246 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:45.980395 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:50:45.980359 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:50:45.988227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:45.988184 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:46.366854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.354426 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq"] Oct 30 19:50:46.366854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.355572 5091 cpu_manager.go:414] "containerMap: removing container" podUID="bf3be4c0-8f20-4d09-992d-c9b52ca72e7f" containerName="revoke-keys" Oct 30 19:50:46.366854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.355592 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf3be4c0-8f20-4d09-992d-c9b52ca72e7f" containerName="revoke-keys" Oct 30 19:50:46.366854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.355834 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf3be4c0-8f20-4d09-992d-c9b52ca72e7f" containerName="revoke-keys" Oct 30 19:50:46.366854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.366329 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq"] Oct 30 19:50:46.366854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.366440 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" Oct 30 19:50:46.377346 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.377306 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:50:46.380019 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.379994 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:50:46.463255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.463221 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhmr5\" (UniqueName: \"kubernetes.io/projected/7bed8e06-c234-4601-a289-f094454b8fef-kube-api-access-fhmr5\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq\" (UID: \"7bed8e06-c234-4601-a289-f094454b8fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" Oct 30 19:50:46.463427 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.463395 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bed8e06-c234-4601-a289-f094454b8fef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq\" (UID: \"7bed8e06-c234-4601-a289-f094454b8fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" Oct 30 19:50:46.463512 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.463494 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bed8e06-c234-4601-a289-f094454b8fef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq\" (UID: \"7bed8e06-c234-4601-a289-f094454b8fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" Oct 30 19:50:46.565454 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.565410 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhmr5\" (UniqueName: \"kubernetes.io/projected/7bed8e06-c234-4601-a289-f094454b8fef-kube-api-access-fhmr5\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq\" (UID: \"7bed8e06-c234-4601-a289-f094454b8fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" Oct 30 19:50:46.565907 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.565528 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bed8e06-c234-4601-a289-f094454b8fef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq\" (UID: \"7bed8e06-c234-4601-a289-f094454b8fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" Oct 30 19:50:46.565907 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.565597 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bed8e06-c234-4601-a289-f094454b8fef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq\" (UID: \"7bed8e06-c234-4601-a289-f094454b8fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" Oct 30 19:50:46.566243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.566204 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bed8e06-c234-4601-a289-f094454b8fef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq\" (UID: \"7bed8e06-c234-4601-a289-f094454b8fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" Oct 30 19:50:46.568072 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.568051 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bed8e06-c234-4601-a289-f094454b8fef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq\" (UID: \"7bed8e06-c234-4601-a289-f094454b8fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" Oct 30 19:50:46.579579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.579550 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhmr5\" (UniqueName: \"kubernetes.io/projected/7bed8e06-c234-4601-a289-f094454b8fef-kube-api-access-fhmr5\") pod \"maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq\" (UID: \"7bed8e06-c234-4601-a289-f094454b8fef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" Oct 30 19:50:46.702186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.702040 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" Oct 30 19:50:46.725527 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:46.725483 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:47.038446 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:50:47.038413 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bed8e06_c234_4601_a289_f094454b8fef.slice/crio-7c3fd3931828e490004a2dfcbd560dcf477c2da3578cc36eb0224781eaf4cb55 WatchSource:0}: Error finding container 7c3fd3931828e490004a2dfcbd560dcf477c2da3578cc36eb0224781eaf4cb55: Status 404 returned error can't find the container with id 7c3fd3931828e490004a2dfcbd560dcf477c2da3578cc36eb0224781eaf4cb55 Oct 30 19:50:47.040004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:47.039972 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq"] Oct 30 19:50:47.731338 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:47.731255 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:47.791027 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:47.790982 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" event={"ID":"7bed8e06-c234-4601-a289-f094454b8fef","Type":"ContainerStarted","Data":"c1ab07600d46f69733408a5b8f19d7c99b299b410c4ec5255b15210fe5291698"} Oct 30 19:50:47.791335 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:47.791297 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" event={"ID":"7bed8e06-c234-4601-a289-f094454b8fef","Type":"ContainerStarted","Data":"7c3fd3931828e490004a2dfcbd560dcf477c2da3578cc36eb0224781eaf4cb55"} Oct 30 19:50:47.835762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:47.835704 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" podStartSLOduration=1.8356806479999999 podStartE2EDuration="1.835680648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:47.832993672 +0000 UTC m=+1167.528201632" watchObservedRunningTime="2026-10-30 19:50:47.835680648 +0000 UTC m=+1167.530888606" Oct 30 19:50:48.652982 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:48.652948 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:48.808704 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:48.808666 5091 generic.go:356] "Generic (PLEG): container finished" podID="7bed8e06-c234-4601-a289-f094454b8fef" containerID="c1ab07600d46f69733408a5b8f19d7c99b299b410c4ec5255b15210fe5291698" exitCode=0 Oct 30 19:50:48.809238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:48.808732 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" event={"ID":"7bed8e06-c234-4601-a289-f094454b8fef","Type":"ContainerDied","Data":"c1ab07600d46f69733408a5b8f19d7c99b299b410c4ec5255b15210fe5291698"} Oct 30 19:50:49.590183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:49.587847 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:50.008488 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:50.008460 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" Oct 30 19:50:50.085367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:50.085317 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bed8e06-c234-4601-a289-f094454b8fef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bed8e06-c234-4601-a289-f094454b8fef-maas-api-xks-ca\") pod \"7bed8e06-c234-4601-a289-f094454b8fef\" (UID: \"7bed8e06-c234-4601-a289-f094454b8fef\") " Oct 30 19:50:50.085614 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:50.085546 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bed8e06-c234-4601-a289-f094454b8fef-kube-api-access-fhmr5\" (UniqueName: \"kubernetes.io/projected/7bed8e06-c234-4601-a289-f094454b8fef-kube-api-access-fhmr5\") pod \"7bed8e06-c234-4601-a289-f094454b8fef\" (UID: \"7bed8e06-c234-4601-a289-f094454b8fef\") " Oct 30 19:50:50.085756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:50.085727 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bed8e06-c234-4601-a289-f094454b8fef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bed8e06-c234-4601-a289-f094454b8fef-maas-api-openshift-ca\") pod \"7bed8e06-c234-4601-a289-f094454b8fef\" (UID: \"7bed8e06-c234-4601-a289-f094454b8fef\") " Oct 30 19:50:50.086329 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:50.086300 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bed8e06-c234-4601-a289-f094454b8fef-maas-api-openshift-ca" pod "7bed8e06-c234-4601-a289-f094454b8fef" (UID: "7bed8e06-c234-4601-a289-f094454b8fef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:50:50.086552 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:50.086531 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bed8e06-c234-4601-a289-f094454b8fef-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:50.088815 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:50.088782 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bed8e06-c234-4601-a289-f094454b8fef-maas-api-xks-ca" pod "7bed8e06-c234-4601-a289-f094454b8fef" (UID: "7bed8e06-c234-4601-a289-f094454b8fef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:50:50.088928 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:50.088787 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bed8e06-c234-4601-a289-f094454b8fef-kube-api-access-fhmr5" pod "7bed8e06-c234-4601-a289-f094454b8fef" (UID: "7bed8e06-c234-4601-a289-f094454b8fef"). InnerVolumeSpecName "kube-api-access-fhmr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:50:50.187737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:50.187654 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bed8e06-c234-4601-a289-f094454b8fef-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:50.187737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:50.187688 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhmr5\" (UniqueName: \"kubernetes.io/projected/7bed8e06-c234-4601-a289-f094454b8fef-kube-api-access-fhmr5\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:50:50.671177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:50.669924 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:50.828019 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:50.827992 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" Oct 30 19:50:50.828019 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:50.828010 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq" event={"ID":"7bed8e06-c234-4601-a289-f094454b8fef","Type":"ContainerDied","Data":"7c3fd3931828e490004a2dfcbd560dcf477c2da3578cc36eb0224781eaf4cb55"} Oct 30 19:50:50.828316 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:50.828059 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c3fd3931828e490004a2dfcbd560dcf477c2da3578cc36eb0224781eaf4cb55" Oct 30 19:50:51.624539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:51.624499 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" Oct 30 19:50:51.903237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:51.902407 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:52.746891 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:52.746857 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:53.797367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:53.793083 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:54.997211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:54.994317 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:55.230599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.230559 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:55.611313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.611277 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn"] Oct 30 19:50:55.612342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.612314 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7bed8e06-c234-4601-a289-f094454b8fef" containerName="revoke-keys" Oct 30 19:50:55.612342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.612335 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bed8e06-c234-4601-a289-f094454b8fef" containerName="revoke-keys" Oct 30 19:50:55.612563 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.612539 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bed8e06-c234-4601-a289-f094454b8fef" containerName="revoke-keys" Oct 30 19:50:55.618780 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.618752 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.625338 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.625312 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn"] Oct 30 19:50:55.627484 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.627464 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ec5e0f-openshift-default-dockercfg-2xxlz\"" Oct 30 19:50:55.692552 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.692501 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-data\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.692724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.692578 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-podinfo\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.692724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.692638 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddwlm\" (UniqueName: \"kubernetes.io/projected/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-kube-api-access-ddwlm\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.692724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.692705 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-envoy\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.692913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.692842 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-workload-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.692913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.692866 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-token\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.693103 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.693052 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istiod-ca-cert\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.693251 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.693224 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-credential-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.693325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.693282 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-workload-certs\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.709432 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.709392 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:55.798170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.797954 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istiod-ca-cert\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.798170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.798067 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-credential-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.825179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.822729 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istiod-ca-cert\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.825179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.823260 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-workload-certs\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.825179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.823358 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-data\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.825179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.823429 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-podinfo\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.825179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.823503 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddwlm\" (UniqueName: \"kubernetes.io/projected/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-kube-api-access-ddwlm\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.825179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.823591 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-envoy\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.825179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.823793 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-workload-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.825179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.823818 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-token\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.825179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.824548 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-credential-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.825179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.824970 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-workload-certs\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.826259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.826211 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-workload-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.833446 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.833415 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-data\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.849742 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.849710 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-podinfo\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.861823 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.861752 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-token\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.891084 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.891049 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-envoy\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:55.958173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.951879 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:55.989171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:55.968785 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddwlm\" (UniqueName: \"kubernetes.io/projected/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-kube-api-access-ddwlm\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:56.036913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:56.036877 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:56.223692 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:56.223643 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:56.447166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:56.447031 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn"] Oct 30 19:50:56.450085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:56.450043 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:50:56.450240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:56.450130 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:50:56.450240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:56.450190 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:50:57.125172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:57.122371 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:57.180611 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:57.180574 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" event={"ID":"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b","Type":"ContainerStarted","Data":"fd94dbb84e8ddf338e641fe05490cfe8c052f8b7e98e29073513c6bdf7c361b0"} Oct 30 19:50:57.180804 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:57.180634 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" event={"ID":"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b","Type":"ContainerStarted","Data":"69adc3cebe7c0b49efdea38e9e0c7b527c997ed3db5b0d597d9127300b6047ee"} Oct 30 19:50:57.214767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:57.214717 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" podStartSLOduration=2.214702486 podStartE2EDuration="2.214702486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:50:57.210734176 +0000 UTC m=+1176.905942134" watchObservedRunningTime="2026-10-30 19:50:57.214702486 +0000 UTC m=+1176.909910443" Oct 30 19:50:58.037476 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:58.037442 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d"] Oct 30 19:50:58.041371 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:58.041345 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77c25d0d-fn72d"] Oct 30 19:50:58.062539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:58.062505 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:58.070808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:58.070784 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:58.147174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:58.145002 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:58.201075 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:58.201048 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:58.202774 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:58.202752 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:50:58.443838 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:58.443765 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2df1aca0-a819-4417-b85a-f9d2fda67997" path="/var/lib/kubelet/pods/2df1aca0-a819-4417-b85a-f9d2fda67997/volumes" Oct 30 19:50:59.199128 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:59.199089 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:59.628626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:59.628594 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:59.774138 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:59.774099 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:50:59.933204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:50:59.933115 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:00.625010 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.624977 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh"] Oct 30 19:51:00.630845 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.630824 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" Oct 30 19:51:00.641251 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.641228 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh"] Oct 30 19:51:00.648705 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.648677 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:51:00.648836 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.648736 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:51:00.732882 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.732847 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:00.768462 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.745868 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e907e4-86gqh\" (UID: \"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" Oct 30 19:51:00.768462 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.757189 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5l4x\" (UniqueName: \"kubernetes.io/projected/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-kube-api-access-j5l4x\") pod \"maas-api-revoke-keys-e2e-api-a-e907e4-86gqh\" (UID: \"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" Oct 30 19:51:00.768462 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.757267 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e907e4-86gqh\" (UID: \"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" Oct 30 19:51:00.859171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.859112 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5l4x\" (UniqueName: \"kubernetes.io/projected/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-kube-api-access-j5l4x\") pod \"maas-api-revoke-keys-e2e-api-a-e907e4-86gqh\" (UID: \"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" Oct 30 19:51:00.859425 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.859226 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e907e4-86gqh\" (UID: \"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" Oct 30 19:51:00.859425 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.859415 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e907e4-86gqh\" (UID: \"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" Oct 30 19:51:00.860245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.860216 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e907e4-86gqh\" (UID: \"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" Oct 30 19:51:00.861925 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.861888 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e907e4-86gqh\" (UID: \"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" Oct 30 19:51:00.875817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.875738 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5l4x\" (UniqueName: \"kubernetes.io/projected/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-kube-api-access-j5l4x\") pod \"maas-api-revoke-keys-e2e-api-a-e907e4-86gqh\" (UID: \"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" Oct 30 19:51:00.949219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:00.949172 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" Oct 30 19:51:01.023297 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:01.023264 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:01.192215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:01.191342 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:01.282255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:01.280256 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6"] Oct 30 19:51:01.308192 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:01.298595 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" Oct 30 19:51:01.438901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:01.337031 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ec5e0f-dockercfg-mptzf\"" Oct 30 19:51:01.438901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:01.337243 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ec5e0f\"" Oct 30 19:51:01.438901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:01.342985 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6"] Oct 30 19:51:01.706841 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:51:01.705978 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8e498a_fce9_49d5_9eeb_5e784bcf66d7.slice/crio-58115276b4bf6d772e81a5ab1b478d92adbc9b25970b27490cbbc88ae8afa326 WatchSource:0}: Error finding container 58115276b4bf6d772e81a5ab1b478d92adbc9b25970b27490cbbc88ae8afa326: Status 404 returned error can't find the container with id 58115276b4bf6d772e81a5ab1b478d92adbc9b25970b27490cbbc88ae8afa326 Oct 30 19:51:01.808170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:01.794533 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6w4m\" (UniqueName: \"kubernetes.io/projected/03fb817b-087c-4744-8f3d-fc12e8444385-kube-api-access-r6w4m\") pod \"payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6\" (UID: \"03fb817b-087c-4744-8f3d-fc12e8444385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" Oct 30 19:51:01.815371 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:01.815330 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03fb817b-087c-4744-8f3d-fc12e8444385-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6\" (UID: \"03fb817b-087c-4744-8f3d-fc12e8444385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" Oct 30 19:51:01.840194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:01.824823 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh"] Oct 30 19:51:01.921213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:01.921142 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03fb817b-087c-4744-8f3d-fc12e8444385-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6\" (UID: \"03fb817b-087c-4744-8f3d-fc12e8444385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" Oct 30 19:51:01.921394 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:01.921347 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6w4m\" (UniqueName: \"kubernetes.io/projected/03fb817b-087c-4744-8f3d-fc12e8444385-kube-api-access-r6w4m\") pod \"payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6\" (UID: \"03fb817b-087c-4744-8f3d-fc12e8444385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" Oct 30 19:51:01.921872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:01.921839 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03fb817b-087c-4744-8f3d-fc12e8444385-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6\" (UID: \"03fb817b-087c-4744-8f3d-fc12e8444385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" Oct 30 19:51:01.943519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:01.943494 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6w4m\" (UniqueName: \"kubernetes.io/projected/03fb817b-087c-4744-8f3d-fc12e8444385-kube-api-access-r6w4m\") pod \"payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6\" (UID: \"03fb817b-087c-4744-8f3d-fc12e8444385\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" Oct 30 19:51:02.104007 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:02.103972 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" Oct 30 19:51:02.178169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:02.175079 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:02.734252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:02.734210 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" event={"ID":"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7","Type":"ContainerStarted","Data":"1cd417639ed5bb86c22564f705d0d4650d37285688bd6915e91b3428ee192c04"} Oct 30 19:51:02.734807 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:02.734299 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" event={"ID":"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7","Type":"ContainerStarted","Data":"58115276b4bf6d772e81a5ab1b478d92adbc9b25970b27490cbbc88ae8afa326"} Oct 30 19:51:02.755303 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:02.755271 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6"] Oct 30 19:51:02.762321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:02.761989 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" podStartSLOduration=2.761969907 podStartE2EDuration="2.761969907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:02.757140243 +0000 UTC m=+1182.452348201" watchObservedRunningTime="2026-10-30 19:51:02.761969907 +0000 UTC m=+1182.457177867" Oct 30 19:51:03.210800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:03.205083 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:03.769383 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:03.769346 5091 generic.go:356] "Generic (PLEG): container finished" podID="4d8e498a-fce9-49d5-9eeb-5e784bcf66d7" containerID="1cd417639ed5bb86c22564f705d0d4650d37285688bd6915e91b3428ee192c04" exitCode=0 Oct 30 19:51:03.770168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:03.769975 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" event={"ID":"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7","Type":"ContainerDied","Data":"1cd417639ed5bb86c22564f705d0d4650d37285688bd6915e91b3428ee192c04"} Oct 30 19:51:03.772889 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:03.772861 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" event={"ID":"03fb817b-087c-4744-8f3d-fc12e8444385","Type":"ContainerStarted","Data":"0ddcfe9479fcfb2ae9ba312ae2752896fbb0b7aea70b74f37cc729044fe6c20d"} Oct 30 19:51:03.773082 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:03.773066 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" event={"ID":"03fb817b-087c-4744-8f3d-fc12e8444385","Type":"ContainerStarted","Data":"53c53dc26d4c72b787fdce53049e1e39c2e38234dd98e496365990cb8429cbff"} Oct 30 19:51:03.774049 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:03.774028 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" Oct 30 19:51:03.836644 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:03.836586 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" podStartSLOduration=2.836569203 podStartE2EDuration="2.836569203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:03.831115833 +0000 UTC m=+1183.526323792" watchObservedRunningTime="2026-10-30 19:51:03.836569203 +0000 UTC m=+1183.531777163" Oct 30 19:51:04.218878 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:04.218782 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:05.025625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:05.025599 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" Oct 30 19:51:05.188840 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:05.188751 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-maas-api-openshift-ca\") pod \"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7\" (UID: \"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7\") " Oct 30 19:51:05.188983 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:05.188962 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-maas-api-xks-ca\") pod \"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7\" (UID: \"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7\") " Oct 30 19:51:05.189086 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:05.189070 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-kube-api-access-j5l4x\" (UniqueName: \"kubernetes.io/projected/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-kube-api-access-j5l4x\") pod \"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7\" (UID: \"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7\") " Oct 30 19:51:05.189304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:05.189275 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-maas-api-openshift-ca" pod "4d8e498a-fce9-49d5-9eeb-5e784bcf66d7" (UID: "4d8e498a-fce9-49d5-9eeb-5e784bcf66d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:05.189841 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:05.189815 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:05.191828 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:05.191800 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-kube-api-access-j5l4x" pod "4d8e498a-fce9-49d5-9eeb-5e784bcf66d7" (UID: "4d8e498a-fce9-49d5-9eeb-5e784bcf66d7"). InnerVolumeSpecName "kube-api-access-j5l4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:05.196561 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:05.196539 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-maas-api-xks-ca" pod "4d8e498a-fce9-49d5-9eeb-5e784bcf66d7" (UID: "4d8e498a-fce9-49d5-9eeb-5e784bcf66d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:05.291777 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:05.291739 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:05.291777 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:05.291776 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5l4x\" (UniqueName: \"kubernetes.io/projected/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7-kube-api-access-j5l4x\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:05.391417 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:05.391251 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:05.797770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:05.797739 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" Oct 30 19:51:05.797918 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:05.797769 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh" event={"ID":"4d8e498a-fce9-49d5-9eeb-5e784bcf66d7","Type":"ContainerDied","Data":"58115276b4bf6d772e81a5ab1b478d92adbc9b25970b27490cbbc88ae8afa326"} Oct 30 19:51:05.797918 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:05.797824 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58115276b4bf6d772e81a5ab1b478d92adbc9b25970b27490cbbc88ae8afa326" Oct 30 19:51:06.291171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:06.290185 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:06.309967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:06.300946 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q"] Oct 30 19:51:06.309967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:06.301021 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk"] Oct 30 19:51:06.309967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:06.301222 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" podUID="47f67a54-7bd4-4c38-b6d3-1e20d963aa1b" containerName="payload-pre-processing" containerID="cri-o://7eed83054d70b439d876294e47479fc7b3d30d689ce1e4c3708ee9e724a62f12" gracePeriod=30 Oct 30 19:51:06.309967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:06.301475 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" podUID="8b6a1c99-8256-4957-bd95-dc3f18f1d147" containerName="payload-processing" containerID="cri-o://f138aad2f627d681920365b8a724e05dc462c1b8763eac8ac7e42e1ee619cd0d" gracePeriod=30 Oct 30 19:51:06.439648 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:06.439530 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:06.593228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:06.593137 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:07.055966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.055917 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq"] Oct 30 19:51:07.122515 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.120806 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:07.127276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.127241 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c9d69c-6r6cq"] Oct 30 19:51:07.127532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.127457 5091 generic.go:356] "Generic (PLEG): container finished" podID="47f67a54-7bd4-4c38-b6d3-1e20d963aa1b" containerID="7eed83054d70b439d876294e47479fc7b3d30d689ce1e4c3708ee9e724a62f12" exitCode=0 Oct 30 19:51:07.127949 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.127545 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" event={"ID":"47f67a54-7bd4-4c38-b6d3-1e20d963aa1b","Type":"ContainerDied","Data":"7eed83054d70b439d876294e47479fc7b3d30d689ce1e4c3708ee9e724a62f12"} Oct 30 19:51:07.130331 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.130307 5091 generic.go:356] "Generic (PLEG): container finished" podID="8b6a1c99-8256-4957-bd95-dc3f18f1d147" containerID="f138aad2f627d681920365b8a724e05dc462c1b8763eac8ac7e42e1ee619cd0d" exitCode=0 Oct 30 19:51:07.130640 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.130383 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" event={"ID":"8b6a1c99-8256-4957-bd95-dc3f18f1d147","Type":"ContainerDied","Data":"f138aad2f627d681920365b8a724e05dc462c1b8763eac8ac7e42e1ee619cd0d"} Oct 30 19:51:07.236648 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.236618 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" Oct 30 19:51:07.294465 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.293953 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:07.341448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.341419 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b6a1c99-8256-4957-bd95-dc3f18f1d147-kube-api-access-rzh8t\" (UniqueName: \"kubernetes.io/projected/8b6a1c99-8256-4957-bd95-dc3f18f1d147-kube-api-access-rzh8t\") pod \"8b6a1c99-8256-4957-bd95-dc3f18f1d147\" (UID: \"8b6a1c99-8256-4957-bd95-dc3f18f1d147\") " Oct 30 19:51:07.341596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.341578 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b6a1c99-8256-4957-bd95-dc3f18f1d147-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b6a1c99-8256-4957-bd95-dc3f18f1d147-plugins-config-volume\") pod \"8b6a1c99-8256-4957-bd95-dc3f18f1d147\" (UID: \"8b6a1c99-8256-4957-bd95-dc3f18f1d147\") " Oct 30 19:51:07.342321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.342297 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b6a1c99-8256-4957-bd95-dc3f18f1d147-plugins-config-volume" pod "8b6a1c99-8256-4957-bd95-dc3f18f1d147" (UID: "8b6a1c99-8256-4957-bd95-dc3f18f1d147"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:07.344478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.344458 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b6a1c99-8256-4957-bd95-dc3f18f1d147-kube-api-access-rzh8t" pod "8b6a1c99-8256-4957-bd95-dc3f18f1d147" (UID: "8b6a1c99-8256-4957-bd95-dc3f18f1d147"). InnerVolumeSpecName "kube-api-access-rzh8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:07.402417 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.402390 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" Oct 30 19:51:07.433607 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.433575 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:07.443827 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.443792 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b6a1c99-8256-4957-bd95-dc3f18f1d147-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:07.443827 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.443824 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzh8t\" (UniqueName: \"kubernetes.io/projected/8b6a1c99-8256-4957-bd95-dc3f18f1d147-kube-api-access-rzh8t\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:07.544750 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.544669 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f67a54-7bd4-4c38-b6d3-1e20d963aa1b-kube-api-access-5cn6c\" (UniqueName: \"kubernetes.io/projected/47f67a54-7bd4-4c38-b6d3-1e20d963aa1b-kube-api-access-5cn6c\") pod \"47f67a54-7bd4-4c38-b6d3-1e20d963aa1b\" (UID: \"47f67a54-7bd4-4c38-b6d3-1e20d963aa1b\") " Oct 30 19:51:07.544885 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.544878 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f67a54-7bd4-4c38-b6d3-1e20d963aa1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f67a54-7bd4-4c38-b6d3-1e20d963aa1b-plugins-config-volume\") pod \"47f67a54-7bd4-4c38-b6d3-1e20d963aa1b\" (UID: \"47f67a54-7bd4-4c38-b6d3-1e20d963aa1b\") " Oct 30 19:51:07.545284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.545253 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f67a54-7bd4-4c38-b6d3-1e20d963aa1b-plugins-config-volume" pod "47f67a54-7bd4-4c38-b6d3-1e20d963aa1b" (UID: "47f67a54-7bd4-4c38-b6d3-1e20d963aa1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:07.558882 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.558838 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f67a54-7bd4-4c38-b6d3-1e20d963aa1b-kube-api-access-5cn6c" pod "47f67a54-7bd4-4c38-b6d3-1e20d963aa1b" (UID: "47f67a54-7bd4-4c38-b6d3-1e20d963aa1b"). InnerVolumeSpecName "kube-api-access-5cn6c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:07.575368 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.574931 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cn6c\" (UniqueName: \"kubernetes.io/projected/47f67a54-7bd4-4c38-b6d3-1e20d963aa1b-kube-api-access-5cn6c\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:07.575368 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.575230 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f67a54-7bd4-4c38-b6d3-1e20d963aa1b-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:07.980223 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:07.979600 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:08.141852 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:08.141826 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" Oct 30 19:51:08.142031 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:08.142010 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q" event={"ID":"8b6a1c99-8256-4957-bd95-dc3f18f1d147","Type":"ContainerDied","Data":"3255c1f96c0ab9c65ac3037845bfbfdea29326fcceb101d4299af6c191d306c7"} Oct 30 19:51:08.142164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:08.142139 5091 scope.go:122] "RemoveContainer" containerID="f138aad2f627d681920365b8a724e05dc462c1b8763eac8ac7e42e1ee619cd0d" Oct 30 19:51:08.143800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:08.143784 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" event={"ID":"47f67a54-7bd4-4c38-b6d3-1e20d963aa1b","Type":"ContainerDied","Data":"bab91d27024f2d00d61810c0e222515045069d2e5aa6d58a21237ae3e72afb6f"} Oct 30 19:51:08.143956 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:08.143936 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk" Oct 30 19:51:08.190138 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:08.190116 5091 scope.go:122] "RemoveContainer" containerID="7eed83054d70b439d876294e47479fc7b3d30d689ce1e4c3708ee9e724a62f12" Oct 30 19:51:08.211698 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:08.211395 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q"] Oct 30 19:51:08.217107 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:08.217085 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c9d69c-588d68c4db-d9d4q"] Oct 30 19:51:08.235117 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:08.235067 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk"] Oct 30 19:51:08.238902 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:08.238880 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-c9d69c-7ff6dbc9c9-8v6rk"] Oct 30 19:51:08.442436 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:08.442395 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f67a54-7bd4-4c38-b6d3-1e20d963aa1b" path="/var/lib/kubelet/pods/47f67a54-7bd4-4c38-b6d3-1e20d963aa1b/volumes" Oct 30 19:51:08.442961 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:08.442943 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bed8e06-c234-4601-a289-f094454b8fef" path="/var/lib/kubelet/pods/7bed8e06-c234-4601-a289-f094454b8fef/volumes" Oct 30 19:51:08.443406 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:08.443391 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b6a1c99-8256-4957-bd95-dc3f18f1d147" path="/var/lib/kubelet/pods/8b6a1c99-8256-4957-bd95-dc3f18f1d147/volumes" Oct 30 19:51:09.379012 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:09.378973 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:10.720240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:10.720100 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:10.791433 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:10.791400 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2"] Oct 30 19:51:10.811464 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:10.810708 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w"] Oct 30 19:51:10.920205 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:10.919899 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:11.076027 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:11.075911 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:11.605175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:11.605123 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:11.867793 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:11.867640 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:12.074247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:12.074211 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:12.117309 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:12.115122 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr"] Oct 30 19:51:12.126216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:12.125035 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-1e3723-kp9dr"] Oct 30 19:51:12.349015 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:12.348984 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:12.350484 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:12.350459 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp"] Oct 30 19:51:12.350714 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:12.350693 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" podUID="1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" containerName="istio-proxy" containerID="cri-o://7c5976ec74d6923e3a7861696dfd4231cce027ab2e8e451028747a33d5e2ba63" gracePeriod=30 Oct 30 19:51:12.357253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:12.357224 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8"] Oct 30 19:51:12.357521 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:12.357491 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" podUID="69602ea5-4de9-4b3a-8e3b-a3045d919c82" containerName="istio-proxy" containerID="cri-o://01de8521f0a1db0c2ecdaa886a44908c271bf920b0eb7f9baf4f4b762bd27d3f" gracePeriod=30 Oct 30 19:51:12.444028 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:12.443958 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf3be4c0-8f20-4d09-992d-c9b52ca72e7f" path="/var/lib/kubelet/pods/bf3be4c0-8f20-4d09-992d-c9b52ca72e7f/volumes" Oct 30 19:51:12.669839 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:12.669785 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:13.283327 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:13.283291 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:14.298677 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:14.298594 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:14.351879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:14.351800 5091 generic.go:356] "Generic (PLEG): container finished" podID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerID="b573b8cfcb3098eec6c22107f9d9a1bb8d6b8a941996207387539fde58fe554d" exitCode=0 Oct 30 19:51:14.354351 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:14.354304 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" event={"ID":"79cd91a7-7d69-46a4-b213-b36fed53e50b","Type":"ContainerDied","Data":"b573b8cfcb3098eec6c22107f9d9a1bb8d6b8a941996207387539fde58fe554d"} Oct 30 19:51:14.355213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:14.355135 5091 scope.go:122] "RemoveContainer" containerID="b573b8cfcb3098eec6c22107f9d9a1bb8d6b8a941996207387539fde58fe554d" Oct 30 19:51:14.357894 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:14.357857 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:51:14.358017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:14.357950 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:51:14.358017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:14.357998 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:51:15.340175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:15.339655 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:15.479171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:15.478557 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" event={"ID":"79cd91a7-7d69-46a4-b213-b36fed53e50b","Type":"ContainerStarted","Data":"2ea5cb7bc2f3a13e656c3b092879fd351a471b9c5e7716760c35c04d945abdff"} Oct 30 19:51:15.526179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:15.526132 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:15.677589 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:15.677485 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:15.779559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:15.779518 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:51:15.779758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:15.779649 5091 patch_prober.go:29] interesting pod/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:51:15.779758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:15.779696 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" podUID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 19:51:15.793013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:15.791834 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" podUID="25a56e07-b00d-4103-8e93-619e8af7abb5" containerName="payload-processing" containerID="cri-o://b420648d0f8cfb717edd21cfbb86f6c04f91b217490ed56cf607dbba1ea4cffc" gracePeriod=25 Oct 30 19:51:15.802227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:15.802206 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" Oct 30 19:51:15.812201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:15.812178 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" podUID="85f732fb-5e96-4788-9136-70a9011d8265" containerName="payload-pre-processing" containerID="cri-o://4f88180b03eb8ebba57b1b17cbd9465fcb4f1a9049f6dea31f618de12649ee82" gracePeriod=25 Oct 30 19:51:15.984221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:15.970994 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:16.023172 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:51:16.022312 5091 logging.go:55] [core] [Channel #1406 SubChannel #1407]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.131:50052", ServerName: "10.132.0.131:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.131:50052: connect: connection refused" Oct 30 19:51:16.202675 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.200027 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:16.237171 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:51:16.236355 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:51:16.761790 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.760324 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:16.804295 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.773744 5091 patch_prober.go:29] interesting pod/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:51:16.804295 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.773797 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" podUID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 19:51:16.904336 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.904043 5091 generic.go:356] "Generic (PLEG): container finished" podID="85f732fb-5e96-4788-9136-70a9011d8265" containerID="4f88180b03eb8ebba57b1b17cbd9465fcb4f1a9049f6dea31f618de12649ee82" exitCode=0 Oct 30 19:51:16.904336 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.904267 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" event={"ID":"85f732fb-5e96-4788-9136-70a9011d8265","Type":"ContainerDied","Data":"4f88180b03eb8ebba57b1b17cbd9465fcb4f1a9049f6dea31f618de12649ee82"} Oct 30 19:51:16.913575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.912713 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b"] Oct 30 19:51:16.915349 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.914037 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8b6a1c99-8256-4957-bd95-dc3f18f1d147" containerName="payload-processing" Oct 30 19:51:16.915349 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.914064 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b6a1c99-8256-4957-bd95-dc3f18f1d147" containerName="payload-processing" Oct 30 19:51:16.915349 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.914088 5091 cpu_manager.go:414] "containerMap: removing container" podUID="47f67a54-7bd4-4c38-b6d3-1e20d963aa1b" containerName="payload-pre-processing" Oct 30 19:51:16.915349 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.914097 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f67a54-7bd4-4c38-b6d3-1e20d963aa1b" containerName="payload-pre-processing" Oct 30 19:51:16.915349 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.914118 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4d8e498a-fce9-49d5-9eeb-5e784bcf66d7" containerName="revoke-keys" Oct 30 19:51:16.915349 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.914126 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d8e498a-fce9-49d5-9eeb-5e784bcf66d7" containerName="revoke-keys" Oct 30 19:51:16.918454 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.917279 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b6a1c99-8256-4957-bd95-dc3f18f1d147" containerName="payload-processing" Oct 30 19:51:16.918454 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.917299 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f67a54-7bd4-4c38-b6d3-1e20d963aa1b" containerName="payload-pre-processing" Oct 30 19:51:16.918454 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.917309 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d8e498a-fce9-49d5-9eeb-5e784bcf66d7" containerName="revoke-keys" Oct 30 19:51:16.920237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.918510 5091 generic.go:356] "Generic (PLEG): container finished" podID="25a56e07-b00d-4103-8e93-619e8af7abb5" containerID="b420648d0f8cfb717edd21cfbb86f6c04f91b217490ed56cf607dbba1ea4cffc" exitCode=0 Oct 30 19:51:16.920237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.919766 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" Oct 30 19:51:16.922028 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.921953 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" Oct 30 19:51:16.926918 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.926895 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b"] Oct 30 19:51:16.927028 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.926940 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" event={"ID":"25a56e07-b00d-4103-8e93-619e8af7abb5","Type":"ContainerDied","Data":"b420648d0f8cfb717edd21cfbb86f6c04f91b217490ed56cf607dbba1ea4cffc"} Oct 30 19:51:16.927028 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.926972 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" event={"ID":"25a56e07-b00d-4103-8e93-619e8af7abb5","Type":"ContainerDied","Data":"eb0bbedf2ef74a9ee9aa6ebd7080ad98a8fdcc85b0a102193ef4638713fa7ac6"} Oct 30 19:51:16.927028 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.927003 5091 scope.go:122] "RemoveContainer" containerID="b420648d0f8cfb717edd21cfbb86f6c04f91b217490ed56cf607dbba1ea4cffc" Oct 30 19:51:16.927248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.927230 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" Oct 30 19:51:16.934021 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.934003 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:51:16.934301 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.934264 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:51:16.975797 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.975769 5091 scope.go:122] "RemoveContainer" containerID="b420648d0f8cfb717edd21cfbb86f6c04f91b217490ed56cf607dbba1ea4cffc" Oct 30 19:51:16.976583 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:51:16.976502 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b420648d0f8cfb717edd21cfbb86f6c04f91b217490ed56cf607dbba1ea4cffc\": container with ID starting with b420648d0f8cfb717edd21cfbb86f6c04f91b217490ed56cf607dbba1ea4cffc not found: ID does not exist" containerID="b420648d0f8cfb717edd21cfbb86f6c04f91b217490ed56cf607dbba1ea4cffc" Oct 30 19:51:16.976583 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:16.976545 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b420648d0f8cfb717edd21cfbb86f6c04f91b217490ed56cf607dbba1ea4cffc"} err="failed to get container status \"b420648d0f8cfb717edd21cfbb86f6c04f91b217490ed56cf607dbba1ea4cffc\": rpc error: code = NotFound desc = could not find container \"b420648d0f8cfb717edd21cfbb86f6c04f91b217490ed56cf607dbba1ea4cffc\": container with ID starting with b420648d0f8cfb717edd21cfbb86f6c04f91b217490ed56cf607dbba1ea4cffc not found: ID does not exist" Oct 30 19:51:17.019709 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.019669 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" podUID="25a56e07-b00d-4103-8e93-619e8af7abb5" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.131:50052\" within 1s: context deadline exceeded" Oct 30 19:51:17.095846 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.095765 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25a56e07-b00d-4103-8e93-619e8af7abb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a56e07-b00d-4103-8e93-619e8af7abb5-plugins-config-volume\") pod \"25a56e07-b00d-4103-8e93-619e8af7abb5\" (UID: \"25a56e07-b00d-4103-8e93-619e8af7abb5\") " Oct 30 19:51:17.095991 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.095858 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85f732fb-5e96-4788-9136-70a9011d8265-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f732fb-5e96-4788-9136-70a9011d8265-plugins-config-volume\") pod \"85f732fb-5e96-4788-9136-70a9011d8265\" (UID: \"85f732fb-5e96-4788-9136-70a9011d8265\") " Oct 30 19:51:17.096056 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.096038 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25a56e07-b00d-4103-8e93-619e8af7abb5-kube-api-access-ndtm2\" (UniqueName: \"kubernetes.io/projected/25a56e07-b00d-4103-8e93-619e8af7abb5-kube-api-access-ndtm2\") pod \"25a56e07-b00d-4103-8e93-619e8af7abb5\" (UID: \"25a56e07-b00d-4103-8e93-619e8af7abb5\") " Oct 30 19:51:17.096307 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.096279 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25a56e07-b00d-4103-8e93-619e8af7abb5-plugins-config-volume" pod "25a56e07-b00d-4103-8e93-619e8af7abb5" (UID: "25a56e07-b00d-4103-8e93-619e8af7abb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:17.096378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.096311 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85f732fb-5e96-4788-9136-70a9011d8265-kube-api-access-r5jsp\" (UniqueName: \"kubernetes.io/projected/85f732fb-5e96-4788-9136-70a9011d8265-kube-api-access-r5jsp\") pod \"85f732fb-5e96-4788-9136-70a9011d8265\" (UID: \"85f732fb-5e96-4788-9136-70a9011d8265\") " Oct 30 19:51:17.096378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.096325 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85f732fb-5e96-4788-9136-70a9011d8265-plugins-config-volume" pod "85f732fb-5e96-4788-9136-70a9011d8265" (UID: "85f732fb-5e96-4788-9136-70a9011d8265"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:17.096817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.096795 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-040623-rll7b\" (UID: \"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" Oct 30 19:51:17.097005 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.096985 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-040623-rll7b\" (UID: \"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" Oct 30 19:51:17.097113 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.097099 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrxbv\" (UniqueName: \"kubernetes.io/projected/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-kube-api-access-wrxbv\") pod \"maas-api-revoke-keys-e2e-ipp-b-040623-rll7b\" (UID: \"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" Oct 30 19:51:17.097423 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.097399 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85f732fb-5e96-4788-9136-70a9011d8265-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:17.097505 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.097430 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25a56e07-b00d-4103-8e93-619e8af7abb5-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:17.098594 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.098570 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85f732fb-5e96-4788-9136-70a9011d8265-kube-api-access-r5jsp" pod "85f732fb-5e96-4788-9136-70a9011d8265" (UID: "85f732fb-5e96-4788-9136-70a9011d8265"). InnerVolumeSpecName "kube-api-access-r5jsp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:17.098687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.098666 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25a56e07-b00d-4103-8e93-619e8af7abb5-kube-api-access-ndtm2" pod "25a56e07-b00d-4103-8e93-619e8af7abb5" (UID: "25a56e07-b00d-4103-8e93-619e8af7abb5"). InnerVolumeSpecName "kube-api-access-ndtm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:17.198726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.198683 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-040623-rll7b\" (UID: \"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" Oct 30 19:51:17.198911 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.198780 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-040623-rll7b\" (UID: \"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" Oct 30 19:51:17.198911 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.198837 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrxbv\" (UniqueName: \"kubernetes.io/projected/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-kube-api-access-wrxbv\") pod \"maas-api-revoke-keys-e2e-ipp-b-040623-rll7b\" (UID: \"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" Oct 30 19:51:17.199032 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.198959 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5jsp\" (UniqueName: \"kubernetes.io/projected/85f732fb-5e96-4788-9136-70a9011d8265-kube-api-access-r5jsp\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:17.199032 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.198975 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndtm2\" (UniqueName: \"kubernetes.io/projected/25a56e07-b00d-4103-8e93-619e8af7abb5-kube-api-access-ndtm2\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:17.199479 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.199454 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-040623-rll7b\" (UID: \"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" Oct 30 19:51:17.201089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.201064 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-040623-rll7b\" (UID: \"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" Oct 30 19:51:17.213217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.213190 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrxbv\" (UniqueName: \"kubernetes.io/projected/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-kube-api-access-wrxbv\") pod \"maas-api-revoke-keys-e2e-ipp-b-040623-rll7b\" (UID: \"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" Oct 30 19:51:17.267168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.265349 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" Oct 30 19:51:17.582378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.581845 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b"] Oct 30 19:51:17.582708 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:51:17.582590 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fd188c5_e29e_45cf_8771_e3b9b12cfe9e.slice/crio-11021bc235b9de5c45c145e528865d6506281af106668f88b41a137512c188ba WatchSource:0}: Error finding container 11021bc235b9de5c45c145e528865d6506281af106668f88b41a137512c188ba: Status 404 returned error can't find the container with id 11021bc235b9de5c45c145e528865d6506281af106668f88b41a137512c188ba Oct 30 19:51:17.709288 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.709255 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:51:17.783299 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.783266 5091 patch_prober.go:29] interesting pod/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:51:17.783899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.783866 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" podUID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 19:51:17.811846 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.811818 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-kube-api-access-qqllx\" (UniqueName: \"kubernetes.io/projected/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-kube-api-access-qqllx\") pod \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " Oct 30 19:51:17.812206 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.812185 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-token\" (UniqueName: \"kubernetes.io/projected/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-token\") pod \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " Oct 30 19:51:17.812378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.812360 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-workload-certs\") pod \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " Oct 30 19:51:17.812581 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.812564 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istiod-ca-cert\") pod \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " Oct 30 19:51:17.812773 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.812755 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-data\") pod \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " Oct 30 19:51:17.813272 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.813242 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-workload-certs" pod "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" (UID: "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:17.813596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.813571 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istiod-ca-cert" pod "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" (UID: "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:17.813701 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.813680 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-credential-socket\") pod \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " Oct 30 19:51:17.813767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.813722 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-envoy\") pod \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " Oct 30 19:51:17.813936 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.813916 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-data" pod "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" (UID: "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:17.814500 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.814471 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-credential-socket" pod "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" (UID: "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:17.814612 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.814592 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-podinfo\") pod \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " Oct 30 19:51:17.814672 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.814639 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-workload-socket\") pod \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\" (UID: \"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57\") " Oct 30 19:51:17.815782 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.815754 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-workload-socket" pod "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" (UID: "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:17.816014 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.815993 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:17.816089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.816021 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:17.816089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.816037 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:17.816089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.816051 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:17.816089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.816064 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:17.817300 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.817280 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-envoy" pod "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" (UID: "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:17.817417 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.817293 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-token" pod "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" (UID: "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:17.817999 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.817972 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-kube-api-access-qqllx" pod "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" (UID: "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57"). InnerVolumeSpecName "kube-api-access-qqllx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:17.818969 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.818942 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-podinfo" pod "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" (UID: "1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:51:17.917998 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.917968 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:17.917998 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.917997 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:17.918195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.918007 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:17.918195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.918017 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqllx\" (UniqueName: \"kubernetes.io/projected/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57-kube-api-access-qqllx\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:17.928884 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.928858 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2" Oct 30 19:51:17.931061 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.931030 5091 generic.go:356] "Generic (PLEG): container finished" podID="69602ea5-4de9-4b3a-8e3b-a3045d919c82" containerID="01de8521f0a1db0c2ecdaa886a44908c271bf920b0eb7f9baf4f4b762bd27d3f" exitCode=0 Oct 30 19:51:17.931195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.931102 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" event={"ID":"69602ea5-4de9-4b3a-8e3b-a3045d919c82","Type":"ContainerDied","Data":"01de8521f0a1db0c2ecdaa886a44908c271bf920b0eb7f9baf4f4b762bd27d3f"} Oct 30 19:51:17.932704 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.932681 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" event={"ID":"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e","Type":"ContainerStarted","Data":"11021bc235b9de5c45c145e528865d6506281af106668f88b41a137512c188ba"} Oct 30 19:51:17.934449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.934423 5091 generic.go:356] "Generic (PLEG): container finished" podID="1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" containerID="7c5976ec74d6923e3a7861696dfd4231cce027ab2e8e451028747a33d5e2ba63" exitCode=0 Oct 30 19:51:17.934657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.934518 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" Oct 30 19:51:17.934768 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.934533 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" event={"ID":"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57","Type":"ContainerDied","Data":"7c5976ec74d6923e3a7861696dfd4231cce027ab2e8e451028747a33d5e2ba63"} Oct 30 19:51:17.934906 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.934802 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp" event={"ID":"1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57","Type":"ContainerDied","Data":"713c055635411394ba11188fbd289a2c7cd2474d6294d819317672b780e66d71"} Oct 30 19:51:17.934906 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.934828 5091 scope.go:122] "RemoveContainer" containerID="7c5976ec74d6923e3a7861696dfd4231cce027ab2e8e451028747a33d5e2ba63" Oct 30 19:51:17.938523 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.938502 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" event={"ID":"85f732fb-5e96-4788-9136-70a9011d8265","Type":"ContainerDied","Data":"3d136f1f165c6cdeba530198cee3919231d799b4cedd6d7f527caa2ce6bbca87"} Oct 30 19:51:17.938619 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.938606 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w" Oct 30 19:51:17.985941 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.985912 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2"] Oct 30 19:51:17.988892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.988871 5091 scope.go:122] "RemoveContainer" containerID="7c5976ec74d6923e3a7861696dfd4231cce027ab2e8e451028747a33d5e2ba63" Oct 30 19:51:17.989653 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:51:17.989266 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5976ec74d6923e3a7861696dfd4231cce027ab2e8e451028747a33d5e2ba63\": container with ID starting with 7c5976ec74d6923e3a7861696dfd4231cce027ab2e8e451028747a33d5e2ba63 not found: ID does not exist" containerID="7c5976ec74d6923e3a7861696dfd4231cce027ab2e8e451028747a33d5e2ba63" Oct 30 19:51:17.989653 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.989302 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5976ec74d6923e3a7861696dfd4231cce027ab2e8e451028747a33d5e2ba63"} err="failed to get container status \"7c5976ec74d6923e3a7861696dfd4231cce027ab2e8e451028747a33d5e2ba63\": rpc error: code = NotFound desc = could not find container \"7c5976ec74d6923e3a7861696dfd4231cce027ab2e8e451028747a33d5e2ba63\": container with ID starting with 7c5976ec74d6923e3a7861696dfd4231cce027ab2e8e451028747a33d5e2ba63 not found: ID does not exist" Oct 30 19:51:17.989653 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.989327 5091 scope.go:122] "RemoveContainer" containerID="4f88180b03eb8ebba57b1b17cbd9465fcb4f1a9049f6dea31f618de12649ee82" Oct 30 19:51:17.990667 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:17.990641 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e907e4-654594658b-fw8n2"] Oct 30 19:51:18.130928 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.130900 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w"] Oct 30 19:51:18.136444 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.136410 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e907e4-64d884bb9f-2cp6w"] Oct 30 19:51:18.144831 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.144808 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp"] Oct 30 19:51:18.186632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.186570 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-lvsnp"] Oct 30 19:51:18.224185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.224142 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:18.233972 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.233918 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:51:18.357770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.357718 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istiod-ca-cert\") pod \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " Oct 30 19:51:18.358042 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.358018 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-workload-socket\") pod \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " Oct 30 19:51:18.362277 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.362249 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-credential-socket\") pod \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " Oct 30 19:51:18.371369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.360641 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istiod-ca-cert" pod "69602ea5-4de9-4b3a-8e3b-a3045d919c82" (UID: "69602ea5-4de9-4b3a-8e3b-a3045d919c82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:18.371518 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.363361 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-credential-socket" pod "69602ea5-4de9-4b3a-8e3b-a3045d919c82" (UID: "69602ea5-4de9-4b3a-8e3b-a3045d919c82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:18.371646 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.371631 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-workload-certs\") pod \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " Oct 30 19:51:18.371856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.371838 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69602ea5-4de9-4b3a-8e3b-a3045d919c82-kube-api-access-rrwsb\" (UniqueName: \"kubernetes.io/projected/69602ea5-4de9-4b3a-8e3b-a3045d919c82-kube-api-access-rrwsb\") pod \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " Oct 30 19:51:18.372051 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.372034 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-envoy\") pod \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " Oct 30 19:51:18.372436 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.372417 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-data\") pod \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " Oct 30 19:51:18.376660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.376353 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-workload-certs" pod "69602ea5-4de9-4b3a-8e3b-a3045d919c82" (UID: "69602ea5-4de9-4b3a-8e3b-a3045d919c82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:18.376660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.376368 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-podinfo\") pod \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " Oct 30 19:51:18.376660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.376463 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-token\" (UniqueName: \"kubernetes.io/projected/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-token\") pod \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\" (UID: \"69602ea5-4de9-4b3a-8e3b-a3045d919c82\") " Oct 30 19:51:18.377237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.377194 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-data" pod "69602ea5-4de9-4b3a-8e3b-a3045d919c82" (UID: "69602ea5-4de9-4b3a-8e3b-a3045d919c82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:18.377237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.377209 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-workload-socket" pod "69602ea5-4de9-4b3a-8e3b-a3045d919c82" (UID: "69602ea5-4de9-4b3a-8e3b-a3045d919c82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:18.377968 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.377941 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:18.378046 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.377975 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:18.378046 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.377992 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:18.378046 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.378009 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:18.378046 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.378023 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:18.379829 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.379808 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-token" pod "69602ea5-4de9-4b3a-8e3b-a3045d919c82" (UID: "69602ea5-4de9-4b3a-8e3b-a3045d919c82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:18.380566 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.380541 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69602ea5-4de9-4b3a-8e3b-a3045d919c82-kube-api-access-rrwsb" pod "69602ea5-4de9-4b3a-8e3b-a3045d919c82" (UID: "69602ea5-4de9-4b3a-8e3b-a3045d919c82"). InnerVolumeSpecName "kube-api-access-rrwsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:18.380750 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.380729 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-podinfo" pod "69602ea5-4de9-4b3a-8e3b-a3045d919c82" (UID: "69602ea5-4de9-4b3a-8e3b-a3045d919c82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:51:18.381059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.381028 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-envoy" pod "69602ea5-4de9-4b3a-8e3b-a3045d919c82" (UID: "69602ea5-4de9-4b3a-8e3b-a3045d919c82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:18.443514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.443482 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" path="/var/lib/kubelet/pods/1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57/volumes" Oct 30 19:51:18.444129 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.444100 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25a56e07-b00d-4103-8e93-619e8af7abb5" path="/var/lib/kubelet/pods/25a56e07-b00d-4103-8e93-619e8af7abb5/volumes" Oct 30 19:51:18.444776 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.444752 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85f732fb-5e96-4788-9136-70a9011d8265" path="/var/lib/kubelet/pods/85f732fb-5e96-4788-9136-70a9011d8265/volumes" Oct 30 19:51:18.479708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.479673 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:18.479708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.479709 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:18.479885 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.479724 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrwsb\" (UniqueName: \"kubernetes.io/projected/69602ea5-4de9-4b3a-8e3b-a3045d919c82-kube-api-access-rrwsb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:18.479885 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.479740 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69602ea5-4de9-4b3a-8e3b-a3045d919c82-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:18.774542 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.774507 5091 patch_prober.go:29] interesting pod/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:51:18.774807 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.774779 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" podUID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 19:51:18.970356 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.970317 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" event={"ID":"69602ea5-4de9-4b3a-8e3b-a3045d919c82","Type":"ContainerDied","Data":"7982f89f9f061becdfe2270c6c0b4316c1154afe56f5cd3ad3d69d0ef7460fc4"} Oct 30 19:51:18.971429 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.971405 5091 scope.go:122] "RemoveContainer" containerID="01de8521f0a1db0c2ecdaa886a44908c271bf920b0eb7f9baf4f4b762bd27d3f" Oct 30 19:51:18.971661 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.970947 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8" Oct 30 19:51:18.980266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.980225 5091 generic.go:356] "Generic (PLEG): container finished" podID="3fd188c5-e29e-45cf-8771-e3b9b12cfe9e" containerID="891af3d7fa9403472eb00775eaae66bae9ebb0b6124cfc54085d05c40a16d42f" exitCode=0 Oct 30 19:51:18.980574 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:18.980548 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" event={"ID":"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e","Type":"ContainerDied","Data":"891af3d7fa9403472eb00775eaae66bae9ebb0b6124cfc54085d05c40a16d42f"} Oct 30 19:51:19.085694 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:19.085614 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8"] Oct 30 19:51:19.095877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:19.095835 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-wcbh8"] Oct 30 19:51:19.346734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:19.346645 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:19.777539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:19.777501 5091 patch_prober.go:29] interesting pod/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:51:19.777829 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:19.777800 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" podUID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 19:51:20.186080 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.186044 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:20.333546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.333520 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" Oct 30 19:51:20.444326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.444251 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69602ea5-4de9-4b3a-8e3b-a3045d919c82" path="/var/lib/kubelet/pods/69602ea5-4de9-4b3a-8e3b-a3045d919c82/volumes" Oct 30 19:51:20.509683 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.509652 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-maas-api-xks-ca\") pod \"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e\" (UID: \"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e\") " Oct 30 19:51:20.509868 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.509699 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-maas-api-openshift-ca\") pod \"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e\" (UID: \"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e\") " Oct 30 19:51:20.509868 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.509718 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-kube-api-access-wrxbv\" (UniqueName: \"kubernetes.io/projected/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-kube-api-access-wrxbv\") pod \"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e\" (UID: \"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e\") " Oct 30 19:51:20.510065 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.510032 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-maas-api-openshift-ca" pod "3fd188c5-e29e-45cf-8771-e3b9b12cfe9e" (UID: "3fd188c5-e29e-45cf-8771-e3b9b12cfe9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:20.510322 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.510302 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:20.512013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.511989 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-kube-api-access-wrxbv" pod "3fd188c5-e29e-45cf-8771-e3b9b12cfe9e" (UID: "3fd188c5-e29e-45cf-8771-e3b9b12cfe9e"). InnerVolumeSpecName "kube-api-access-wrxbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:20.512457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.512011 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-maas-api-xks-ca" pod "3fd188c5-e29e-45cf-8771-e3b9b12cfe9e" (UID: "3fd188c5-e29e-45cf-8771-e3b9b12cfe9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:20.612517 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.612484 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:20.612678 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.612528 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrxbv\" (UniqueName: \"kubernetes.io/projected/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e-kube-api-access-wrxbv\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:20.729907 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.729830 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:51:20.737139 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.737114 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:51:20.773646 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.773608 5091 patch_prober.go:29] interesting pod/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" start-of-body= Oct 30 19:51:20.773815 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:20.773671 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" podUID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.134:15021/healthz/ready\": dial tcp 10.132.0.134:15021: connect: connection refused" Oct 30 19:51:21.003388 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.003352 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" event={"ID":"3fd188c5-e29e-45cf-8771-e3b9b12cfe9e","Type":"ContainerDied","Data":"11021bc235b9de5c45c145e528865d6506281af106668f88b41a137512c188ba"} Oct 30 19:51:21.003555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.003436 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11021bc235b9de5c45c145e528865d6506281af106668f88b41a137512c188ba" Oct 30 19:51:21.003555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.003371 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b" Oct 30 19:51:21.037747 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.037715 5091 scope.go:122] "RemoveContainer" containerID="0f352e1c7af0bf83bf1f7d828ff1da393847d4de4d7edf17c5cdcd6b6748e96e" Oct 30 19:51:21.057037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.057002 5091 scope.go:122] "RemoveContainer" containerID="cb25472ae1b8d2349cd50d95945719250f987681d7ece4e04d254c6353f67e63" Oct 30 19:51:21.071634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.071611 5091 scope.go:122] "RemoveContainer" containerID="ffb11d3e7901d13df1b5ffae1fbbec09e9ecf36210545cfba515a37e6fe98362" Oct 30 19:51:21.082942 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.082911 5091 scope.go:122] "RemoveContainer" containerID="6fbc5aaf4fc82f744801fdf135b42ba839a6c5a707d051196e4793b47a99502f" Oct 30 19:51:21.085718 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.085685 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:21.092751 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.092730 5091 scope.go:122] "RemoveContainer" containerID="63e39f8ea3567cd95d67e253c56201ea6cdfe08d89da44e007d5f65f0972e363" Oct 30 19:51:21.101872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.101857 5091 scope.go:122] "RemoveContainer" containerID="6189afd4db11ba12f5c4e3643433c23ec09188306a6f72ee975a9e605f653586" Oct 30 19:51:21.110770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.110754 5091 scope.go:122] "RemoveContainer" containerID="324a95a81392ff1d07dd412e16fcc13064c260c7f03dd24ac4382eecae25c533" Oct 30 19:51:21.119667 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.119647 5091 scope.go:122] "RemoveContainer" containerID="8cecc0433cb3e81cac9b2391fc78560cc9fe977771aa86d1371b47b02a9e56df" Oct 30 19:51:21.128267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.128248 5091 scope.go:122] "RemoveContainer" containerID="e90ab94e108c38259a5e6ec9c661d0d8092f387662e486cd9e5969b16fee629a" Oct 30 19:51:21.137282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.137263 5091 scope.go:122] "RemoveContainer" containerID="9ae0dbeba10bf7ceadae63779255d22566515bbab5f7bfc600712ad1a28a6f25" Oct 30 19:51:21.243122 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.237293 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:21.536800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.536696 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:21.792216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.791986 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:51:21.796168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.793598 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:51:21.796343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.796312 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:51:21.989671 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.989637 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577"] Oct 30 19:51:21.990338 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.990323 5091 cpu_manager.go:414] "containerMap: removing container" podUID="25a56e07-b00d-4103-8e93-619e8af7abb5" containerName="payload-processing" Oct 30 19:51:21.990387 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.990339 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25a56e07-b00d-4103-8e93-619e8af7abb5" containerName="payload-processing" Oct 30 19:51:21.990387 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.990356 5091 cpu_manager.go:414] "containerMap: removing container" podUID="69602ea5-4de9-4b3a-8e3b-a3045d919c82" containerName="istio-proxy" Oct 30 19:51:21.990387 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.990362 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69602ea5-4de9-4b3a-8e3b-a3045d919c82" containerName="istio-proxy" Oct 30 19:51:21.990387 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.990377 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3fd188c5-e29e-45cf-8771-e3b9b12cfe9e" containerName="revoke-keys" Oct 30 19:51:21.990387 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.990382 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fd188c5-e29e-45cf-8771-e3b9b12cfe9e" containerName="revoke-keys" Oct 30 19:51:21.990536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.990399 5091 cpu_manager.go:414] "containerMap: removing container" podUID="85f732fb-5e96-4788-9136-70a9011d8265" containerName="payload-pre-processing" Oct 30 19:51:21.990536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.990404 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85f732fb-5e96-4788-9136-70a9011d8265" containerName="payload-pre-processing" Oct 30 19:51:21.990536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.990413 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" containerName="istio-proxy" Oct 30 19:51:21.990536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.990418 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" containerName="istio-proxy" Oct 30 19:51:21.990536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.990525 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="25a56e07-b00d-4103-8e93-619e8af7abb5" containerName="payload-processing" Oct 30 19:51:21.990536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.990535 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="69602ea5-4de9-4b3a-8e3b-a3045d919c82" containerName="istio-proxy" Oct 30 19:51:21.990730 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.990544 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fd188c5-e29e-45cf-8771-e3b9b12cfe9e" containerName="revoke-keys" Oct 30 19:51:21.990730 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.990551 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="85f732fb-5e96-4788-9136-70a9011d8265" containerName="payload-pre-processing" Oct 30 19:51:21.990730 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.990558 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b7a5ece-deae-4bfe-b0c3-3f45da9e9f57" containerName="istio-proxy" Oct 30 19:51:21.997945 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:21.997927 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" Oct 30 19:51:22.006012 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.005982 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577"] Oct 30 19:51:22.006917 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.006895 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:51:22.007027 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.006966 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:51:22.155398 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.155297 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z58hh\" (UniqueName: \"kubernetes.io/projected/553c0e2c-3812-4e18-b527-a5f74d006e27-kube-api-access-z58hh\") pod \"maas-api-revoke-keys-e2e-api-b-2beb7b-zm577\" (UID: \"553c0e2c-3812-4e18-b527-a5f74d006e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" Oct 30 19:51:22.155598 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.155569 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/553c0e2c-3812-4e18-b527-a5f74d006e27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2beb7b-zm577\" (UID: \"553c0e2c-3812-4e18-b527-a5f74d006e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" Oct 30 19:51:22.155816 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.155789 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/553c0e2c-3812-4e18-b527-a5f74d006e27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2beb7b-zm577\" (UID: \"553c0e2c-3812-4e18-b527-a5f74d006e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" Oct 30 19:51:22.256799 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.256737 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z58hh\" (UniqueName: \"kubernetes.io/projected/553c0e2c-3812-4e18-b527-a5f74d006e27-kube-api-access-z58hh\") pod \"maas-api-revoke-keys-e2e-api-b-2beb7b-zm577\" (UID: \"553c0e2c-3812-4e18-b527-a5f74d006e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" Oct 30 19:51:22.257213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.256886 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/553c0e2c-3812-4e18-b527-a5f74d006e27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2beb7b-zm577\" (UID: \"553c0e2c-3812-4e18-b527-a5f74d006e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" Oct 30 19:51:22.257213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.256967 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/553c0e2c-3812-4e18-b527-a5f74d006e27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2beb7b-zm577\" (UID: \"553c0e2c-3812-4e18-b527-a5f74d006e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" Oct 30 19:51:22.257869 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.257836 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/553c0e2c-3812-4e18-b527-a5f74d006e27-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2beb7b-zm577\" (UID: \"553c0e2c-3812-4e18-b527-a5f74d006e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" Oct 30 19:51:22.259398 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.259376 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/553c0e2c-3812-4e18-b527-a5f74d006e27-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2beb7b-zm577\" (UID: \"553c0e2c-3812-4e18-b527-a5f74d006e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" Oct 30 19:51:22.271885 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.271858 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z58hh\" (UniqueName: \"kubernetes.io/projected/553c0e2c-3812-4e18-b527-a5f74d006e27-kube-api-access-z58hh\") pod \"maas-api-revoke-keys-e2e-api-b-2beb7b-zm577\" (UID: \"553c0e2c-3812-4e18-b527-a5f74d006e27\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" Oct 30 19:51:22.316981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.316951 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" Oct 30 19:51:22.662797 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.659804 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:22.672315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.671355 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577"] Oct 30 19:51:22.942134 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.942029 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx"] Oct 30 19:51:22.942339 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.942296 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" podUID="036310d3-eb3a-410b-8716-886f139c818c" containerName="istio-proxy" containerID="cri-o://89ebf0935a92e280530073a5021ebccb63e811aa67c619791d4dec6a7897b67e" gracePeriod=30 Oct 30 19:51:22.946691 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.946662 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2"] Oct 30 19:51:22.946899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:22.946878 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" podUID="5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" containerName="istio-proxy" containerID="cri-o://e85529baea4d463925b09ba162c122fccfa0c9f7fd8b805d2bd60a73c636c15b" gracePeriod=30 Oct 30 19:51:23.023212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:23.023174 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" event={"ID":"553c0e2c-3812-4e18-b527-a5f74d006e27","Type":"ContainerStarted","Data":"8ba078c472a7b83a40e9d39c184fefbb30e63f3d44dce24ded81205cec3f4cd7"} Oct 30 19:51:23.629168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:23.628311 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:24.046505 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:24.046464 5091 generic.go:356] "Generic (PLEG): container finished" podID="553c0e2c-3812-4e18-b527-a5f74d006e27" containerID="23192f8ce42fb233ee559a275b4fb4f5818fba05bf41eea328fd76fd27fdcedf" exitCode=0 Oct 30 19:51:24.046669 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:24.046549 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" event={"ID":"553c0e2c-3812-4e18-b527-a5f74d006e27","Type":"ContainerDied","Data":"23192f8ce42fb233ee559a275b4fb4f5818fba05bf41eea328fd76fd27fdcedf"} Oct 30 19:51:24.734169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:24.727745 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:25.243596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:25.243569 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" Oct 30 19:51:25.323878 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:25.323794 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/553c0e2c-3812-4e18-b527-a5f74d006e27-kube-api-access-z58hh\" (UniqueName: \"kubernetes.io/projected/553c0e2c-3812-4e18-b527-a5f74d006e27-kube-api-access-z58hh\") pod \"553c0e2c-3812-4e18-b527-a5f74d006e27\" (UID: \"553c0e2c-3812-4e18-b527-a5f74d006e27\") " Oct 30 19:51:25.324050 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:25.323957 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/553c0e2c-3812-4e18-b527-a5f74d006e27-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/553c0e2c-3812-4e18-b527-a5f74d006e27-maas-api-xks-ca\") pod \"553c0e2c-3812-4e18-b527-a5f74d006e27\" (UID: \"553c0e2c-3812-4e18-b527-a5f74d006e27\") " Oct 30 19:51:25.324221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:25.324200 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/553c0e2c-3812-4e18-b527-a5f74d006e27-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/553c0e2c-3812-4e18-b527-a5f74d006e27-maas-api-openshift-ca\") pod \"553c0e2c-3812-4e18-b527-a5f74d006e27\" (UID: \"553c0e2c-3812-4e18-b527-a5f74d006e27\") " Oct 30 19:51:25.324648 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:25.324615 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/553c0e2c-3812-4e18-b527-a5f74d006e27-maas-api-openshift-ca" pod "553c0e2c-3812-4e18-b527-a5f74d006e27" (UID: "553c0e2c-3812-4e18-b527-a5f74d006e27"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:25.324926 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:25.324903 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/553c0e2c-3812-4e18-b527-a5f74d006e27-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:25.326123 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:25.326096 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/553c0e2c-3812-4e18-b527-a5f74d006e27-kube-api-access-z58hh" pod "553c0e2c-3812-4e18-b527-a5f74d006e27" (UID: "553c0e2c-3812-4e18-b527-a5f74d006e27"). InnerVolumeSpecName "kube-api-access-z58hh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:25.326358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:25.326333 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/553c0e2c-3812-4e18-b527-a5f74d006e27-maas-api-xks-ca" pod "553c0e2c-3812-4e18-b527-a5f74d006e27" (UID: "553c0e2c-3812-4e18-b527-a5f74d006e27"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:25.426213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:25.426182 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z58hh\" (UniqueName: \"kubernetes.io/projected/553c0e2c-3812-4e18-b527-a5f74d006e27-kube-api-access-z58hh\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:25.426213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:25.426210 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/553c0e2c-3812-4e18-b527-a5f74d006e27-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:26.103412 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:26.102881 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:26.108393 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:26.108356 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" event={"ID":"553c0e2c-3812-4e18-b527-a5f74d006e27","Type":"ContainerDied","Data":"8ba078c472a7b83a40e9d39c184fefbb30e63f3d44dce24ded81205cec3f4cd7"} Oct 30 19:51:26.108569 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:26.108419 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ba078c472a7b83a40e9d39c184fefbb30e63f3d44dce24ded81205cec3f4cd7" Oct 30 19:51:26.108569 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:26.108478 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577" Oct 30 19:51:27.145318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:27.145280 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:27.942729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:27.937223 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:28.128497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.126791 5091 generic.go:356] "Generic (PLEG): container finished" podID="5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" containerID="e85529baea4d463925b09ba162c122fccfa0c9f7fd8b805d2bd60a73c636c15b" exitCode=0 Oct 30 19:51:28.128497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.127029 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" event={"ID":"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58","Type":"ContainerDied","Data":"e85529baea4d463925b09ba162c122fccfa0c9f7fd8b805d2bd60a73c636c15b"} Oct 30 19:51:28.129624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.129599 5091 generic.go:356] "Generic (PLEG): container finished" podID="036310d3-eb3a-410b-8716-886f139c818c" containerID="89ebf0935a92e280530073a5021ebccb63e811aa67c619791d4dec6a7897b67e" exitCode=0 Oct 30 19:51:28.129730 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.129655 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" event={"ID":"036310d3-eb3a-410b-8716-886f139c818c","Type":"ContainerDied","Data":"89ebf0935a92e280530073a5021ebccb63e811aa67c619791d4dec6a7897b67e"} Oct 30 19:51:28.294877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.294849 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:51:28.298279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.298256 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:51:28.413381 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.413348 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036310d3-eb3a-410b-8716-886f139c818c-istio-token\" (UniqueName: \"kubernetes.io/projected/036310d3-eb3a-410b-8716-886f139c818c-istio-token\") pod \"036310d3-eb3a-410b-8716-886f139c818c\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " Oct 30 19:51:28.413552 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.413446 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-workload-socket\") pod \"036310d3-eb3a-410b-8716-886f139c818c\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " Oct 30 19:51:28.413552 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.413505 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-workload-certs\") pod \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " Oct 30 19:51:28.413662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.413588 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/036310d3-eb3a-410b-8716-886f139c818c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036310d3-eb3a-410b-8716-886f139c818c-istiod-ca-cert\") pod \"036310d3-eb3a-410b-8716-886f139c818c\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " Oct 30 19:51:28.413728 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.413692 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/036310d3-eb3a-410b-8716-886f139c818c-kube-api-access-j9lhv\" (UniqueName: \"kubernetes.io/projected/036310d3-eb3a-410b-8716-886f139c818c-kube-api-access-j9lhv\") pod \"036310d3-eb3a-410b-8716-886f139c818c\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " Oct 30 19:51:28.413787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.413777 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-workload-socket\") pod \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " Oct 30 19:51:28.413848 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.413761 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-workload-socket" pod "036310d3-eb3a-410b-8716-886f139c818c" (UID: "036310d3-eb3a-410b-8716-886f139c818c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:28.413907 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.413843 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-workload-certs" pod "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" (UID: "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:28.413962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.413899 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-token\" (UniqueName: \"kubernetes.io/projected/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-token\") pod \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " Oct 30 19:51:28.413962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.413950 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istiod-ca-cert\") pod \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " Oct 30 19:51:28.414074 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.413978 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-credential-socket\") pod \"036310d3-eb3a-410b-8716-886f139c818c\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " Oct 30 19:51:28.414074 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.414015 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-workload-certs\") pod \"036310d3-eb3a-410b-8716-886f139c818c\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " Oct 30 19:51:28.414205 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.414080 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-istio-envoy\") pod \"036310d3-eb3a-410b-8716-886f139c818c\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " Oct 30 19:51:28.414205 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.414140 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-data\") pod \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " Oct 30 19:51:28.414319 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.414212 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-podinfo\") pod \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " Oct 30 19:51:28.414319 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.414247 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/036310d3-eb3a-410b-8716-886f139c818c-istiod-ca-cert" pod "036310d3-eb3a-410b-8716-886f139c818c" (UID: "036310d3-eb3a-410b-8716-886f139c818c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:28.414319 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.414257 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-envoy\") pod \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " Oct 30 19:51:28.414486 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.414332 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-istio-data\") pod \"036310d3-eb3a-410b-8716-886f139c818c\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " Oct 30 19:51:28.414486 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.414369 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/036310d3-eb3a-410b-8716-886f139c818c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036310d3-eb3a-410b-8716-886f139c818c-istio-podinfo\") pod \"036310d3-eb3a-410b-8716-886f139c818c\" (UID: \"036310d3-eb3a-410b-8716-886f139c818c\") " Oct 30 19:51:28.414486 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.414469 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-credential-socket\") pod \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " Oct 30 19:51:28.414650 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.414516 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-kube-api-access-8stcn\" (UniqueName: \"kubernetes.io/projected/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-kube-api-access-8stcn\") pod \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\" (UID: \"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58\") " Oct 30 19:51:28.416415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.415418 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-workload-certs" pod "036310d3-eb3a-410b-8716-886f139c818c" (UID: "036310d3-eb3a-410b-8716-886f139c818c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:28.416415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.415508 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.416415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.415534 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.416415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.415567 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/036310d3-eb3a-410b-8716-886f139c818c-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.416415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.415641 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-data" pod "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" (UID: "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:28.416415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.415754 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istiod-ca-cert" pod "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" (UID: "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:28.416415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.415777 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-istio-data" pod "036310d3-eb3a-410b-8716-886f139c818c" (UID: "036310d3-eb3a-410b-8716-886f139c818c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:28.416415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.416090 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-workload-socket" pod "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" (UID: "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:28.416415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.416133 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-credential-socket" pod "036310d3-eb3a-410b-8716-886f139c818c" (UID: "036310d3-eb3a-410b-8716-886f139c818c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:28.416415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.416325 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036310d3-eb3a-410b-8716-886f139c818c-kube-api-access-j9lhv" pod "036310d3-eb3a-410b-8716-886f139c818c" (UID: "036310d3-eb3a-410b-8716-886f139c818c"). InnerVolumeSpecName "kube-api-access-j9lhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:28.416415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.416383 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/036310d3-eb3a-410b-8716-886f139c818c-istio-token" pod "036310d3-eb3a-410b-8716-886f139c818c" (UID: "036310d3-eb3a-410b-8716-886f139c818c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:28.416415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.416359 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-credential-socket" pod "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" (UID: "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:28.417114 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.416648 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-token" pod "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" (UID: "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:28.417114 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.416766 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-envoy" pod "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" (UID: "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:28.417935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.417916 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/036310d3-eb3a-410b-8716-886f139c818c-istio-podinfo" pod "036310d3-eb3a-410b-8716-886f139c818c" (UID: "036310d3-eb3a-410b-8716-886f139c818c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:51:28.418160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.418124 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-istio-envoy" pod "036310d3-eb3a-410b-8716-886f139c818c" (UID: "036310d3-eb3a-410b-8716-886f139c818c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:28.418560 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.418540 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-podinfo" pod "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" (UID: "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:51:28.418626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.418558 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-kube-api-access-8stcn" pod "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" (UID: "5e5b9c8e-97eb-4c35-8032-92bb7acb8d58"). InnerVolumeSpecName "kube-api-access-8stcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:28.524086 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.524044 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.546641 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.546610 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.546878 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.546856 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.546994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.546980 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.547085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.547073 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.547207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.547192 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/036310d3-eb3a-410b-8716-886f139c818c-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.547308 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.547295 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.547402 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.547386 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8stcn\" (UniqueName: \"kubernetes.io/projected/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-kube-api-access-8stcn\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.547596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.547485 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/036310d3-eb3a-410b-8716-886f139c818c-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.547596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.547506 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9lhv\" (UniqueName: \"kubernetes.io/projected/036310d3-eb3a-410b-8716-886f139c818c-kube-api-access-j9lhv\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.547596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.547521 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.547596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.547535 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.547596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.547548 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.547596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.547563 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.547596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.547576 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/036310d3-eb3a-410b-8716-886f139c818c-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:28.969061 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:28.968900 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:29.139689 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:29.139658 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" Oct 30 19:51:29.139854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:29.139683 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2" event={"ID":"5e5b9c8e-97eb-4c35-8032-92bb7acb8d58","Type":"ContainerDied","Data":"d44046cc4e6778d41154f5df4188557411e025b6126b4b7242ca38f51cc07d91"} Oct 30 19:51:29.139854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:29.139750 5091 scope.go:122] "RemoveContainer" containerID="e85529baea4d463925b09ba162c122fccfa0c9f7fd8b805d2bd60a73c636c15b" Oct 30 19:51:29.141945 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:29.141898 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" event={"ID":"036310d3-eb3a-410b-8716-886f139c818c","Type":"ContainerDied","Data":"59f4e765536af398d1de81079b516a203883bd58acdbd912982ae76fe5c1f15c"} Oct 30 19:51:29.141945 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:29.141928 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx" Oct 30 19:51:29.151977 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:29.151959 5091 scope.go:122] "RemoveContainer" containerID="89ebf0935a92e280530073a5021ebccb63e811aa67c619791d4dec6a7897b67e" Oct 30 19:51:29.167545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:29.167514 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2"] Oct 30 19:51:29.174480 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:29.174456 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-2qpx2"] Oct 30 19:51:29.189336 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:29.189307 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx"] Oct 30 19:51:29.192960 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:29.192935 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-67dtx"] Oct 30 19:51:30.046189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:30.043739 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:30.444689 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:30.444613 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="036310d3-eb3a-410b-8716-886f139c818c" path="/var/lib/kubelet/pods/036310d3-eb3a-410b-8716-886f139c818c/volumes" Oct 30 19:51:30.449171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:30.445261 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" path="/var/lib/kubelet/pods/5e5b9c8e-97eb-4c35-8032-92bb7acb8d58/volumes" Oct 30 19:51:30.821097 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:30.818719 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:31.066173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:31.049713 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:31.383822 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:31.383793 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:31.529966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:31.529930 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b"] Oct 30 19:51:31.539038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:31.538998 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-040623-rll7b"] Oct 30 19:51:31.597750 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:31.596404 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:31.796871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:31.796829 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:32.255167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:32.252576 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:32.279824 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:32.279785 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l"] Oct 30 19:51:32.443375 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:32.443342 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fd188c5-e29e-45cf-8771-e3b9b12cfe9e" path="/var/lib/kubelet/pods/3fd188c5-e29e-45cf-8771-e3b9b12cfe9e/volumes" Oct 30 19:51:32.527225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:32.521632 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt"] Oct 30 19:51:32.527225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:32.521857 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" podUID="d037606e-0d57-4527-b5a4-fa485eb976f3" containerName="istio-proxy" containerID="cri-o://1db9a07cdd1181dea78fdc6eacbf6f9ac462c88e41a90fe3fe72f3b5faa8ef62" gracePeriod=30 Oct 30 19:51:32.527225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:32.523703 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:32.629419 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:32.629380 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k"] Oct 30 19:51:32.629927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:32.629890 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" podUID="46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" containerName="istio-proxy" containerID="cri-o://43be83664ad5a73529879f057a787f6bc22dc5c75ef77db6f586fe2da0e80576" gracePeriod=30 Oct 30 19:51:32.786128 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:32.786049 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:33.128536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:33.128440 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:34.130260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:34.130217 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:34.995901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:34.995869 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:35.282575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:35.282534 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:35.475103 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:35.475064 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6"] Oct 30 19:51:35.500376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:35.500344 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh"] Oct 30 19:51:35.505404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:35.505316 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e907e4-86gqh"] Oct 30 19:51:35.507038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:35.506803 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:35.758668 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:35.758595 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:36.132073 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:36.132032 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:36.446897 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:36.446816 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d8e498a-fce9-49d5-9eeb-5e784bcf66d7" path="/var/lib/kubelet/pods/4d8e498a-fce9-49d5-9eeb-5e784bcf66d7/volumes" Oct 30 19:51:37.280384 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.280187 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" podUID="fc0f7a23-6124-419b-9b33-11c6ac5f4276" containerName="payload-processing" containerID="cri-o://a552591f13f6d618212700cfe53b0bba297adcacef6fa185c09b89416bfd14a8" gracePeriod=25 Oct 30 19:51:37.357498 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.357451 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:37.434010 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:51:37.433975 5091 logging.go:55] [core] [Channel #1471 SubChannel #1472]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.133:50052", ServerName: "10.132.0.133:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.133:50052: connect: connection refused" Oct 30 19:51:37.502502 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.502470 5091 generic.go:356] "Generic (PLEG): container finished" podID="fc0f7a23-6124-419b-9b33-11c6ac5f4276" containerID="a552591f13f6d618212700cfe53b0bba297adcacef6fa185c09b89416bfd14a8" exitCode=0 Oct 30 19:51:37.502913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.502615 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" event={"ID":"fc0f7a23-6124-419b-9b33-11c6ac5f4276","Type":"ContainerDied","Data":"a552591f13f6d618212700cfe53b0bba297adcacef6fa185c09b89416bfd14a8"} Oct 30 19:51:37.646898 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.645322 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" Oct 30 19:51:37.816166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.815183 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0f7a23-6124-419b-9b33-11c6ac5f4276-kube-api-access-pdnqb\" (UniqueName: \"kubernetes.io/projected/fc0f7a23-6124-419b-9b33-11c6ac5f4276-kube-api-access-pdnqb\") pod \"fc0f7a23-6124-419b-9b33-11c6ac5f4276\" (UID: \"fc0f7a23-6124-419b-9b33-11c6ac5f4276\") " Oct 30 19:51:37.816166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.815253 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc0f7a23-6124-419b-9b33-11c6ac5f4276-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0f7a23-6124-419b-9b33-11c6ac5f4276-plugins-config-volume\") pod \"fc0f7a23-6124-419b-9b33-11c6ac5f4276\" (UID: \"fc0f7a23-6124-419b-9b33-11c6ac5f4276\") " Oct 30 19:51:37.819688 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.819479 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0f7a23-6124-419b-9b33-11c6ac5f4276-plugins-config-volume" pod "fc0f7a23-6124-419b-9b33-11c6ac5f4276" (UID: "fc0f7a23-6124-419b-9b33-11c6ac5f4276"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:37.839385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.832821 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0f7a23-6124-419b-9b33-11c6ac5f4276-kube-api-access-pdnqb" pod "fc0f7a23-6124-419b-9b33-11c6ac5f4276" (UID: "fc0f7a23-6124-419b-9b33-11c6ac5f4276"). InnerVolumeSpecName "kube-api-access-pdnqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:37.883180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.879780 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:51:37.912947 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.912921 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:51:37.919481 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.919455 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-podinfo\") pod \"d037606e-0d57-4527-b5a4-fa485eb976f3\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " Oct 30 19:51:37.919621 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.919599 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-podinfo\") pod \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " Oct 30 19:51:37.919697 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.919657 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-kube-api-access-wlqw8\" (UniqueName: \"kubernetes.io/projected/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-kube-api-access-wlqw8\") pod \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " Oct 30 19:51:37.919760 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.919724 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-token\" (UniqueName: \"kubernetes.io/projected/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-token\") pod \"d037606e-0d57-4527-b5a4-fa485eb976f3\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " Oct 30 19:51:37.919820 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.919758 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-data\") pod \"d037606e-0d57-4527-b5a4-fa485eb976f3\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " Oct 30 19:51:37.919873 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.919828 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-workload-socket\") pod \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " Oct 30 19:51:37.919873 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.919860 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-workload-socket\") pod \"d037606e-0d57-4527-b5a4-fa485eb976f3\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " Oct 30 19:51:37.919973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.919907 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d037606e-0d57-4527-b5a4-fa485eb976f3-kube-api-access-wghc8\" (UniqueName: \"kubernetes.io/projected/d037606e-0d57-4527-b5a4-fa485eb976f3-kube-api-access-wghc8\") pod \"d037606e-0d57-4527-b5a4-fa485eb976f3\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " Oct 30 19:51:37.920036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.920000 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-workload-certs\") pod \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " Oct 30 19:51:37.920086 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.920061 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istiod-ca-cert\") pod \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " Oct 30 19:51:37.920157 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.920093 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d037606e-0d57-4527-b5a4-fa485eb976f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d037606e-0d57-4527-b5a4-fa485eb976f3-istiod-ca-cert\") pod \"d037606e-0d57-4527-b5a4-fa485eb976f3\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " Oct 30 19:51:37.920157 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.920128 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-credential-socket\") pod \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " Oct 30 19:51:37.920269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.920225 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-data\") pod \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " Oct 30 19:51:37.920324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.920273 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-workload-socket" pod "d037606e-0d57-4527-b5a4-fa485eb976f3" (UID: "d037606e-0d57-4527-b5a4-fa485eb976f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:37.920324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.920291 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-token\" (UniqueName: \"kubernetes.io/projected/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-token\") pod \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " Oct 30 19:51:37.920421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.920382 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-credential-socket\") pod \"d037606e-0d57-4527-b5a4-fa485eb976f3\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " Oct 30 19:51:37.920474 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.920418 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-workload-certs\") pod \"d037606e-0d57-4527-b5a4-fa485eb976f3\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " Oct 30 19:51:37.920474 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.920431 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-workload-socket" pod "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" (UID: "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:37.920584 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.920481 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-envoy\") pod \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\" (UID: \"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f\") " Oct 30 19:51:37.920584 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.920559 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-data" pod "d037606e-0d57-4527-b5a4-fa485eb976f3" (UID: "d037606e-0d57-4527-b5a4-fa485eb976f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:37.920785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.920677 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-data" pod "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" (UID: "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:37.921016 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.920994 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istiod-ca-cert" pod "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" (UID: "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:37.921483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.921234 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-credential-socket" pod "d037606e-0d57-4527-b5a4-fa485eb976f3" (UID: "d037606e-0d57-4527-b5a4-fa485eb976f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:37.921483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.921254 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:37.921483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.921280 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdnqb\" (UniqueName: \"kubernetes.io/projected/fc0f7a23-6124-419b-9b33-11c6ac5f4276-kube-api-access-pdnqb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:37.921483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.921297 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0f7a23-6124-419b-9b33-11c6ac5f4276-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:37.921483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.921313 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:37.921483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.921328 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:37.921483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.921343 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:37.921483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.921357 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:37.921483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.921379 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d037606e-0d57-4527-b5a4-fa485eb976f3-istiod-ca-cert" pod "d037606e-0d57-4527-b5a4-fa485eb976f3" (UID: "d037606e-0d57-4527-b5a4-fa485eb976f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:37.922098 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.921583 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-workload-certs" pod "d037606e-0d57-4527-b5a4-fa485eb976f3" (UID: "d037606e-0d57-4527-b5a4-fa485eb976f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:37.922098 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.921804 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-credential-socket" pod "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" (UID: "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:37.922098 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.922066 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-workload-certs" pod "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" (UID: "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:37.922252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.922101 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-podinfo" pod "d037606e-0d57-4527-b5a4-fa485eb976f3" (UID: "d037606e-0d57-4527-b5a4-fa485eb976f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:51:37.922444 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.922421 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-token" pod "d037606e-0d57-4527-b5a4-fa485eb976f3" (UID: "d037606e-0d57-4527-b5a4-fa485eb976f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:37.922598 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.922579 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-kube-api-access-wlqw8" pod "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" (UID: "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f"). InnerVolumeSpecName "kube-api-access-wlqw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:37.922731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.922715 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-podinfo" pod "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" (UID: "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:51:37.923526 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.923502 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-envoy" pod "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" (UID: "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:37.924113 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.924095 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-token" pod "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" (UID: "46e2c9c5-557b-4b1e-baaf-f0f4a762f78f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:37.924457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:37.924438 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d037606e-0d57-4527-b5a4-fa485eb976f3-kube-api-access-wghc8" pod "d037606e-0d57-4527-b5a4-fa485eb976f3" (UID: "d037606e-0d57-4527-b5a4-fa485eb976f3"). InnerVolumeSpecName "kube-api-access-wghc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:38.022227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.022193 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-envoy\") pod \"d037606e-0d57-4527-b5a4-fa485eb976f3\" (UID: \"d037606e-0d57-4527-b5a4-fa485eb976f3\") " Oct 30 19:51:38.022703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.022680 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:38.022854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.022838 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:38.022967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.022954 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:38.023053 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.023041 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:38.023187 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.023170 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:38.023284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.023271 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:38.023372 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.023358 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlqw8\" (UniqueName: \"kubernetes.io/projected/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-kube-api-access-wlqw8\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:38.023457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.023443 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:38.023550 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.023538 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wghc8\" (UniqueName: \"kubernetes.io/projected/d037606e-0d57-4527-b5a4-fa485eb976f3-kube-api-access-wghc8\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:38.023626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.023616 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:38.023712 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.023700 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d037606e-0d57-4527-b5a4-fa485eb976f3-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:38.023801 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.023789 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:38.024760 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.024730 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-envoy" pod "d037606e-0d57-4527-b5a4-fa485eb976f3" (UID: "d037606e-0d57-4527-b5a4-fa485eb976f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:38.125255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.125182 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d037606e-0d57-4527-b5a4-fa485eb976f3-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:38.443216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.433489 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" podUID="fc0f7a23-6124-419b-9b33-11c6ac5f4276" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.133:50052\" within 1s: context deadline exceeded" Oct 30 19:51:38.643762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.639087 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" event={"ID":"fc0f7a23-6124-419b-9b33-11c6ac5f4276","Type":"ContainerDied","Data":"4086d0e1fb888a3fc106a1cea3921a831e8d8d7461f306ad439e0d286ee702a9"} Oct 30 19:51:38.643762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.639184 5091 scope.go:122] "RemoveContainer" containerID="a552591f13f6d618212700cfe53b0bba297adcacef6fa185c09b89416bfd14a8" Oct 30 19:51:38.643762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.639340 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l" Oct 30 19:51:38.647658 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.647632 5091 generic.go:356] "Generic (PLEG): container finished" podID="d037606e-0d57-4527-b5a4-fa485eb976f3" containerID="1db9a07cdd1181dea78fdc6eacbf6f9ac462c88e41a90fe3fe72f3b5faa8ef62" exitCode=0 Oct 30 19:51:38.647843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.647823 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" event={"ID":"d037606e-0d57-4527-b5a4-fa485eb976f3","Type":"ContainerDied","Data":"1db9a07cdd1181dea78fdc6eacbf6f9ac462c88e41a90fe3fe72f3b5faa8ef62"} Oct 30 19:51:38.647925 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.647877 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" event={"ID":"d037606e-0d57-4527-b5a4-fa485eb976f3","Type":"ContainerDied","Data":"56b7a0fddd44b2c48871eee8debf67225a6deaf60c2b3277869d9d89eb7aa16d"} Oct 30 19:51:38.647987 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.647955 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt" Oct 30 19:51:38.651330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.650592 5091 generic.go:356] "Generic (PLEG): container finished" podID="46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" containerID="43be83664ad5a73529879f057a787f6bc22dc5c75ef77db6f586fe2da0e80576" exitCode=0 Oct 30 19:51:38.651330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.650683 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" event={"ID":"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f","Type":"ContainerDied","Data":"43be83664ad5a73529879f057a787f6bc22dc5c75ef77db6f586fe2da0e80576"} Oct 30 19:51:38.651330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.650719 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" event={"ID":"46e2c9c5-557b-4b1e-baaf-f0f4a762f78f","Type":"ContainerDied","Data":"362cf0027136b7cf48edb668a9a3e31a1d6ce89dc05df031e00ec53852e061cb"} Oct 30 19:51:38.651330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.650791 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k" Oct 30 19:51:38.663900 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.663877 5091 scope.go:122] "RemoveContainer" containerID="1db9a07cdd1181dea78fdc6eacbf6f9ac462c88e41a90fe3fe72f3b5faa8ef62" Oct 30 19:51:38.677743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.677711 5091 scope.go:122] "RemoveContainer" containerID="1db9a07cdd1181dea78fdc6eacbf6f9ac462c88e41a90fe3fe72f3b5faa8ef62" Oct 30 19:51:38.678077 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:51:38.678055 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db9a07cdd1181dea78fdc6eacbf6f9ac462c88e41a90fe3fe72f3b5faa8ef62\": container with ID starting with 1db9a07cdd1181dea78fdc6eacbf6f9ac462c88e41a90fe3fe72f3b5faa8ef62 not found: ID does not exist" containerID="1db9a07cdd1181dea78fdc6eacbf6f9ac462c88e41a90fe3fe72f3b5faa8ef62" Oct 30 19:51:38.678170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.678088 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db9a07cdd1181dea78fdc6eacbf6f9ac462c88e41a90fe3fe72f3b5faa8ef62"} err="failed to get container status \"1db9a07cdd1181dea78fdc6eacbf6f9ac462c88e41a90fe3fe72f3b5faa8ef62\": rpc error: code = NotFound desc = could not find container \"1db9a07cdd1181dea78fdc6eacbf6f9ac462c88e41a90fe3fe72f3b5faa8ef62\": container with ID starting with 1db9a07cdd1181dea78fdc6eacbf6f9ac462c88e41a90fe3fe72f3b5faa8ef62 not found: ID does not exist" Oct 30 19:51:38.678170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.678111 5091 scope.go:122] "RemoveContainer" containerID="43be83664ad5a73529879f057a787f6bc22dc5c75ef77db6f586fe2da0e80576" Oct 30 19:51:38.686403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.686377 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k"] Oct 30 19:51:38.694299 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.694281 5091 scope.go:122] "RemoveContainer" containerID="43be83664ad5a73529879f057a787f6bc22dc5c75ef77db6f586fe2da0e80576" Oct 30 19:51:38.694929 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:51:38.694901 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43be83664ad5a73529879f057a787f6bc22dc5c75ef77db6f586fe2da0e80576\": container with ID starting with 43be83664ad5a73529879f057a787f6bc22dc5c75ef77db6f586fe2da0e80576 not found: ID does not exist" containerID="43be83664ad5a73529879f057a787f6bc22dc5c75ef77db6f586fe2da0e80576" Oct 30 19:51:38.695027 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.694937 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43be83664ad5a73529879f057a787f6bc22dc5c75ef77db6f586fe2da0e80576"} err="failed to get container status \"43be83664ad5a73529879f057a787f6bc22dc5c75ef77db6f586fe2da0e80576\": rpc error: code = NotFound desc = could not find container \"43be83664ad5a73529879f057a787f6bc22dc5c75ef77db6f586fe2da0e80576\": container with ID starting with 43be83664ad5a73529879f057a787f6bc22dc5c75ef77db6f586fe2da0e80576 not found: ID does not exist" Oct 30 19:51:38.700805 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.700782 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-mg64k"] Oct 30 19:51:38.727718 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.727687 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l"] Oct 30 19:51:38.741565 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.741539 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2beb7b-5bbfdfc944-hmv9l"] Oct 30 19:51:38.759799 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.759776 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt"] Oct 30 19:51:38.768173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.768132 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-qdtdt"] Oct 30 19:51:38.812182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:38.800482 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:40.419319 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.419161 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:40.456325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.456293 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" path="/var/lib/kubelet/pods/46e2c9c5-557b-4b1e-baaf-f0f4a762f78f/volumes" Oct 30 19:51:40.456952 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.456927 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d037606e-0d57-4527-b5a4-fa485eb976f3" path="/var/lib/kubelet/pods/d037606e-0d57-4527-b5a4-fa485eb976f3/volumes" Oct 30 19:51:40.457376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.457362 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc0f7a23-6124-419b-9b33-11c6ac5f4276" path="/var/lib/kubelet/pods/fc0f7a23-6124-419b-9b33-11c6ac5f4276/volumes" Oct 30 19:51:40.502334 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.502287 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" podUID="03fb817b-087c-4744-8f3d-fc12e8444385" containerName="payload-pre-processing" containerID="cri-o://0ddcfe9479fcfb2ae9ba312ae2752896fbb0b7aea70b74f37cc729044fe6c20d" gracePeriod=25 Oct 30 19:51:40.682333 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.682304 5091 generic.go:356] "Generic (PLEG): container finished" podID="03fb817b-087c-4744-8f3d-fc12e8444385" containerID="0ddcfe9479fcfb2ae9ba312ae2752896fbb0b7aea70b74f37cc729044fe6c20d" exitCode=0 Oct 30 19:51:40.682529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.682392 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" event={"ID":"03fb817b-087c-4744-8f3d-fc12e8444385","Type":"ContainerDied","Data":"0ddcfe9479fcfb2ae9ba312ae2752896fbb0b7aea70b74f37cc729044fe6c20d"} Oct 30 19:51:40.685518 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.685487 5091 generic.go:356] "Generic (PLEG): container finished" podID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" containerID="fd94dbb84e8ddf338e641fe05490cfe8c052f8b7e98e29073513c6bdf7c361b0" exitCode=0 Oct 30 19:51:40.685658 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.685536 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" event={"ID":"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b","Type":"ContainerDied","Data":"fd94dbb84e8ddf338e641fe05490cfe8c052f8b7e98e29073513c6bdf7c361b0"} Oct 30 19:51:40.686166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.686043 5091 scope.go:122] "RemoveContainer" containerID="fd94dbb84e8ddf338e641fe05490cfe8c052f8b7e98e29073513c6bdf7c361b0" Oct 30 19:51:40.689026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.688988 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:51:40.689114 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.689085 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:51:40.689202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.689135 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:51:40.822731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.822690 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl"] Oct 30 19:51:40.823693 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.823671 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" containerName="istio-proxy" Oct 30 19:51:40.823801 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.823792 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" containerName="istio-proxy" Oct 30 19:51:40.823940 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.823930 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fc0f7a23-6124-419b-9b33-11c6ac5f4276" containerName="payload-processing" Oct 30 19:51:40.823993 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.823986 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0f7a23-6124-419b-9b33-11c6ac5f4276" containerName="payload-processing" Oct 30 19:51:40.824049 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.824042 5091 cpu_manager.go:414] "containerMap: removing container" podUID="036310d3-eb3a-410b-8716-886f139c818c" containerName="istio-proxy" Oct 30 19:51:40.824091 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.824085 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="036310d3-eb3a-410b-8716-886f139c818c" containerName="istio-proxy" Oct 30 19:51:40.824180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.824169 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d037606e-0d57-4527-b5a4-fa485eb976f3" containerName="istio-proxy" Oct 30 19:51:40.824258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.824197 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d037606e-0d57-4527-b5a4-fa485eb976f3" containerName="istio-proxy" Oct 30 19:51:40.824258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.824224 5091 cpu_manager.go:414] "containerMap: removing container" podUID="553c0e2c-3812-4e18-b527-a5f74d006e27" containerName="revoke-keys" Oct 30 19:51:40.824258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.824231 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="553c0e2c-3812-4e18-b527-a5f74d006e27" containerName="revoke-keys" Oct 30 19:51:40.824258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.824254 5091 cpu_manager.go:414] "containerMap: removing container" podUID="46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" containerName="istio-proxy" Oct 30 19:51:40.824450 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.824263 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" containerName="istio-proxy" Oct 30 19:51:40.824511 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.824437 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e5b9c8e-97eb-4c35-8032-92bb7acb8d58" containerName="istio-proxy" Oct 30 19:51:40.824511 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.824474 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0f7a23-6124-419b-9b33-11c6ac5f4276" containerName="payload-processing" Oct 30 19:51:40.824511 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.824487 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="036310d3-eb3a-410b-8716-886f139c818c" containerName="istio-proxy" Oct 30 19:51:40.824511 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.824500 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d037606e-0d57-4527-b5a4-fa485eb976f3" containerName="istio-proxy" Oct 30 19:51:40.824712 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.824512 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="553c0e2c-3812-4e18-b527-a5f74d006e27" containerName="revoke-keys" Oct 30 19:51:40.824712 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.824529 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="46e2c9c5-557b-4b1e-baaf-f0f4a762f78f" containerName="istio-proxy" Oct 30 19:51:40.833438 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.833412 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:40.836951 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.836930 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" Oct 30 19:51:40.877091 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.877021 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl"] Oct 30 19:51:40.979168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.978814 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03fb817b-087c-4744-8f3d-fc12e8444385-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03fb817b-087c-4744-8f3d-fc12e8444385-plugins-config-volume\") pod \"03fb817b-087c-4744-8f3d-fc12e8444385\" (UID: \"03fb817b-087c-4744-8f3d-fc12e8444385\") " Oct 30 19:51:40.979168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.978904 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03fb817b-087c-4744-8f3d-fc12e8444385-kube-api-access-r6w4m\" (UniqueName: \"kubernetes.io/projected/03fb817b-087c-4744-8f3d-fc12e8444385-kube-api-access-r6w4m\") pod \"03fb817b-087c-4744-8f3d-fc12e8444385\" (UID: \"03fb817b-087c-4744-8f3d-fc12e8444385\") " Oct 30 19:51:40.979412 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.979252 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-data\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:40.979412 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.979300 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03fb817b-087c-4744-8f3d-fc12e8444385-plugins-config-volume" pod "03fb817b-087c-4744-8f3d-fc12e8444385" (UID: "03fb817b-087c-4744-8f3d-fc12e8444385"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:40.979412 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.979317 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-workload-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:40.979576 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.979467 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-credential-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:40.979576 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.979527 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-workload-certs\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:40.980083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.979773 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q22tg\" (UniqueName: \"kubernetes.io/projected/192dd6ce-3921-41dc-ba51-be3b5535cb51-kube-api-access-q22tg\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:40.980083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.979824 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192dd6ce-3921-41dc-ba51-be3b5535cb51-istiod-ca-cert\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:40.980083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.979854 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-envoy\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:40.980083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.979878 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-podinfo\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:40.980083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.979903 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-token\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:40.980083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.979991 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03fb817b-087c-4744-8f3d-fc12e8444385-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:40.990783 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:40.990763 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03fb817b-087c-4744-8f3d-fc12e8444385-kube-api-access-r6w4m" pod "03fb817b-087c-4744-8f3d-fc12e8444385" (UID: "03fb817b-087c-4744-8f3d-fc12e8444385"). InnerVolumeSpecName "kube-api-access-r6w4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:41.081798 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.081756 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-data\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.081999 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.081841 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-workload-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.081999 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.081965 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-credential-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.082117 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.082024 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-workload-certs\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.082254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.082217 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q22tg\" (UniqueName: \"kubernetes.io/projected/192dd6ce-3921-41dc-ba51-be3b5535cb51-kube-api-access-q22tg\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.082254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.082246 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-data\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.082455 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.082265 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192dd6ce-3921-41dc-ba51-be3b5535cb51-istiod-ca-cert\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.082455 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.082324 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-workload-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.082455 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.082329 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-envoy\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.082455 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.082362 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-podinfo\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.082455 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.082404 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-token\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.083046 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.082958 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192dd6ce-3921-41dc-ba51-be3b5535cb51-istiod-ca-cert\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.083046 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.083018 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6w4m\" (UniqueName: \"kubernetes.io/projected/03fb817b-087c-4744-8f3d-fc12e8444385-kube-api-access-r6w4m\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:41.083302 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.083278 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-workload-certs\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.083595 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.083570 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-credential-socket\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.085219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.085194 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-podinfo\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.085351 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.085333 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-envoy\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.096214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.096141 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q22tg\" (UniqueName: \"kubernetes.io/projected/192dd6ce-3921-41dc-ba51-be3b5535cb51-kube-api-access-q22tg\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.102529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.102502 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-token\") pod \"e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.192250 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.192161 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:41.520309 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.520275 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl"] Oct 30 19:51:41.527106 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.527068 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:51:41.527279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.527246 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:51:41.530246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.527348 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:51:41.706037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.701718 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" event={"ID":"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b","Type":"ContainerStarted","Data":"c0b37806ac0863fbf43a3ac740d75b21e3d206f6f0614f1b3cee6a5a1ace469d"} Oct 30 19:51:41.706037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.704706 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" event={"ID":"03fb817b-087c-4744-8f3d-fc12e8444385","Type":"ContainerDied","Data":"53c53dc26d4c72b787fdce53049e1e39c2e38234dd98e496365990cb8429cbff"} Oct 30 19:51:41.706037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.704759 5091 scope.go:122] "RemoveContainer" containerID="0ddcfe9479fcfb2ae9ba312ae2752896fbb0b7aea70b74f37cc729044fe6c20d" Oct 30 19:51:41.706037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.704916 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6" Oct 30 19:51:41.709210 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.708645 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" event={"ID":"192dd6ce-3921-41dc-ba51-be3b5535cb51","Type":"ContainerStarted","Data":"1d9ef5ec373924f04fed2748b9ed3ce54de13b437716e306dd6d1006bfc4c510"} Oct 30 19:51:41.709210 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.708704 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" event={"ID":"192dd6ce-3921-41dc-ba51-be3b5535cb51","Type":"ContainerStarted","Data":"eeb5d1669ff836a427b78ee3e05bd96ed5cd5ce0dcc7a5cec8b474fcf87eaa02"} Oct 30 19:51:41.852305 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.848942 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:41.918170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.915582 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" podStartSLOduration=1.915558302 podStartE2EDuration="1.915558302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:41.844546914 +0000 UTC m=+1221.539754873" watchObservedRunningTime="2026-10-30 19:51:41.915558302 +0000 UTC m=+1221.610766286" Oct 30 19:51:41.943180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.940242 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6"] Oct 30 19:51:42.014843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:41.982676 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-695bcb6dd7-cvnn6"] Oct 30 19:51:42.073204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:42.065284 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:51:42.103560 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:42.101648 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:51:42.103560 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:42.101715 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" podUID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 19:51:42.103560 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:42.102849 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:42.192468 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:42.192390 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:42.194654 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:42.194623 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:51:42.194826 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:42.194677 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" podUID="192dd6ce-3921-41dc-ba51-be3b5535cb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 19:51:42.377228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:42.360593 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t"] Oct 30 19:51:42.377228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:42.360892 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" containerID="cri-o://3a0da0bf34e7cf2a671266fe0ee9d48905a98a9a75cac2795be5c94fbf332a6f" gracePeriod=30 Oct 30 19:51:42.524543 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:42.522607 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:42.619181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:42.602318 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03fb817b-087c-4744-8f3d-fc12e8444385" path="/var/lib/kubelet/pods/03fb817b-087c-4744-8f3d-fc12e8444385/volumes" Oct 30 19:51:43.068502 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:43.068179 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:51:43.068502 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:43.068235 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" podUID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 19:51:43.092170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:43.070217 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:43.203592 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:43.194889 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:51:43.203592 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:43.194949 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" podUID="192dd6ce-3921-41dc-ba51-be3b5535cb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 19:51:43.203592 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:43.203503 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:51:44.062222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.062183 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:51:44.062821 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.062790 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" podUID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 19:51:44.063294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.063262 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:44.197601 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.197559 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:51:44.197894 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.197864 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" podUID="192dd6ce-3921-41dc-ba51-be3b5535cb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 19:51:44.272504 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.268373 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:44.619448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.614831 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l"] Oct 30 19:51:44.619448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.615823 5091 cpu_manager.go:414] "containerMap: removing container" podUID="03fb817b-087c-4744-8f3d-fc12e8444385" containerName="payload-pre-processing" Oct 30 19:51:44.619448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.615843 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03fb817b-087c-4744-8f3d-fc12e8444385" containerName="payload-pre-processing" Oct 30 19:51:44.619448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.616082 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="03fb817b-087c-4744-8f3d-fc12e8444385" containerName="payload-pre-processing" Oct 30 19:51:44.623183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.622849 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.725359 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.683706 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-b17e94-openshift-default-dockercfg-lz6js\"" Oct 30 19:51:44.725359 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.684507 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l"] Oct 30 19:51:44.778172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.775107 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:44.809169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.808203 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-workload-certs\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.809169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.808305 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-podinfo\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.809169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.808443 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-workload-socket\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.809169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.808541 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-token\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.809169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.808577 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-data\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.809169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.808650 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5fph\" (UniqueName: \"kubernetes.io/projected/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-kube-api-access-x5fph\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.809169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.808699 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-credential-socket\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.809169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.808754 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-envoy\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.809169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.808789 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istiod-ca-cert\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.934743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.931462 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-token\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.950349 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.942267 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-data\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.960321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.960285 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5fph\" (UniqueName: \"kubernetes.io/projected/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-kube-api-access-x5fph\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.968175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.964229 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-credential-socket\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.981444 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.981393 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-envoy\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.981689 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.981671 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istiod-ca-cert\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:44.987320 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.981953 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-workload-certs\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:45.012168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.987568 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-podinfo\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:45.025332 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.014345 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-workload-socket\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:45.030652 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.952365 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-data\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:45.063313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.011841 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istiod-ca-cert\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:45.063313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.013657 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-workload-certs\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:45.063313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:44.977479 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-credential-socket\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:45.063313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.026315 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-workload-socket\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:45.063313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.042493 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-envoy\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:45.063313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.043059 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5fph\" (UniqueName: \"kubernetes.io/projected/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-kube-api-access-x5fph\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:45.196269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.184844 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:45.217171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.201240 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:51:45.217171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.201300 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" podUID="192dd6ce-3921-41dc-ba51-be3b5535cb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 19:51:45.217171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.215390 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-token\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:45.217171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.216571 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-podinfo\") pod \"e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:45.218707 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.218354 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:51:45.218707 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.218426 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" podUID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 19:51:45.379354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.379043 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:45.529062 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.528648 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:45.762926 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:45.761849 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:46.085530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.004823 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:46.085530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.075361 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:51:46.085530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.075425 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" podUID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 19:51:46.139415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.087295 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49"] Oct 30 19:51:46.139415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.087671 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerName="istio-proxy" containerID="cri-o://c4c1f5093aeb7f6937e21519ac309e0620da01bad97fe0fd393ff0a5f0c61dca" gracePeriod=30 Oct 30 19:51:46.285355 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.284052 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l"] Oct 30 19:51:46.285355 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.284301 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:51:46.285355 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.284366 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" podUID="192dd6ce-3921-41dc-ba51-be3b5535cb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 19:51:46.314410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.309812 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:51:46.314410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.309896 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:51:46.314410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.309940 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:51:46.432035 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.424031 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd"] Oct 30 19:51:46.438342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.438303 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd"] Oct 30 19:51:46.443079 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.443057 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" Oct 30 19:51:46.444930 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:51:46.444895 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:51:46.483574 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.464897 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:51:46.483574 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.469007 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:51:46.516662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.509441 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92f3caf-4ae7-431a-928a-7812319a3f38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd\" (UID: \"b92f3caf-4ae7-431a-928a-7812319a3f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" Oct 30 19:51:46.516662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.509501 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92f3caf-4ae7-431a-928a-7812319a3f38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd\" (UID: \"b92f3caf-4ae7-431a-928a-7812319a3f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" Oct 30 19:51:46.516662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.509695 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qc5gn\" (UniqueName: \"kubernetes.io/projected/b92f3caf-4ae7-431a-928a-7812319a3f38-kube-api-access-qc5gn\") pod \"maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd\" (UID: \"b92f3caf-4ae7-431a-928a-7812319a3f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" Oct 30 19:51:46.612049 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.611795 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qc5gn\" (UniqueName: \"kubernetes.io/projected/b92f3caf-4ae7-431a-928a-7812319a3f38-kube-api-access-qc5gn\") pod \"maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd\" (UID: \"b92f3caf-4ae7-431a-928a-7812319a3f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" Oct 30 19:51:46.612436 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.612167 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92f3caf-4ae7-431a-928a-7812319a3f38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd\" (UID: \"b92f3caf-4ae7-431a-928a-7812319a3f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" Oct 30 19:51:46.612436 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.612233 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92f3caf-4ae7-431a-928a-7812319a3f38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd\" (UID: \"b92f3caf-4ae7-431a-928a-7812319a3f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" Oct 30 19:51:46.613329 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.613304 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92f3caf-4ae7-431a-928a-7812319a3f38-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd\" (UID: \"b92f3caf-4ae7-431a-928a-7812319a3f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" Oct 30 19:51:46.616570 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.616546 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92f3caf-4ae7-431a-928a-7812319a3f38-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd\" (UID: \"b92f3caf-4ae7-431a-928a-7812319a3f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" Oct 30 19:51:46.626672 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.626645 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qc5gn\" (UniqueName: \"kubernetes.io/projected/b92f3caf-4ae7-431a-928a-7812319a3f38-kube-api-access-qc5gn\") pod \"maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd\" (UID: \"b92f3caf-4ae7-431a-928a-7812319a3f38\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" Oct 30 19:51:46.829078 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:46.829049 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" Oct 30 19:51:47.076204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:47.068251 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Oct 30 19:51:47.076204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:47.068332 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" podUID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Oct 30 19:51:47.217885 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:47.217763 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:51:47.240948 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:47.240852 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" podUID="192dd6ce-3921-41dc-ba51-be3b5535cb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 19:51:47.295821 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:47.295783 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:47.365215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:47.365182 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd"] Oct 30 19:51:47.373030 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:47.372991 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" event={"ID":"8b12e993-577a-4dc7-94a0-41c65c7b1fe3","Type":"ContainerStarted","Data":"c213ab747484af8ee780f4f5f1dc2e460f261e47c573f32380995d70f42cfd30"} Oct 30 19:51:47.373255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:47.373233 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" event={"ID":"8b12e993-577a-4dc7-94a0-41c65c7b1fe3","Type":"ContainerStarted","Data":"9cc3acb510662984ac1697d45747a8f75b88418e579f0053d37f0b5d5a92d877"} Oct 30 19:51:47.409690 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:47.409617 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:47.424183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:47.418765 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:47.444197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:47.443282 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" podStartSLOduration=3.443267681 podStartE2EDuration="3.443267681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:47.440835309 +0000 UTC m=+1227.136043269" watchObservedRunningTime="2026-10-30 19:51:47.443267681 +0000 UTC m=+1227.138475704" Oct 30 19:51:48.137725 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.137331 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:48.144894 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.144467 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:51:48.205175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.201348 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:51:48.205175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.201409 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" podUID="192dd6ce-3921-41dc-ba51-be3b5535cb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 19:51:48.205175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.202603 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:51:48.296376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.296348 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff"] Oct 30 19:51:48.302512 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.302491 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" Oct 30 19:51:48.329452 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.326438 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ec5e0f-dockercfg-rn2h5\"" Oct 30 19:51:48.329452 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.327129 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ec5e0f\"" Oct 30 19:51:48.331518 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.331434 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff"] Oct 30 19:51:48.334052 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.334026 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:51:48.341745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.341720 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-workload-socket\") pod \"54f332bc-3668-4f09-a354-697e1610cd3d\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " Oct 30 19:51:48.341863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.341759 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-istio-data\") pod \"54f332bc-3668-4f09-a354-697e1610cd3d\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " Oct 30 19:51:48.341863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.341834 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f332bc-3668-4f09-a354-697e1610cd3d-kube-api-access-6qk9m\" (UniqueName: \"kubernetes.io/projected/54f332bc-3668-4f09-a354-697e1610cd3d-kube-api-access-6qk9m\") pod \"54f332bc-3668-4f09-a354-697e1610cd3d\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " Oct 30 19:51:48.341863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.341850 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-workload-certs\") pod \"54f332bc-3668-4f09-a354-697e1610cd3d\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " Oct 30 19:51:48.342017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.341905 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54f332bc-3668-4f09-a354-697e1610cd3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54f332bc-3668-4f09-a354-697e1610cd3d-istiod-ca-cert\") pod \"54f332bc-3668-4f09-a354-697e1610cd3d\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " Oct 30 19:51:48.342017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.341938 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54f332bc-3668-4f09-a354-697e1610cd3d-istio-token\" (UniqueName: \"kubernetes.io/projected/54f332bc-3668-4f09-a354-697e1610cd3d-istio-token\") pod \"54f332bc-3668-4f09-a354-697e1610cd3d\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " Oct 30 19:51:48.342017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.341965 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-credential-socket\") pod \"54f332bc-3668-4f09-a354-697e1610cd3d\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " Oct 30 19:51:48.342017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.341991 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-istio-envoy\") pod \"54f332bc-3668-4f09-a354-697e1610cd3d\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " Oct 30 19:51:48.342401 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.342375 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54f332bc-3668-4f09-a354-697e1610cd3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54f332bc-3668-4f09-a354-697e1610cd3d-istio-podinfo\") pod \"54f332bc-3668-4f09-a354-697e1610cd3d\" (UID: \"54f332bc-3668-4f09-a354-697e1610cd3d\") " Oct 30 19:51:48.343283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.342568 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-workload-certs" pod "54f332bc-3668-4f09-a354-697e1610cd3d" (UID: "54f332bc-3668-4f09-a354-697e1610cd3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:48.343283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.342785 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433ca046-a713-4f04-a152-027d1bdeee27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff\" (UID: \"433ca046-a713-4f04-a152-027d1bdeee27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" Oct 30 19:51:48.343283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.342863 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54f332bc-3668-4f09-a354-697e1610cd3d-istiod-ca-cert" pod "54f332bc-3668-4f09-a354-697e1610cd3d" (UID: "54f332bc-3668-4f09-a354-697e1610cd3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:48.343283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.342915 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvhgr\" (UniqueName: \"kubernetes.io/projected/433ca046-a713-4f04-a152-027d1bdeee27-kube-api-access-pvhgr\") pod \"payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff\" (UID: \"433ca046-a713-4f04-a152-027d1bdeee27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" Oct 30 19:51:48.343283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.343086 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:48.343283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.343098 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54f332bc-3668-4f09-a354-697e1610cd3d-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:48.343283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.343198 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-istio-data" pod "54f332bc-3668-4f09-a354-697e1610cd3d" (UID: "54f332bc-3668-4f09-a354-697e1610cd3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:48.343283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.343259 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-credential-socket" pod "54f332bc-3668-4f09-a354-697e1610cd3d" (UID: "54f332bc-3668-4f09-a354-697e1610cd3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:48.353618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.353592 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-workload-socket" pod "54f332bc-3668-4f09-a354-697e1610cd3d" (UID: "54f332bc-3668-4f09-a354-697e1610cd3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:48.365373 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.365343 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f332bc-3668-4f09-a354-697e1610cd3d-istio-token" pod "54f332bc-3668-4f09-a354-697e1610cd3d" (UID: "54f332bc-3668-4f09-a354-697e1610cd3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:48.386196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.384289 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54f332bc-3668-4f09-a354-697e1610cd3d-istio-podinfo" pod "54f332bc-3668-4f09-a354-697e1610cd3d" (UID: "54f332bc-3668-4f09-a354-697e1610cd3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:51:48.397263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.388062 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-istio-envoy" pod "54f332bc-3668-4f09-a354-697e1610cd3d" (UID: "54f332bc-3668-4f09-a354-697e1610cd3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:48.404358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.404324 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:48.407426 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.407399 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54f332bc-3668-4f09-a354-697e1610cd3d-kube-api-access-6qk9m" pod "54f332bc-3668-4f09-a354-697e1610cd3d" (UID: "54f332bc-3668-4f09-a354-697e1610cd3d"). InnerVolumeSpecName "kube-api-access-6qk9m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:48.412219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.412133 5091 generic.go:356] "Generic (PLEG): container finished" podID="54f332bc-3668-4f09-a354-697e1610cd3d" containerID="3a0da0bf34e7cf2a671266fe0ee9d48905a98a9a75cac2795be5c94fbf332a6f" exitCode=0 Oct 30 19:51:48.412323 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.412308 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" event={"ID":"54f332bc-3668-4f09-a354-697e1610cd3d","Type":"ContainerDied","Data":"3a0da0bf34e7cf2a671266fe0ee9d48905a98a9a75cac2795be5c94fbf332a6f"} Oct 30 19:51:48.412396 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.412360 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" event={"ID":"54f332bc-3668-4f09-a354-697e1610cd3d","Type":"ContainerDied","Data":"e7ad7f669c1601e6647ad82b365d4f2df50b522969f2e6c3ed7103a51ed106d2"} Oct 30 19:51:48.412787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.412391 5091 scope.go:122] "RemoveContainer" containerID="3a0da0bf34e7cf2a671266fe0ee9d48905a98a9a75cac2795be5c94fbf332a6f" Oct 30 19:51:48.412787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.412571 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t" Oct 30 19:51:48.417803 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.417772 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" event={"ID":"b92f3caf-4ae7-431a-928a-7812319a3f38","Type":"ContainerStarted","Data":"34fefb684f739b0da27fee7f02cb8ae87a240ddc08274806ea7fb1f2c217b445"} Oct 30 19:51:48.417918 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.417828 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" event={"ID":"b92f3caf-4ae7-431a-928a-7812319a3f38","Type":"ContainerStarted","Data":"3d06bb5e07928d42b41e2ed2ab7f620c997b9e645e2a697f5d16ad31817fbbfb"} Oct 30 19:51:48.419544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.418936 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:48.420129 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.420110 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:51:48.436990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.436258 5091 scope.go:122] "RemoveContainer" containerID="1a8ce22bc4efc48cdfbbde310e448600f6ed47f295b68d59fb5fbf7295f5bb3d" Oct 30 19:51:48.467219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.444902 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433ca046-a713-4f04-a152-027d1bdeee27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff\" (UID: \"433ca046-a713-4f04-a152-027d1bdeee27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" Oct 30 19:51:48.467219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.445296 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvhgr\" (UniqueName: \"kubernetes.io/projected/433ca046-a713-4f04-a152-027d1bdeee27-kube-api-access-pvhgr\") pod \"payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff\" (UID: \"433ca046-a713-4f04-a152-027d1bdeee27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" Oct 30 19:51:48.467219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.445950 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:48.467219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.451301 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433ca046-a713-4f04-a152-027d1bdeee27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff\" (UID: \"433ca046-a713-4f04-a152-027d1bdeee27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" Oct 30 19:51:48.467219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.451589 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:48.467219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.451613 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qk9m\" (UniqueName: \"kubernetes.io/projected/54f332bc-3668-4f09-a354-697e1610cd3d-kube-api-access-6qk9m\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:48.467219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.451630 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54f332bc-3668-4f09-a354-697e1610cd3d-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:48.467219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.451645 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:48.467219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.451659 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54f332bc-3668-4f09-a354-697e1610cd3d-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:48.467219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.451674 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54f332bc-3668-4f09-a354-697e1610cd3d-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:48.586597 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.581994 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvhgr\" (UniqueName: \"kubernetes.io/projected/433ca046-a713-4f04-a152-027d1bdeee27-kube-api-access-pvhgr\") pod \"payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff\" (UID: \"433ca046-a713-4f04-a152-027d1bdeee27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" Oct 30 19:51:48.614171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.590500 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" podStartSLOduration=2.59048043 podStartE2EDuration="2.59048043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:48.481859191 +0000 UTC m=+1228.177067142" watchObservedRunningTime="2026-10-30 19:51:48.59048043 +0000 UTC m=+1228.285688390" Oct 30 19:51:48.722692 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.722659 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" Oct 30 19:51:48.728161 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.728119 5091 scope.go:122] "RemoveContainer" containerID="3a0da0bf34e7cf2a671266fe0ee9d48905a98a9a75cac2795be5c94fbf332a6f" Oct 30 19:51:48.757231 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:51:48.740290 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0da0bf34e7cf2a671266fe0ee9d48905a98a9a75cac2795be5c94fbf332a6f\": container with ID starting with 3a0da0bf34e7cf2a671266fe0ee9d48905a98a9a75cac2795be5c94fbf332a6f not found: ID does not exist" containerID="3a0da0bf34e7cf2a671266fe0ee9d48905a98a9a75cac2795be5c94fbf332a6f" Oct 30 19:51:48.757231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.740342 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0da0bf34e7cf2a671266fe0ee9d48905a98a9a75cac2795be5c94fbf332a6f"} err="failed to get container status \"3a0da0bf34e7cf2a671266fe0ee9d48905a98a9a75cac2795be5c94fbf332a6f\": rpc error: code = NotFound desc = could not find container \"3a0da0bf34e7cf2a671266fe0ee9d48905a98a9a75cac2795be5c94fbf332a6f\": container with ID starting with 3a0da0bf34e7cf2a671266fe0ee9d48905a98a9a75cac2795be5c94fbf332a6f not found: ID does not exist" Oct 30 19:51:48.757231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.740369 5091 scope.go:122] "RemoveContainer" containerID="1a8ce22bc4efc48cdfbbde310e448600f6ed47f295b68d59fb5fbf7295f5bb3d" Oct 30 19:51:48.757231 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:51:48.745917 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a8ce22bc4efc48cdfbbde310e448600f6ed47f295b68d59fb5fbf7295f5bb3d\": container with ID starting with 1a8ce22bc4efc48cdfbbde310e448600f6ed47f295b68d59fb5fbf7295f5bb3d not found: ID does not exist" containerID="1a8ce22bc4efc48cdfbbde310e448600f6ed47f295b68d59fb5fbf7295f5bb3d" Oct 30 19:51:48.757231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.745960 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a8ce22bc4efc48cdfbbde310e448600f6ed47f295b68d59fb5fbf7295f5bb3d"} err="failed to get container status \"1a8ce22bc4efc48cdfbbde310e448600f6ed47f295b68d59fb5fbf7295f5bb3d\": rpc error: code = NotFound desc = could not find container \"1a8ce22bc4efc48cdfbbde310e448600f6ed47f295b68d59fb5fbf7295f5bb3d\": container with ID starting with 1a8ce22bc4efc48cdfbbde310e448600f6ed47f295b68d59fb5fbf7295f5bb3d not found: ID does not exist" Oct 30 19:51:48.759646 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.759616 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l"] Oct 30 19:51:48.765121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.765092 5091 cpu_manager.go:414] "containerMap: removing container" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" Oct 30 19:51:48.765321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.765304 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" Oct 30 19:51:48.765638 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.765623 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" Oct 30 19:51:48.833462 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.833429 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l"] Oct 30 19:51:48.833699 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.833687 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t"] Oct 30 19:51:48.833790 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.833782 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-040623-openshift-default-74b5f7d8d7-q8p6t"] Oct 30 19:51:48.833881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:48.833586 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" Oct 30 19:51:49.012245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.008779 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt5cj\" (UniqueName: \"kubernetes.io/projected/be3e710b-55f4-4cd6-8872-f7db69725afc-kube-api-access-tt5cj\") pod \"payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l\" (UID: \"be3e710b-55f4-4cd6-8872-f7db69725afc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" Oct 30 19:51:49.012245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.008945 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3e710b-55f4-4cd6-8872-f7db69725afc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l\" (UID: \"be3e710b-55f4-4cd6-8872-f7db69725afc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" Oct 30 19:51:49.183020 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.178440 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5cj\" (UniqueName: \"kubernetes.io/projected/be3e710b-55f4-4cd6-8872-f7db69725afc-kube-api-access-tt5cj\") pod \"payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l\" (UID: \"be3e710b-55f4-4cd6-8872-f7db69725afc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" Oct 30 19:51:49.183020 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.178548 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3e710b-55f4-4cd6-8872-f7db69725afc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l\" (UID: \"be3e710b-55f4-4cd6-8872-f7db69725afc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" Oct 30 19:51:49.183020 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.179687 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3e710b-55f4-4cd6-8872-f7db69725afc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l\" (UID: \"be3e710b-55f4-4cd6-8872-f7db69725afc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" Oct 30 19:51:49.186052 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.185612 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:49.197490 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.197405 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Oct 30 19:51:49.197490 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.197461 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" podUID="192dd6ce-3921-41dc-ba51-be3b5535cb51" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Oct 30 19:51:49.198401 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.198375 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt5cj\" (UniqueName: \"kubernetes.io/projected/be3e710b-55f4-4cd6-8872-f7db69725afc-kube-api-access-tt5cj\") pod \"payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l\" (UID: \"be3e710b-55f4-4cd6-8872-f7db69725afc\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" Oct 30 19:51:49.246792 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.246751 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" Oct 30 19:51:49.258277 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.258247 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff"] Oct 30 19:51:49.262537 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:51:49.262409 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433ca046_a713_4f04_a152_027d1bdeee27.slice/crio-89370883f2fcd8fa61aee61c10fd8f65e4f8f6e2e3f205f2775723845fea80db WatchSource:0}: Error finding container 89370883f2fcd8fa61aee61c10fd8f65e4f8f6e2e3f205f2775723845fea80db: Status 404 returned error can't find the container with id 89370883f2fcd8fa61aee61c10fd8f65e4f8f6e2e3f205f2775723845fea80db Oct 30 19:51:49.427688 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.427650 5091 generic.go:356] "Generic (PLEG): container finished" podID="b92f3caf-4ae7-431a-928a-7812319a3f38" containerID="34fefb684f739b0da27fee7f02cb8ae87a240ddc08274806ea7fb1f2c217b445" exitCode=0 Oct 30 19:51:49.428064 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.427893 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" event={"ID":"b92f3caf-4ae7-431a-928a-7812319a3f38","Type":"ContainerDied","Data":"34fefb684f739b0da27fee7f02cb8ae87a240ddc08274806ea7fb1f2c217b445"} Oct 30 19:51:49.431282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.431255 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" event={"ID":"433ca046-a713-4f04-a152-027d1bdeee27","Type":"ContainerStarted","Data":"69fa1c39bffae983c8a2940dbcaa62afe338ca72593c7b4c4e53187a38a3285b"} Oct 30 19:51:49.431399 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.431298 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" Oct 30 19:51:49.431399 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.431312 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" event={"ID":"433ca046-a713-4f04-a152-027d1bdeee27","Type":"ContainerStarted","Data":"89370883f2fcd8fa61aee61c10fd8f65e4f8f6e2e3f205f2775723845fea80db"} Oct 30 19:51:49.480762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.480561 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" podStartSLOduration=1.480542772 podStartE2EDuration="1.480542772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:49.479214002 +0000 UTC m=+1229.174421960" watchObservedRunningTime="2026-10-30 19:51:49.480542772 +0000 UTC m=+1229.175750731" Oct 30 19:51:49.564680 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:49.564640 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l"] Oct 30 19:51:50.062137 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.062036 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:50.369217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.367462 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:50.369217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.369076 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:50.378419 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.378392 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:51:50.442048 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.442017 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:50.451432 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.451407 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" path="/var/lib/kubelet/pods/54f332bc-3668-4f09-a354-697e1610cd3d/volumes" Oct 30 19:51:50.452964 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.452944 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" Oct 30 19:51:50.453161 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.453125 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" event={"ID":"be3e710b-55f4-4cd6-8872-f7db69725afc","Type":"ContainerStarted","Data":"8227ceee5ca652df67bf3a3118fa741fde4208cedeb130146cbda24054463b56"} Oct 30 19:51:50.453292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.453276 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" event={"ID":"be3e710b-55f4-4cd6-8872-f7db69725afc","Type":"ContainerStarted","Data":"09760b90891f3825911f233da6abd05cfc9dab94231c2fb5c2d79b97d02f5c23"} Oct 30 19:51:50.744892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.744855 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" Oct 30 19:51:50.775222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.772910 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" podStartSLOduration=2.772890923 podStartE2EDuration="2.772890923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:50.53850126 +0000 UTC m=+1230.233709220" watchObservedRunningTime="2026-10-30 19:51:50.772890923 +0000 UTC m=+1230.468098884" Oct 30 19:51:50.857365 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.857324 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b92f3caf-4ae7-431a-928a-7812319a3f38-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92f3caf-4ae7-431a-928a-7812319a3f38-maas-api-xks-ca\") pod \"b92f3caf-4ae7-431a-928a-7812319a3f38\" (UID: \"b92f3caf-4ae7-431a-928a-7812319a3f38\") " Oct 30 19:51:50.857560 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.857375 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b92f3caf-4ae7-431a-928a-7812319a3f38-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92f3caf-4ae7-431a-928a-7812319a3f38-maas-api-openshift-ca\") pod \"b92f3caf-4ae7-431a-928a-7812319a3f38\" (UID: \"b92f3caf-4ae7-431a-928a-7812319a3f38\") " Oct 30 19:51:50.857727 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.857710 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b92f3caf-4ae7-431a-928a-7812319a3f38-kube-api-access-qc5gn\" (UniqueName: \"kubernetes.io/projected/b92f3caf-4ae7-431a-928a-7812319a3f38-kube-api-access-qc5gn\") pod \"b92f3caf-4ae7-431a-928a-7812319a3f38\" (UID: \"b92f3caf-4ae7-431a-928a-7812319a3f38\") " Oct 30 19:51:50.857827 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.857799 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b92f3caf-4ae7-431a-928a-7812319a3f38-maas-api-openshift-ca" pod "b92f3caf-4ae7-431a-928a-7812319a3f38" (UID: "b92f3caf-4ae7-431a-928a-7812319a3f38"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:50.858378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.858350 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b92f3caf-4ae7-431a-928a-7812319a3f38-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:50.860051 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.860026 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b92f3caf-4ae7-431a-928a-7812319a3f38-maas-api-xks-ca" pod "b92f3caf-4ae7-431a-928a-7812319a3f38" (UID: "b92f3caf-4ae7-431a-928a-7812319a3f38"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:51:50.860140 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.860101 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b92f3caf-4ae7-431a-928a-7812319a3f38-kube-api-access-qc5gn" pod "b92f3caf-4ae7-431a-928a-7812319a3f38" (UID: "b92f3caf-4ae7-431a-928a-7812319a3f38"). InnerVolumeSpecName "kube-api-access-qc5gn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:50.960370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.960294 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qc5gn\" (UniqueName: \"kubernetes.io/projected/b92f3caf-4ae7-431a-928a-7812319a3f38-kube-api-access-qc5gn\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:50.960370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:50.960329 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b92f3caf-4ae7-431a-928a-7812319a3f38-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:51.327644 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.327604 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:51.340750 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.340720 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc"] Oct 30 19:51:51.341455 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.341439 5091 cpu_manager.go:414] "containerMap: removing container" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" Oct 30 19:51:51.341563 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.341457 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" Oct 30 19:51:51.341563 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.341484 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b92f3caf-4ae7-431a-928a-7812319a3f38" containerName="revoke-keys" Oct 30 19:51:51.341563 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.341489 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b92f3caf-4ae7-431a-928a-7812319a3f38" containerName="revoke-keys" Oct 30 19:51:51.341662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.341585 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="54f332bc-3668-4f09-a354-697e1610cd3d" containerName="istio-proxy" Oct 30 19:51:51.341662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.341598 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b92f3caf-4ae7-431a-928a-7812319a3f38" containerName="revoke-keys" Oct 30 19:51:51.345387 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.345365 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" Oct 30 19:51:51.352443 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.351552 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-b17e94\"" Oct 30 19:51:51.358701 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.358671 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc"] Oct 30 19:51:51.395604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.395386 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk"] Oct 30 19:51:51.400422 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.400399 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" Oct 30 19:51:51.411417 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.411390 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-b17e94-dockercfg-c6dnm\"" Oct 30 19:51:51.413639 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.413612 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk"] Oct 30 19:51:51.465481 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.465451 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" Oct 30 19:51:51.471962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.470545 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd" event={"ID":"b92f3caf-4ae7-431a-928a-7812319a3f38","Type":"ContainerDied","Data":"3d06bb5e07928d42b41e2ed2ab7f620c997b9e645e2a697f5d16ad31817fbbfb"} Oct 30 19:51:51.471962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.470596 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d06bb5e07928d42b41e2ed2ab7f620c997b9e645e2a697f5d16ad31817fbbfb" Oct 30 19:51:51.472788 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.472338 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sxwc\" (UniqueName: \"kubernetes.io/projected/afbd3cf9-21cd-4b40-94d2-df49a1146de6-kube-api-access-2sxwc\") pod \"payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk\" (UID: \"afbd3cf9-21cd-4b40-94d2-df49a1146de6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" Oct 30 19:51:51.472788 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.472421 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afbd3cf9-21cd-4b40-94d2-df49a1146de6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk\" (UID: \"afbd3cf9-21cd-4b40-94d2-df49a1146de6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" Oct 30 19:51:51.472788 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.472456 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c150c5f-1392-4a50-964d-a5e1593c2a19-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc\" (UID: \"3c150c5f-1392-4a50-964d-a5e1593c2a19\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" Oct 30 19:51:51.472788 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.472547 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6k5n\" (UniqueName: \"kubernetes.io/projected/3c150c5f-1392-4a50-964d-a5e1593c2a19-kube-api-access-b6k5n\") pod \"payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc\" (UID: \"3c150c5f-1392-4a50-964d-a5e1593c2a19\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" Oct 30 19:51:51.596222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.590238 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sxwc\" (UniqueName: \"kubernetes.io/projected/afbd3cf9-21cd-4b40-94d2-df49a1146de6-kube-api-access-2sxwc\") pod \"payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk\" (UID: \"afbd3cf9-21cd-4b40-94d2-df49a1146de6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" Oct 30 19:51:51.596222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.590374 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afbd3cf9-21cd-4b40-94d2-df49a1146de6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk\" (UID: \"afbd3cf9-21cd-4b40-94d2-df49a1146de6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" Oct 30 19:51:51.596222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.590410 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c150c5f-1392-4a50-964d-a5e1593c2a19-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc\" (UID: \"3c150c5f-1392-4a50-964d-a5e1593c2a19\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" Oct 30 19:51:51.596222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.590590 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6k5n\" (UniqueName: \"kubernetes.io/projected/3c150c5f-1392-4a50-964d-a5e1593c2a19-kube-api-access-b6k5n\") pod \"payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc\" (UID: \"3c150c5f-1392-4a50-964d-a5e1593c2a19\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" Oct 30 19:51:51.596222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.592952 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afbd3cf9-21cd-4b40-94d2-df49a1146de6-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk\" (UID: \"afbd3cf9-21cd-4b40-94d2-df49a1146de6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" Oct 30 19:51:51.612421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.611561 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:51.619833 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.619796 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c150c5f-1392-4a50-964d-a5e1593c2a19-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc\" (UID: \"3c150c5f-1392-4a50-964d-a5e1593c2a19\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" Oct 30 19:51:51.797169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.794968 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6k5n\" (UniqueName: \"kubernetes.io/projected/3c150c5f-1392-4a50-964d-a5e1593c2a19-kube-api-access-b6k5n\") pod \"payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc\" (UID: \"3c150c5f-1392-4a50-964d-a5e1593c2a19\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" Oct 30 19:51:51.814512 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.814482 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sxwc\" (UniqueName: \"kubernetes.io/projected/afbd3cf9-21cd-4b40-94d2-df49a1146de6-kube-api-access-2sxwc\") pod \"payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk\" (UID: \"afbd3cf9-21cd-4b40-94d2-df49a1146de6\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" Oct 30 19:51:51.877306 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.877266 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:51.983202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:51.983168 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:51:52.034526 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.034496 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" Oct 30 19:51:52.049891 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.049865 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" Oct 30 19:51:52.052298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.052265 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-token\" (UniqueName: \"kubernetes.io/projected/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-token\") pod \"b7ab432f-a929-40f9-8ed4-042d9528a20b\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " Oct 30 19:51:52.052419 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.052371 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7ab432f-a929-40f9-8ed4-042d9528a20b-kube-api-access-t2rk6\" (UniqueName: \"kubernetes.io/projected/b7ab432f-a929-40f9-8ed4-042d9528a20b-kube-api-access-t2rk6\") pod \"b7ab432f-a929-40f9-8ed4-042d9528a20b\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " Oct 30 19:51:52.052493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.052429 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-workload-socket\") pod \"b7ab432f-a929-40f9-8ed4-042d9528a20b\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " Oct 30 19:51:52.052493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.052465 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-podinfo\") pod \"b7ab432f-a929-40f9-8ed4-042d9528a20b\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " Oct 30 19:51:52.052603 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.052503 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-credential-socket\") pod \"b7ab432f-a929-40f9-8ed4-042d9528a20b\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " Oct 30 19:51:52.052603 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.052553 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-data\") pod \"b7ab432f-a929-40f9-8ed4-042d9528a20b\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " Oct 30 19:51:52.052706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.052616 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-envoy\") pod \"b7ab432f-a929-40f9-8ed4-042d9528a20b\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " Oct 30 19:51:52.052706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.052684 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7ab432f-a929-40f9-8ed4-042d9528a20b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ab432f-a929-40f9-8ed4-042d9528a20b-istiod-ca-cert\") pod \"b7ab432f-a929-40f9-8ed4-042d9528a20b\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " Oct 30 19:51:52.052809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.052732 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-workload-certs\") pod \"b7ab432f-a929-40f9-8ed4-042d9528a20b\" (UID: \"b7ab432f-a929-40f9-8ed4-042d9528a20b\") " Oct 30 19:51:52.053859 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.053831 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-credential-socket" pod "b7ab432f-a929-40f9-8ed4-042d9528a20b" (UID: "b7ab432f-a929-40f9-8ed4-042d9528a20b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:52.054185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.054139 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-workload-socket" pod "b7ab432f-a929-40f9-8ed4-042d9528a20b" (UID: "b7ab432f-a929-40f9-8ed4-042d9528a20b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:52.054494 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.054473 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7ab432f-a929-40f9-8ed4-042d9528a20b-istiod-ca-cert" pod "b7ab432f-a929-40f9-8ed4-042d9528a20b" (UID: "b7ab432f-a929-40f9-8ed4-042d9528a20b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:51:52.054943 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.054920 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-workload-certs" pod "b7ab432f-a929-40f9-8ed4-042d9528a20b" (UID: "b7ab432f-a929-40f9-8ed4-042d9528a20b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:52.055405 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.055384 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-data" pod "b7ab432f-a929-40f9-8ed4-042d9528a20b" (UID: "b7ab432f-a929-40f9-8ed4-042d9528a20b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:52.080541 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.080417 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-envoy" pod "b7ab432f-a929-40f9-8ed4-042d9528a20b" (UID: "b7ab432f-a929-40f9-8ed4-042d9528a20b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:51:52.080541 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.080486 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-token" pod "b7ab432f-a929-40f9-8ed4-042d9528a20b" (UID: "b7ab432f-a929-40f9-8ed4-042d9528a20b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:52.080727 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.080633 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7ab432f-a929-40f9-8ed4-042d9528a20b-kube-api-access-t2rk6" pod "b7ab432f-a929-40f9-8ed4-042d9528a20b" (UID: "b7ab432f-a929-40f9-8ed4-042d9528a20b"). InnerVolumeSpecName "kube-api-access-t2rk6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:51:52.083647 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.083622 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-podinfo" pod "b7ab432f-a929-40f9-8ed4-042d9528a20b" (UID: "b7ab432f-a929-40f9-8ed4-042d9528a20b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:51:52.166973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.166810 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2rk6\" (UniqueName: \"kubernetes.io/projected/b7ab432f-a929-40f9-8ed4-042d9528a20b-kube-api-access-t2rk6\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:52.166973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.166853 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:52.166973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.166868 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:52.166973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.166882 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:52.166973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.166896 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:52.166973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.166908 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:52.166973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.166922 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7ab432f-a929-40f9-8ed4-042d9528a20b-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:52.166973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.166935 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7ab432f-a929-40f9-8ed4-042d9528a20b-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:52.166973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.166948 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7ab432f-a929-40f9-8ed4-042d9528a20b-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:51:52.358534 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.358443 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc"] Oct 30 19:51:52.364617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.364404 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk"] Oct 30 19:51:52.476585 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.476542 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" event={"ID":"afbd3cf9-21cd-4b40-94d2-df49a1146de6","Type":"ContainerStarted","Data":"6eb83afccbfa90bb7d61d4072b2963763176e07c05eb2fb897715a80281c8b1a"} Oct 30 19:51:52.476966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.476605 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" event={"ID":"afbd3cf9-21cd-4b40-94d2-df49a1146de6","Type":"ContainerStarted","Data":"c8162b05cad5f69434aed90f75f28f335a13df4052cd77e35594c862a5941999"} Oct 30 19:51:52.476966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.476636 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" Oct 30 19:51:52.478757 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.478730 5091 generic.go:356] "Generic (PLEG): container finished" podID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerID="c4c1f5093aeb7f6937e21519ac309e0620da01bad97fe0fd393ff0a5f0c61dca" exitCode=0 Oct 30 19:51:52.478877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.478781 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" Oct 30 19:51:52.478877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.478831 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" event={"ID":"b7ab432f-a929-40f9-8ed4-042d9528a20b","Type":"ContainerDied","Data":"c4c1f5093aeb7f6937e21519ac309e0620da01bad97fe0fd393ff0a5f0c61dca"} Oct 30 19:51:52.478999 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.478888 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49" event={"ID":"b7ab432f-a929-40f9-8ed4-042d9528a20b","Type":"ContainerDied","Data":"31657b3c81d6f28c2583b77517b4bebeeb195941e7a15d5733b02a83c3ece41a"} Oct 30 19:51:52.478999 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.478921 5091 scope.go:122] "RemoveContainer" containerID="c4c1f5093aeb7f6937e21519ac309e0620da01bad97fe0fd393ff0a5f0c61dca" Oct 30 19:51:52.482356 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.482331 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" event={"ID":"3c150c5f-1392-4a50-964d-a5e1593c2a19","Type":"ContainerStarted","Data":"88dd0237e19085a5e64ecc9e1cc25d9923d11fa3cb3258495b84047907f1a658"} Oct 30 19:51:52.482451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.482382 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" event={"ID":"3c150c5f-1392-4a50-964d-a5e1593c2a19","Type":"ContainerStarted","Data":"a8ff125e85deb9654814e7299959d76dd3c22578f6388ee9fea5b01cd5242b74"} Oct 30 19:51:52.482451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.482413 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" Oct 30 19:51:52.493050 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.493029 5091 scope.go:122] "RemoveContainer" containerID="be1259bb71acb8835bd67056bd1b78f202e95c8543001a0cc4482a20efd6152a" Oct 30 19:51:52.502794 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.502752 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" podStartSLOduration=1.502739488 podStartE2EDuration="1.502739488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:52.498835637 +0000 UTC m=+1232.194043597" watchObservedRunningTime="2026-10-30 19:51:52.502739488 +0000 UTC m=+1232.197947446" Oct 30 19:51:52.505532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.505511 5091 scope.go:122] "RemoveContainer" containerID="c4c1f5093aeb7f6937e21519ac309e0620da01bad97fe0fd393ff0a5f0c61dca" Oct 30 19:51:52.505834 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:51:52.505816 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4c1f5093aeb7f6937e21519ac309e0620da01bad97fe0fd393ff0a5f0c61dca\": container with ID starting with c4c1f5093aeb7f6937e21519ac309e0620da01bad97fe0fd393ff0a5f0c61dca not found: ID does not exist" containerID="c4c1f5093aeb7f6937e21519ac309e0620da01bad97fe0fd393ff0a5f0c61dca" Oct 30 19:51:52.505884 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.505864 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4c1f5093aeb7f6937e21519ac309e0620da01bad97fe0fd393ff0a5f0c61dca"} err="failed to get container status \"c4c1f5093aeb7f6937e21519ac309e0620da01bad97fe0fd393ff0a5f0c61dca\": rpc error: code = NotFound desc = could not find container \"c4c1f5093aeb7f6937e21519ac309e0620da01bad97fe0fd393ff0a5f0c61dca\": container with ID starting with c4c1f5093aeb7f6937e21519ac309e0620da01bad97fe0fd393ff0a5f0c61dca not found: ID does not exist" Oct 30 19:51:52.505935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.505884 5091 scope.go:122] "RemoveContainer" containerID="be1259bb71acb8835bd67056bd1b78f202e95c8543001a0cc4482a20efd6152a" Oct 30 19:51:52.506121 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:51:52.506103 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be1259bb71acb8835bd67056bd1b78f202e95c8543001a0cc4482a20efd6152a\": container with ID starting with be1259bb71acb8835bd67056bd1b78f202e95c8543001a0cc4482a20efd6152a not found: ID does not exist" containerID="be1259bb71acb8835bd67056bd1b78f202e95c8543001a0cc4482a20efd6152a" Oct 30 19:51:52.506183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.506128 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be1259bb71acb8835bd67056bd1b78f202e95c8543001a0cc4482a20efd6152a"} err="failed to get container status \"be1259bb71acb8835bd67056bd1b78f202e95c8543001a0cc4482a20efd6152a\": rpc error: code = NotFound desc = could not find container \"be1259bb71acb8835bd67056bd1b78f202e95c8543001a0cc4482a20efd6152a\": container with ID starting with be1259bb71acb8835bd67056bd1b78f202e95c8543001a0cc4482a20efd6152a not found: ID does not exist" Oct 30 19:51:52.544023 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.543981 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" podStartSLOduration=1.543968897 podStartE2EDuration="1.543968897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:51:52.525232652 +0000 UTC m=+1232.220440615" watchObservedRunningTime="2026-10-30 19:51:52.543968897 +0000 UTC m=+1232.239176908" Oct 30 19:51:52.547821 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.547793 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49"] Oct 30 19:51:52.559017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.558988 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e907e4-openshift-default-5565b5d845-x8n49"] Oct 30 19:51:52.672270 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:52.672179 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:53.532018 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:53.531979 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:54.442998 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:54.442963 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" path="/var/lib/kubelet/pods/b7ab432f-a929-40f9-8ed4-042d9528a20b/volumes" Oct 30 19:51:54.775181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:54.774897 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:56.111470 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:56.110620 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:56.693012 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:56.692975 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:57.044094 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:57.044053 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:57.405230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:57.401611 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:58.404181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:58.403785 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:51:59.381448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:51:59.381406 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:00.364533 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:00.364488 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:00.452097 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:00.452055 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" Oct 30 19:52:01.721187 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:01.721122 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:01.901375 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:01.901336 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:02.075239 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.075192 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:02.394054 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.393966 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s"] Oct 30 19:52:02.395025 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.395005 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerName="istio-proxy" Oct 30 19:52:02.395079 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.395029 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerName="istio-proxy" Oct 30 19:52:02.395263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.395250 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerName="istio-proxy" Oct 30 19:52:02.395303 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.395271 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerName="istio-proxy" Oct 30 19:52:02.400961 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.400934 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" Oct 30 19:52:02.405156 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.405119 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s"] Oct 30 19:52:02.408209 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.408188 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:52:02.484368 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.484338 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" Oct 30 19:52:02.510731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.510699 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:52:02.539447 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.529215 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6769a587-f6fd-4b20-bebb-273b8f774b19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-913351-zm97s\" (UID: \"6769a587-f6fd-4b20-bebb-273b8f774b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" Oct 30 19:52:02.539447 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.529365 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5k4w\" (UniqueName: \"kubernetes.io/projected/6769a587-f6fd-4b20-bebb-273b8f774b19-kube-api-access-m5k4w\") pod \"maas-api-revoke-keys-e2e-ipp-a-913351-zm97s\" (UID: \"6769a587-f6fd-4b20-bebb-273b8f774b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" Oct 30 19:52:02.539447 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.529459 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6769a587-f6fd-4b20-bebb-273b8f774b19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-913351-zm97s\" (UID: \"6769a587-f6fd-4b20-bebb-273b8f774b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" Oct 30 19:52:02.651226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.633268 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5k4w\" (UniqueName: \"kubernetes.io/projected/6769a587-f6fd-4b20-bebb-273b8f774b19-kube-api-access-m5k4w\") pod \"maas-api-revoke-keys-e2e-ipp-a-913351-zm97s\" (UID: \"6769a587-f6fd-4b20-bebb-273b8f774b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" Oct 30 19:52:02.659167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.651586 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6769a587-f6fd-4b20-bebb-273b8f774b19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-913351-zm97s\" (UID: \"6769a587-f6fd-4b20-bebb-273b8f774b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" Oct 30 19:52:02.669489 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.659638 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6769a587-f6fd-4b20-bebb-273b8f774b19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-913351-zm97s\" (UID: \"6769a587-f6fd-4b20-bebb-273b8f774b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" Oct 30 19:52:02.669859 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.656940 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6769a587-f6fd-4b20-bebb-273b8f774b19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-913351-zm97s\" (UID: \"6769a587-f6fd-4b20-bebb-273b8f774b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" Oct 30 19:52:02.673861 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.673828 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6769a587-f6fd-4b20-bebb-273b8f774b19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-913351-zm97s\" (UID: \"6769a587-f6fd-4b20-bebb-273b8f774b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" Oct 30 19:52:02.674006 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.673889 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5k4w\" (UniqueName: \"kubernetes.io/projected/6769a587-f6fd-4b20-bebb-273b8f774b19-kube-api-access-m5k4w\") pod \"maas-api-revoke-keys-e2e-ipp-a-913351-zm97s\" (UID: \"6769a587-f6fd-4b20-bebb-273b8f774b19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" Oct 30 19:52:02.719780 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.719747 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" Oct 30 19:52:02.884177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:02.884004 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:03.092182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:03.088694 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s"] Oct 30 19:52:03.476839 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:03.476806 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577"] Oct 30 19:52:03.480601 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:03.480572 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2beb7b-zm577"] Oct 30 19:52:03.580223 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:03.574410 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" Oct 30 19:52:03.592873 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:03.580556 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" Oct 30 19:52:03.619260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:03.619224 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:03.693182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:03.691137 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" event={"ID":"6769a587-f6fd-4b20-bebb-273b8f774b19","Type":"ContainerStarted","Data":"8177dab00e2245eeb07f2d9fd020e64653d07c97dff98d42fbfc18ef679e08da"} Oct 30 19:52:03.693182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:03.691219 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" event={"ID":"6769a587-f6fd-4b20-bebb-273b8f774b19","Type":"ContainerStarted","Data":"23cb9f2f07c8a7780bc171abaa2a017abf60de46788bbc9b3d7c4457da35c531"} Oct 30 19:52:03.729493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:03.729381 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" podStartSLOduration=1.729362696 podStartE2EDuration="1.729362696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:03.7284986 +0000 UTC m=+1243.423706562" watchObservedRunningTime="2026-10-30 19:52:03.729362696 +0000 UTC m=+1243.424570656" Oct 30 19:52:03.865044 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:03.865007 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:04.079903 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:04.079863 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:04.453611 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:04.453483 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="553c0e2c-3812-4e18-b527-a5f74d006e27" path="/var/lib/kubelet/pods/553c0e2c-3812-4e18-b527-a5f74d006e27/volumes" Oct 30 19:52:04.700829 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:04.700796 5091 generic.go:356] "Generic (PLEG): container finished" podID="6769a587-f6fd-4b20-bebb-273b8f774b19" containerID="8177dab00e2245eeb07f2d9fd020e64653d07c97dff98d42fbfc18ef679e08da" exitCode=0 Oct 30 19:52:04.700978 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:04.700886 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" event={"ID":"6769a587-f6fd-4b20-bebb-273b8f774b19","Type":"ContainerDied","Data":"8177dab00e2245eeb07f2d9fd020e64653d07c97dff98d42fbfc18ef679e08da"} Oct 30 19:52:05.024104 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:05.024043 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:05.907228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:05.907189 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" Oct 30 19:52:05.961243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:05.961169 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6769a587-f6fd-4b20-bebb-273b8f774b19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6769a587-f6fd-4b20-bebb-273b8f774b19-maas-api-xks-ca\") pod \"6769a587-f6fd-4b20-bebb-273b8f774b19\" (UID: \"6769a587-f6fd-4b20-bebb-273b8f774b19\") " Oct 30 19:52:05.961412 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:05.961347 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6769a587-f6fd-4b20-bebb-273b8f774b19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6769a587-f6fd-4b20-bebb-273b8f774b19-maas-api-openshift-ca\") pod \"6769a587-f6fd-4b20-bebb-273b8f774b19\" (UID: \"6769a587-f6fd-4b20-bebb-273b8f774b19\") " Oct 30 19:52:05.961586 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:05.961561 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6769a587-f6fd-4b20-bebb-273b8f774b19-kube-api-access-m5k4w\" (UniqueName: \"kubernetes.io/projected/6769a587-f6fd-4b20-bebb-273b8f774b19-kube-api-access-m5k4w\") pod \"6769a587-f6fd-4b20-bebb-273b8f774b19\" (UID: \"6769a587-f6fd-4b20-bebb-273b8f774b19\") " Oct 30 19:52:05.962006 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:05.961981 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6769a587-f6fd-4b20-bebb-273b8f774b19-maas-api-openshift-ca" pod "6769a587-f6fd-4b20-bebb-273b8f774b19" (UID: "6769a587-f6fd-4b20-bebb-273b8f774b19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:05.962342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:05.962319 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6769a587-f6fd-4b20-bebb-273b8f774b19-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:05.963900 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:05.963859 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6769a587-f6fd-4b20-bebb-273b8f774b19-maas-api-xks-ca" pod "6769a587-f6fd-4b20-bebb-273b8f774b19" (UID: "6769a587-f6fd-4b20-bebb-273b8f774b19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:52:05.965510 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:05.965458 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6769a587-f6fd-4b20-bebb-273b8f774b19-kube-api-access-m5k4w" pod "6769a587-f6fd-4b20-bebb-273b8f774b19" (UID: "6769a587-f6fd-4b20-bebb-273b8f774b19"). InnerVolumeSpecName "kube-api-access-m5k4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:06.064134 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:06.064104 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5k4w\" (UniqueName: \"kubernetes.io/projected/6769a587-f6fd-4b20-bebb-273b8f774b19-kube-api-access-m5k4w\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:06.064134 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:06.064134 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6769a587-f6fd-4b20-bebb-273b8f774b19-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:06.109094 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:06.108857 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:06.717884 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:06.717811 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" Oct 30 19:52:06.717884 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:06.717823 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s" event={"ID":"6769a587-f6fd-4b20-bebb-273b8f774b19","Type":"ContainerDied","Data":"23cb9f2f07c8a7780bc171abaa2a017abf60de46788bbc9b3d7c4457da35c531"} Oct 30 19:52:06.718070 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:06.717888 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23cb9f2f07c8a7780bc171abaa2a017abf60de46788bbc9b3d7c4457da35c531" Oct 30 19:52:07.383084 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:07.383048 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:08.528167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:08.528085 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:09.894003 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:09.893968 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:10.597393 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:10.597363 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l_be3e710b-55f4-4cd6-8872-f7db69725afc/payload-processing/0.log" Oct 30 19:52:11.334162 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.334122 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb"] Oct 30 19:52:11.335026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.335003 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6769a587-f6fd-4b20-bebb-273b8f774b19" containerName="revoke-keys" Oct 30 19:52:11.335138 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.335028 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6769a587-f6fd-4b20-bebb-273b8f774b19" containerName="revoke-keys" Oct 30 19:52:11.335138 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.335086 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerName="istio-proxy" Oct 30 19:52:11.335138 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.335095 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7ab432f-a929-40f9-8ed4-042d9528a20b" containerName="istio-proxy" Oct 30 19:52:11.335414 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.335308 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6769a587-f6fd-4b20-bebb-273b8f774b19" containerName="revoke-keys" Oct 30 19:52:11.339073 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.339053 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" Oct 30 19:52:11.349525 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.349501 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb"] Oct 30 19:52:11.352903 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.351724 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:52:11.352903 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.351792 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:52:11.374160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.374108 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:11.473632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.473595 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f198c-ad87-454f-b7a4-19c194d4ac34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb\" (UID: \"cc8f198c-ad87-454f-b7a4-19c194d4ac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" Oct 30 19:52:11.473793 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.473733 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7k8h\" (UniqueName: \"kubernetes.io/projected/cc8f198c-ad87-454f-b7a4-19c194d4ac34-kube-api-access-p7k8h\") pod \"maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb\" (UID: \"cc8f198c-ad87-454f-b7a4-19c194d4ac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" Oct 30 19:52:11.473836 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.473802 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8f198c-ad87-454f-b7a4-19c194d4ac34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb\" (UID: \"cc8f198c-ad87-454f-b7a4-19c194d4ac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" Oct 30 19:52:11.574824 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.574790 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f198c-ad87-454f-b7a4-19c194d4ac34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb\" (UID: \"cc8f198c-ad87-454f-b7a4-19c194d4ac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" Oct 30 19:52:11.575010 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.574906 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7k8h\" (UniqueName: \"kubernetes.io/projected/cc8f198c-ad87-454f-b7a4-19c194d4ac34-kube-api-access-p7k8h\") pod \"maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb\" (UID: \"cc8f198c-ad87-454f-b7a4-19c194d4ac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" Oct 30 19:52:11.575010 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.574963 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8f198c-ad87-454f-b7a4-19c194d4ac34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb\" (UID: \"cc8f198c-ad87-454f-b7a4-19c194d4ac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" Oct 30 19:52:11.575672 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.575648 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f198c-ad87-454f-b7a4-19c194d4ac34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb\" (UID: \"cc8f198c-ad87-454f-b7a4-19c194d4ac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" Oct 30 19:52:11.577783 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.577759 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8f198c-ad87-454f-b7a4-19c194d4ac34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb\" (UID: \"cc8f198c-ad87-454f-b7a4-19c194d4ac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" Oct 30 19:52:11.590963 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.590916 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7k8h\" (UniqueName: \"kubernetes.io/projected/cc8f198c-ad87-454f-b7a4-19c194d4ac34-kube-api-access-p7k8h\") pod \"maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb\" (UID: \"cc8f198c-ad87-454f-b7a4-19c194d4ac34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" Oct 30 19:52:11.661055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.661022 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" Oct 30 19:52:11.824858 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.824806 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:11.971454 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:11.971424 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb"] Oct 30 19:52:11.971890 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:52:11.971863 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8f198c_ad87_454f_b7a4_19c194d4ac34.slice/crio-cd7d23a02345a08f7e201860c8dd6a3e96f256bd19030ee519c832d3d2bea38b WatchSource:0}: Error finding container cd7d23a02345a08f7e201860c8dd6a3e96f256bd19030ee519c832d3d2bea38b: Status 404 returned error can't find the container with id cd7d23a02345a08f7e201860c8dd6a3e96f256bd19030ee519c832d3d2bea38b Oct 30 19:52:12.068246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:12.062481 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:12.242582 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:12.242514 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:12.707168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:12.707003 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:12.783313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:12.782451 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" event={"ID":"cc8f198c-ad87-454f-b7a4-19c194d4ac34","Type":"ContainerStarted","Data":"6456ca098c6e82f3fc20fa6e5a845f1744d3fb6d139dcbabc8824ab4923c8075"} Oct 30 19:52:12.783313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:12.782528 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" event={"ID":"cc8f198c-ad87-454f-b7a4-19c194d4ac34","Type":"ContainerStarted","Data":"cd7d23a02345a08f7e201860c8dd6a3e96f256bd19030ee519c832d3d2bea38b"} Oct 30 19:52:12.840679 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:12.840629 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" podStartSLOduration=1.840615109 podStartE2EDuration="1.840615109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:12.838757766 +0000 UTC m=+1252.533965729" watchObservedRunningTime="2026-10-30 19:52:12.840615109 +0000 UTC m=+1252.535823095" Oct 30 19:52:13.045535 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:13.045497 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:13.792291 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:13.792255 5091 generic.go:356] "Generic (PLEG): container finished" podID="cc8f198c-ad87-454f-b7a4-19c194d4ac34" containerID="6456ca098c6e82f3fc20fa6e5a845f1744d3fb6d139dcbabc8824ab4923c8075" exitCode=0 Oct 30 19:52:13.792675 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:13.792341 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" event={"ID":"cc8f198c-ad87-454f-b7a4-19c194d4ac34","Type":"ContainerDied","Data":"6456ca098c6e82f3fc20fa6e5a845f1744d3fb6d139dcbabc8824ab4923c8075"} Oct 30 19:52:13.905188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:13.904231 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:14.535779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:14.535744 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:14.928454 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:14.928419 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd"] Oct 30 19:52:14.930900 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:14.930862 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:14.933118 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:14.933095 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:14.945071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:14.945033 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-490cdf-openshift-default-dockercfg-8fw4b\"" Oct 30 19:52:14.992337 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:14.991848 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd"] Oct 30 19:52:15.023672 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.023645 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" Oct 30 19:52:15.034729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.034692 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-workload-certs\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.035017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.034995 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4xf4\" (UniqueName: \"kubernetes.io/projected/0b09a84c-ae99-434b-8562-2a921c923e33-kube-api-access-b4xf4\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.035206 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.035187 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b09a84c-ae99-434b-8562-2a921c923e33-istio-token\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.035356 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.035341 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b09a84c-ae99-434b-8562-2a921c923e33-istio-podinfo\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.035557 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.035540 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-istio-envoy\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.035774 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.035756 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-istio-data\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.035952 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.035934 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b09a84c-ae99-434b-8562-2a921c923e33-istiod-ca-cert\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.036094 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.036078 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-workload-socket\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.036289 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.036271 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-credential-socket\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.141747 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.141652 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-workload-certs\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.141999 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.141972 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xf4\" (UniqueName: \"kubernetes.io/projected/0b09a84c-ae99-434b-8562-2a921c923e33-kube-api-access-b4xf4\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.142261 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.142140 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b09a84c-ae99-434b-8562-2a921c923e33-istio-token\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.142436 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.142419 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b09a84c-ae99-434b-8562-2a921c923e33-istio-podinfo\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.221173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.209473 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-istio-envoy\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.221173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.209519 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-istio-data\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.221173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.209576 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b09a84c-ae99-434b-8562-2a921c923e33-istiod-ca-cert\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.221173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.209617 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-workload-socket\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.221173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.209703 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-credential-socket\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.221173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.210340 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-credential-socket\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.221173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.196586 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-workload-certs\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.221173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.208273 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:15.221173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.207752 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b09a84c-ae99-434b-8562-2a921c923e33-istio-podinfo\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.221173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.210960 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-istio-data\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.221173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.213161 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4xf4\" (UniqueName: \"kubernetes.io/projected/0b09a84c-ae99-434b-8562-2a921c923e33-kube-api-access-b4xf4\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.221173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.213444 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-workload-socket\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.240171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.236823 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b09a84c-ae99-434b-8562-2a921c923e33-istiod-ca-cert\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.242199 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.242169 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-istio-envoy\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.292498 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.283691 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b09a84c-ae99-434b-8562-2a921c923e33-istio-token\") pod \"e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.327662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.327291 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8f198c-ad87-454f-b7a4-19c194d4ac34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f198c-ad87-454f-b7a4-19c194d4ac34-maas-api-openshift-ca\") pod \"cc8f198c-ad87-454f-b7a4-19c194d4ac34\" (UID: \"cc8f198c-ad87-454f-b7a4-19c194d4ac34\") " Oct 30 19:52:15.327662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.327375 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc8f198c-ad87-454f-b7a4-19c194d4ac34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8f198c-ad87-454f-b7a4-19c194d4ac34-maas-api-xks-ca\") pod \"cc8f198c-ad87-454f-b7a4-19c194d4ac34\" (UID: \"cc8f198c-ad87-454f-b7a4-19c194d4ac34\") " Oct 30 19:52:15.327662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.327438 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8f198c-ad87-454f-b7a4-19c194d4ac34-kube-api-access-p7k8h\" (UniqueName: \"kubernetes.io/projected/cc8f198c-ad87-454f-b7a4-19c194d4ac34-kube-api-access-p7k8h\") pod \"cc8f198c-ad87-454f-b7a4-19c194d4ac34\" (UID: \"cc8f198c-ad87-454f-b7a4-19c194d4ac34\") " Oct 30 19:52:15.332745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.332711 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8f198c-ad87-454f-b7a4-19c194d4ac34-maas-api-openshift-ca" pod "cc8f198c-ad87-454f-b7a4-19c194d4ac34" (UID: "cc8f198c-ad87-454f-b7a4-19c194d4ac34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:15.335891 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.335834 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc8f198c-ad87-454f-b7a4-19c194d4ac34-maas-api-xks-ca" pod "cc8f198c-ad87-454f-b7a4-19c194d4ac34" (UID: "cc8f198c-ad87-454f-b7a4-19c194d4ac34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:52:15.339486 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.339447 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8f198c-ad87-454f-b7a4-19c194d4ac34-kube-api-access-p7k8h" pod "cc8f198c-ad87-454f-b7a4-19c194d4ac34" (UID: "cc8f198c-ad87-454f-b7a4-19c194d4ac34"). InnerVolumeSpecName "kube-api-access-p7k8h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:15.429744 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.429673 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc8f198c-ad87-454f-b7a4-19c194d4ac34-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:15.429744 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.429703 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc8f198c-ad87-454f-b7a4-19c194d4ac34-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:15.429744 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.429714 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7k8h\" (UniqueName: \"kubernetes.io/projected/cc8f198c-ad87-454f-b7a4-19c194d4ac34-kube-api-access-p7k8h\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:15.556409 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.556381 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:15.812075 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.812048 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" Oct 30 19:52:15.812263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.812083 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb" event={"ID":"cc8f198c-ad87-454f-b7a4-19c194d4ac34","Type":"ContainerDied","Data":"cd7d23a02345a08f7e201860c8dd6a3e96f256bd19030ee519c832d3d2bea38b"} Oct 30 19:52:15.812263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.812138 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd7d23a02345a08f7e201860c8dd6a3e96f256bd19030ee519c832d3d2bea38b" Oct 30 19:52:15.848429 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:52:15.848405 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b09a84c_ae99_434b_8562_2a921c923e33.slice/crio-2d1035ce0c174b8dd45c73e5e942cf0ff31b6dabdcce62f770e75dc595668096 WatchSource:0}: Error finding container 2d1035ce0c174b8dd45c73e5e942cf0ff31b6dabdcce62f770e75dc595668096: Status 404 returned error can't find the container with id 2d1035ce0c174b8dd45c73e5e942cf0ff31b6dabdcce62f770e75dc595668096 Oct 30 19:52:15.849994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.849971 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd"] Oct 30 19:52:15.851505 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.851475 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:52:15.851601 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.851535 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:52:15.851601 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:15.851566 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:52:15.989036 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:52:15.988985 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:52:16.286223 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:16.283557 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:16.822977 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:16.822940 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" event={"ID":"0b09a84c-ae99-434b-8562-2a921c923e33","Type":"ContainerStarted","Data":"282c799af4a39cb8f9aab241a18179721065caeb52aa98b75b7f49c8abdf7621"} Oct 30 19:52:16.823128 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:16.822989 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" event={"ID":"0b09a84c-ae99-434b-8562-2a921c923e33","Type":"ContainerStarted","Data":"2d1035ce0c174b8dd45c73e5e942cf0ff31b6dabdcce62f770e75dc595668096"} Oct 30 19:52:16.861072 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:16.861008 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" podStartSLOduration=2.860988583 podStartE2EDuration="2.860988583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:16.852807501 +0000 UTC m=+1256.548015461" watchObservedRunningTime="2026-10-30 19:52:16.860988583 +0000 UTC m=+1256.556196542" Oct 30 19:52:17.556809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:17.556769 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:17.565280 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:17.564589 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:17.567357 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:17.567319 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:17.831783 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:17.831696 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:17.832945 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:17.832918 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:52:19.012315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:19.012269 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:19.375746 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:19.375636 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:19.587943 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:19.587896 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:19.989050 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:19.989016 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc"] Oct 30 19:52:19.989731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:19.989713 5091 cpu_manager.go:414] "containerMap: removing container" podUID="cc8f198c-ad87-454f-b7a4-19c194d4ac34" containerName="revoke-keys" Oct 30 19:52:19.989731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:19.989730 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8f198c-ad87-454f-b7a4-19c194d4ac34" containerName="revoke-keys" Oct 30 19:52:19.989896 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:19.989848 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8f198c-ad87-454f-b7a4-19c194d4ac34" containerName="revoke-keys" Oct 30 19:52:19.993773 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:19.993749 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" Oct 30 19:52:20.011378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.011337 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:52:20.012027 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.012000 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc"] Oct 30 19:52:20.013062 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.013040 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:52:20.110418 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.110388 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ed83de-zx5cc\" (UID: \"4220b4fa-54d6-4fc4-bc66-24f89798c1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" Oct 30 19:52:20.110626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.110587 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ed83de-zx5cc\" (UID: \"4220b4fa-54d6-4fc4-bc66-24f89798c1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" Oct 30 19:52:20.110720 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.110625 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6tt6\" (UniqueName: \"kubernetes.io/projected/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-kube-api-access-t6tt6\") pod \"maas-api-revoke-sub-f2ed83de-zx5cc\" (UID: \"4220b4fa-54d6-4fc4-bc66-24f89798c1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" Oct 30 19:52:20.212185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.212124 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ed83de-zx5cc\" (UID: \"4220b4fa-54d6-4fc4-bc66-24f89798c1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" Oct 30 19:52:20.212426 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.212191 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6tt6\" (UniqueName: \"kubernetes.io/projected/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-kube-api-access-t6tt6\") pod \"maas-api-revoke-sub-f2ed83de-zx5cc\" (UID: \"4220b4fa-54d6-4fc4-bc66-24f89798c1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" Oct 30 19:52:20.212506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.212428 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ed83de-zx5cc\" (UID: \"4220b4fa-54d6-4fc4-bc66-24f89798c1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" Oct 30 19:52:20.212921 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.212892 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2ed83de-zx5cc\" (UID: \"4220b4fa-54d6-4fc4-bc66-24f89798c1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" Oct 30 19:52:20.215023 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.215003 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2ed83de-zx5cc\" (UID: \"4220b4fa-54d6-4fc4-bc66-24f89798c1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" Oct 30 19:52:20.227462 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.227443 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6tt6\" (UniqueName: \"kubernetes.io/projected/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-kube-api-access-t6tt6\") pod \"maas-api-revoke-sub-f2ed83de-zx5cc\" (UID: \"4220b4fa-54d6-4fc4-bc66-24f89798c1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" Oct 30 19:52:20.280441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.280403 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:20.311334 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.311299 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" Oct 30 19:52:20.540584 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.540491 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:20.675141 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.675108 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc"] Oct 30 19:52:20.676055 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:52:20.676021 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4220b4fa_54d6_4fc4_bc66_24f89798c1bf.slice/crio-672caa2496bd2d4f369a8561058d172c1557a499552213ce48bc860b08c96dc2 WatchSource:0}: Error finding container 672caa2496bd2d4f369a8561058d172c1557a499552213ce48bc860b08c96dc2: Status 404 returned error can't find the container with id 672caa2496bd2d4f369a8561058d172c1557a499552213ce48bc860b08c96dc2 Oct 30 19:52:20.812736 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.812650 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:20.977229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:20.977194 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m"] Oct 30 19:52:21.055211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.038346 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" event={"ID":"4220b4fa-54d6-4fc4-bc66-24f89798c1bf","Type":"ContainerStarted","Data":"672caa2496bd2d4f369a8561058d172c1557a499552213ce48bc860b08c96dc2"} Oct 30 19:52:21.055211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.038402 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m"] Oct 30 19:52:21.055211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.038509 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" Oct 30 19:52:21.055211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.053539 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-490cdf\"" Oct 30 19:52:21.066277 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.057337 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-490cdf-dockercfg-dptjf\"" Oct 30 19:52:21.106171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.099856 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:21.108289 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.108257 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z"] Oct 30 19:52:21.122752 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.122724 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" Oct 30 19:52:21.217189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.211140 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb6f324-f820-459e-9419-5a89e5b59e6f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m\" (UID: \"afb6f324-f820-459e-9419-5a89e5b59e6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" Oct 30 19:52:21.217189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.211485 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2c9b06-2e9f-4a52-9f31-48e043063a1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z\" (UID: \"fe2c9b06-2e9f-4a52-9f31-48e043063a1d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" Oct 30 19:52:21.217189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.211518 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whbmh\" (UniqueName: \"kubernetes.io/projected/afb6f324-f820-459e-9419-5a89e5b59e6f-kube-api-access-whbmh\") pod \"payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m\" (UID: \"afb6f324-f820-459e-9419-5a89e5b59e6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" Oct 30 19:52:21.217189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.211575 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ht5c\" (UniqueName: \"kubernetes.io/projected/fe2c9b06-2e9f-4a52-9f31-48e043063a1d-kube-api-access-9ht5c\") pod \"payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z\" (UID: \"fe2c9b06-2e9f-4a52-9f31-48e043063a1d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" Oct 30 19:52:21.249824 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.249796 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z"] Oct 30 19:52:21.313078 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.313033 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2c9b06-2e9f-4a52-9f31-48e043063a1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z\" (UID: \"fe2c9b06-2e9f-4a52-9f31-48e043063a1d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" Oct 30 19:52:21.313078 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.313080 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whbmh\" (UniqueName: \"kubernetes.io/projected/afb6f324-f820-459e-9419-5a89e5b59e6f-kube-api-access-whbmh\") pod \"payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m\" (UID: \"afb6f324-f820-459e-9419-5a89e5b59e6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" Oct 30 19:52:21.313353 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.313125 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ht5c\" (UniqueName: \"kubernetes.io/projected/fe2c9b06-2e9f-4a52-9f31-48e043063a1d-kube-api-access-9ht5c\") pod \"payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z\" (UID: \"fe2c9b06-2e9f-4a52-9f31-48e043063a1d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" Oct 30 19:52:21.313353 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.313202 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb6f324-f820-459e-9419-5a89e5b59e6f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m\" (UID: \"afb6f324-f820-459e-9419-5a89e5b59e6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" Oct 30 19:52:21.313863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.313838 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2c9b06-2e9f-4a52-9f31-48e043063a1d-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z\" (UID: \"fe2c9b06-2e9f-4a52-9f31-48e043063a1d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" Oct 30 19:52:21.314012 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.313991 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb6f324-f820-459e-9419-5a89e5b59e6f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m\" (UID: \"afb6f324-f820-459e-9419-5a89e5b59e6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" Oct 30 19:52:21.330617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.330553 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ht5c\" (UniqueName: \"kubernetes.io/projected/fe2c9b06-2e9f-4a52-9f31-48e043063a1d-kube-api-access-9ht5c\") pod \"payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z\" (UID: \"fe2c9b06-2e9f-4a52-9f31-48e043063a1d\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" Oct 30 19:52:21.330754 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.330681 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whbmh\" (UniqueName: \"kubernetes.io/projected/afb6f324-f820-459e-9419-5a89e5b59e6f-kube-api-access-whbmh\") pod \"payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m\" (UID: \"afb6f324-f820-459e-9419-5a89e5b59e6f\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" Oct 30 19:52:21.353627 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.353596 5091 scope.go:122] "RemoveContainer" containerID="887e56383e2b89311f336d4b2f7adfde157fa8ea945c44ffabe4d314cd589a76" Oct 30 19:52:21.368377 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.368353 5091 scope.go:122] "RemoveContainer" containerID="047571ec2427200680926b3e3e1a0f4855a5356ea0383234ce731ff70debb2d4" Oct 30 19:52:21.409286 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.409258 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" Oct 30 19:52:21.577539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.577506 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" Oct 30 19:52:21.660220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.658092 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:21.734505 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.734473 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m"] Oct 30 19:52:21.744907 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:52:21.744861 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb6f324_f820_459e_9419_5a89e5b59e6f.slice/crio-b036e5ffcc019144dcd79b037a1edcfbe5e7b91fcf2a86fb3d7f4d6310690ed4 WatchSource:0}: Error finding container b036e5ffcc019144dcd79b037a1edcfbe5e7b91fcf2a86fb3d7f4d6310690ed4: Status 404 returned error can't find the container with id b036e5ffcc019144dcd79b037a1edcfbe5e7b91fcf2a86fb3d7f4d6310690ed4 Oct 30 19:52:21.861403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.860863 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:21.958750 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:21.958716 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z"] Oct 30 19:52:21.959636 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:52:21.959606 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe2c9b06_2e9f_4a52_9f31_48e043063a1d.slice/crio-cf768937739fddcf83aadf3765dae30a184afc1fce6ca1879e78df850c6cc525 WatchSource:0}: Error finding container cf768937739fddcf83aadf3765dae30a184afc1fce6ca1879e78df850c6cc525: Status 404 returned error can't find the container with id cf768937739fddcf83aadf3765dae30a184afc1fce6ca1879e78df850c6cc525 Oct 30 19:52:22.035188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.028813 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" event={"ID":"afb6f324-f820-459e-9419-5a89e5b59e6f","Type":"ContainerStarted","Data":"7eda6418d319208bcdf7b372b303f5aca4ce90f70a1bc3d2d2cd0e5ee428ca10"} Oct 30 19:52:22.035188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.028877 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" event={"ID":"afb6f324-f820-459e-9419-5a89e5b59e6f","Type":"ContainerStarted","Data":"b036e5ffcc019144dcd79b037a1edcfbe5e7b91fcf2a86fb3d7f4d6310690ed4"} Oct 30 19:52:22.035188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.030003 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" Oct 30 19:52:22.035408 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.035257 5091 generic.go:356] "Generic (PLEG): container finished" podID="4220b4fa-54d6-4fc4-bc66-24f89798c1bf" containerID="4c3db7eaa8f860ce6eebcc2bfea78bec262704c892ea34ebf36a30a2735a36ef" exitCode=0 Oct 30 19:52:22.035408 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.035383 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" event={"ID":"4220b4fa-54d6-4fc4-bc66-24f89798c1bf","Type":"ContainerDied","Data":"4c3db7eaa8f860ce6eebcc2bfea78bec262704c892ea34ebf36a30a2735a36ef"} Oct 30 19:52:22.035893 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.035868 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd"] Oct 30 19:52:22.042599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.042575 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" event={"ID":"fe2c9b06-2e9f-4a52-9f31-48e043063a1d","Type":"ContainerStarted","Data":"cf768937739fddcf83aadf3765dae30a184afc1fce6ca1879e78df850c6cc525"} Oct 30 19:52:22.042721 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.042710 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.052070 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.052042 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd"] Oct 30 19:52:22.059647 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.059624 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-f03120-openshift-default-dockercfg-wxvjh\"" Oct 30 19:52:22.103534 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.103466 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" podStartSLOduration=2.103445324 podStartE2EDuration="2.103445324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:22.098302083 +0000 UTC m=+1261.793510042" watchObservedRunningTime="2026-10-30 19:52:22.103445324 +0000 UTC m=+1261.798653282" Oct 30 19:52:22.248245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.229307 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-istio-envoy\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.248245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.229388 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bff8956-093c-4a78-a281-8571789127a7-istiod-ca-cert\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.248245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.229432 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bff8956-093c-4a78-a281-8571789127a7-istio-token\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.248245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.229498 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-istio-data\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.248245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.229534 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5c47\" (UniqueName: \"kubernetes.io/projected/2bff8956-093c-4a78-a281-8571789127a7-kube-api-access-h5c47\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.248245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.229567 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-workload-socket\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.248245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.229665 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-workload-certs\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.248245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.229715 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-credential-socket\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.248245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.229825 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bff8956-093c-4a78-a281-8571789127a7-istio-podinfo\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.300185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.283343 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:22.337175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.333212 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-istio-data\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.337175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.333268 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5c47\" (UniqueName: \"kubernetes.io/projected/2bff8956-093c-4a78-a281-8571789127a7-kube-api-access-h5c47\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.337175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.333300 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-workload-socket\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.337175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.333390 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-workload-certs\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.337175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.333442 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-credential-socket\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.337175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.333554 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bff8956-093c-4a78-a281-8571789127a7-istio-podinfo\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.337175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.333685 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-istio-envoy\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.337175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.333746 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bff8956-093c-4a78-a281-8571789127a7-istiod-ca-cert\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.337175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.333795 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bff8956-093c-4a78-a281-8571789127a7-istio-token\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.357196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.346139 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-istio-data\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.357196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.350520 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-credential-socket\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.357196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.350791 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-workload-socket\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.357196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.351020 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-workload-certs\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.376820 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.375204 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bff8956-093c-4a78-a281-8571789127a7-istiod-ca-cert\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.421049 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.420978 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bff8956-093c-4a78-a281-8571789127a7-istio-token\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.429859 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.429816 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bff8956-093c-4a78-a281-8571789127a7-istio-podinfo\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.430116 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.430066 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5c47\" (UniqueName: \"kubernetes.io/projected/2bff8956-093c-4a78-a281-8571789127a7-kube-api-access-h5c47\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.430232 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.430165 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-istio-envoy\") pod \"e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.523168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.522971 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:22.675286 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.675242 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:22.979525 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:52:22.979494 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bff8956_093c_4a78_a281_8571789127a7.slice/crio-5c7330de0e69f575f65f9eb6c5b8bd22fee895383274048943b6718a2caa2647 WatchSource:0}: Error finding container 5c7330de0e69f575f65f9eb6c5b8bd22fee895383274048943b6718a2caa2647: Status 404 returned error can't find the container with id 5c7330de0e69f575f65f9eb6c5b8bd22fee895383274048943b6718a2caa2647 Oct 30 19:52:22.981434 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.981405 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd"] Oct 30 19:52:22.982698 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.982663 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:52:22.982785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.982727 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:52:22.982785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:22.982755 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:52:23.048229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:23.048154 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" event={"ID":"fe2c9b06-2e9f-4a52-9f31-48e043063a1d","Type":"ContainerStarted","Data":"21c497742ea62d23a8a62bb517145505dc6f54aa7db646ce77b9d6330faf2a70"} Oct 30 19:52:23.048352 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:23.048262 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" Oct 30 19:52:23.049915 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:23.049890 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" event={"ID":"2bff8956-093c-4a78-a281-8571789127a7","Type":"ContainerStarted","Data":"5c7330de0e69f575f65f9eb6c5b8bd22fee895383274048943b6718a2caa2647"} Oct 30 19:52:23.089680 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:23.089606 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" podStartSLOduration=3.089585025 podStartE2EDuration="3.089585025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:23.088905382 +0000 UTC m=+1262.784113341" watchObservedRunningTime="2026-10-30 19:52:23.089585025 +0000 UTC m=+1262.784792983" Oct 30 19:52:23.306167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:23.298818 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:23.322168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:23.317591 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" Oct 30 19:52:23.411170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:23.411104 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-maas-api-xks-ca\") pod \"4220b4fa-54d6-4fc4-bc66-24f89798c1bf\" (UID: \"4220b4fa-54d6-4fc4-bc66-24f89798c1bf\") " Oct 30 19:52:23.414708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:23.411385 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-maas-api-openshift-ca\") pod \"4220b4fa-54d6-4fc4-bc66-24f89798c1bf\" (UID: \"4220b4fa-54d6-4fc4-bc66-24f89798c1bf\") " Oct 30 19:52:23.414708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:23.411535 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-kube-api-access-t6tt6\" (UniqueName: \"kubernetes.io/projected/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-kube-api-access-t6tt6\") pod \"4220b4fa-54d6-4fc4-bc66-24f89798c1bf\" (UID: \"4220b4fa-54d6-4fc4-bc66-24f89798c1bf\") " Oct 30 19:52:23.431218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:23.415561 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-maas-api-openshift-ca" pod "4220b4fa-54d6-4fc4-bc66-24f89798c1bf" (UID: "4220b4fa-54d6-4fc4-bc66-24f89798c1bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:23.431218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:23.415688 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-kube-api-access-t6tt6" pod "4220b4fa-54d6-4fc4-bc66-24f89798c1bf" (UID: "4220b4fa-54d6-4fc4-bc66-24f89798c1bf"). InnerVolumeSpecName "kube-api-access-t6tt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:23.455355 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:23.455047 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-maas-api-xks-ca" pod "4220b4fa-54d6-4fc4-bc66-24f89798c1bf" (UID: "4220b4fa-54d6-4fc4-bc66-24f89798c1bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:52:23.513985 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:23.513953 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:23.513985 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:23.513982 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6tt6\" (UniqueName: \"kubernetes.io/projected/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-kube-api-access-t6tt6\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:23.514287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:23.514002 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4220b4fa-54d6-4fc4-bc66-24f89798c1bf-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:24.129168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:24.116414 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" event={"ID":"4220b4fa-54d6-4fc4-bc66-24f89798c1bf","Type":"ContainerDied","Data":"672caa2496bd2d4f369a8561058d172c1557a499552213ce48bc860b08c96dc2"} Oct 30 19:52:24.129168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:24.116483 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="672caa2496bd2d4f369a8561058d172c1557a499552213ce48bc860b08c96dc2" Oct 30 19:52:24.129168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:24.116558 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc" Oct 30 19:52:24.177706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:24.177669 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" event={"ID":"2bff8956-093c-4a78-a281-8571789127a7","Type":"ContainerStarted","Data":"daaa5ef9b5bc570c59b69658ec8d3fc45424a35ba2b8d89db66a46dbb8df6743"} Oct 30 19:52:24.245913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:24.245842 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" podStartSLOduration=3.2458205270000002 podStartE2EDuration="3.245820527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:24.227611165 +0000 UTC m=+1263.922819124" watchObservedRunningTime="2026-10-30 19:52:24.245820527 +0000 UTC m=+1263.941028486" Oct 30 19:52:24.680774 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:24.680728 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:24.686301 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:24.686273 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:24.773818 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:24.773779 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:25.235412 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:25.235386 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:25.236520 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:25.236495 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:52:25.278096 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:25.278042 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:25.294945 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:25.294913 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd"] Oct 30 19:52:25.308641 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:25.308606 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f61e4b-cqszd"] Oct 30 19:52:25.524228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:25.524122 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:25.877206 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:25.877088 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:26.450172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:26.445534 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b92f3caf-4ae7-431a-928a-7812319a3f38" path="/var/lib/kubelet/pods/b92f3caf-4ae7-431a-928a-7812319a3f38/volumes" Oct 30 19:52:26.664095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:26.664046 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:26.873169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:26.873087 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:27.948139 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:27.947614 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:28.127883 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:28.127846 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:28.388169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:28.380253 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:29.093169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:29.085976 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:30.032575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:30.032542 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f"] Oct 30 19:52:30.036237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:30.036214 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6939c6cb-m654f"] Oct 30 19:52:30.315757 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:30.315667 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:30.457543 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:30.457508 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64cb85bc-70a8-4e3a-b28b-1d2a79c825d1" path="/var/lib/kubelet/pods/64cb85bc-70a8-4e3a-b28b-1d2a79c825d1/volumes" Oct 30 19:52:31.388171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:31.385699 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:32.346982 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:32.346945 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:32.498214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:32.498139 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:32.688936 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:32.688853 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:33.518535 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:33.516710 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:34.129976 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:34.129891 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:34.180386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:34.180358 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" Oct 30 19:52:34.181818 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:34.181798 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" Oct 30 19:52:34.336001 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:34.335958 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:34.378047 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:34.378016 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj"] Oct 30 19:52:34.378560 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:34.378532 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" podUID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerName="istio-proxy" containerID="cri-o://2ea5cb7bc2f3a13e656c3b092879fd351a471b9c5e7716760c35c04d945abdff" gracePeriod=30 Oct 30 19:52:34.461345 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:34.461259 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:52:34.461512 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:34.461338 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:52:34.541726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:34.539432 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:34.983182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:34.982504 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:35.442770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:35.442679 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:35.713125 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:35.713085 5091 generic.go:356] "Generic (PLEG): container finished" podID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerID="c213ab747484af8ee780f4f5f1dc2e460f261e47c573f32380995d70f42cfd30" exitCode=0 Oct 30 19:52:35.713618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:35.713389 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" event={"ID":"8b12e993-577a-4dc7-94a0-41c65c7b1fe3","Type":"ContainerDied","Data":"c213ab747484af8ee780f4f5f1dc2e460f261e47c573f32380995d70f42cfd30"} Oct 30 19:52:35.713865 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:35.713843 5091 scope.go:122] "RemoveContainer" containerID="c213ab747484af8ee780f4f5f1dc2e460f261e47c573f32380995d70f42cfd30" Oct 30 19:52:35.716061 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:35.715824 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:52:35.716061 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:35.715910 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:52:35.716061 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:35.715954 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:52:35.979712 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:35.979632 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk"] Oct 30 19:52:35.980802 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:35.980764 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4220b4fa-54d6-4fc4-bc66-24f89798c1bf" containerName="revoke-keys" Oct 30 19:52:35.980912 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:35.980803 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4220b4fa-54d6-4fc4-bc66-24f89798c1bf" containerName="revoke-keys" Oct 30 19:52:35.981073 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:35.981058 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4220b4fa-54d6-4fc4-bc66-24f89798c1bf" containerName="revoke-keys" Oct 30 19:52:35.986282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:35.986256 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j"] Oct 30 19:52:35.986428 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:35.986391 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:35.986549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:35.986518 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" podUID="7ba6caa8-6ed3-40a3-bf9d-43d4008e9339" containerName="payload-pre-processing" containerID="cri-o://fd44830725da38a8c72890eb44fd49871605b67dd7753abc02aeb2a616486b5d" gracePeriod=30 Oct 30 19:52:36.016647 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.014773 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-426bab-openshift-default-dockercfg-sc2kg\"" Oct 30 19:52:36.019701 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.017682 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:36.025940 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.025850 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk"] Oct 30 19:52:36.126048 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.126010 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-workload-socket\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.126213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.126069 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec5bb43-1784-459b-87b0-a7bff9143031-istiod-ca-cert\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.126359 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.126333 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-podinfo\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.126439 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.126418 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-credential-socket\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.126494 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.126471 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-workload-certs\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.126532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.126508 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql298\" (UniqueName: \"kubernetes.io/projected/1ec5bb43-1784-459b-87b0-a7bff9143031-kube-api-access-ql298\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.126567 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.126543 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-envoy\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.126625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.126613 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-token\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.126670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.126657 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-data\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.228632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.227697 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-podinfo\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.228632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.227787 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-credential-socket\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.228632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.227814 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-workload-certs\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.228632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.227848 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql298\" (UniqueName: \"kubernetes.io/projected/1ec5bb43-1784-459b-87b0-a7bff9143031-kube-api-access-ql298\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.228632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.227892 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-envoy\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.228632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.227960 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-token\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.228632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.228012 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-data\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.228632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.228074 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-workload-socket\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.228632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.228110 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec5bb43-1784-459b-87b0-a7bff9143031-istiod-ca-cert\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.230186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.229938 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec5bb43-1784-459b-87b0-a7bff9143031-istiod-ca-cert\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.243561 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.241220 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-credential-socket\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.243561 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.242201 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-workload-certs\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.243561 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.242772 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-data\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.243880 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.243801 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-envoy\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.245786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.245735 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-workload-socket\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.255650 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.255453 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:36.268182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.267175 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-podinfo\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.387164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.385778 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-token\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.628921 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.628648 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql298\" (UniqueName: \"kubernetes.io/projected/1ec5bb43-1784-459b-87b0-a7bff9143031-kube-api-access-ql298\") pod \"e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.641721 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.641571 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:36.651786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.651728 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb"] Oct 30 19:52:36.656482 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.656375 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4f2325-zb9lb"] Oct 30 19:52:36.669815 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.669511 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:36.702274 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.702244 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" Oct 30 19:52:36.801364 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.801328 5091 patch_prober.go:29] interesting pod/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:52:36.801967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.801933 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" podUID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:52:36.820712 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.817577 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:36.820712 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.820004 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" event={"ID":"8b12e993-577a-4dc7-94a0-41c65c7b1fe3","Type":"ContainerStarted","Data":"7b35a104a2c0206cddd13cdd5bad53831309527bfedf7c0908b8201c04b763a6"} Oct 30 19:52:36.822445 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.822417 5091 generic.go:356] "Generic (PLEG): container finished" podID="7ba6caa8-6ed3-40a3-bf9d-43d4008e9339" containerID="fd44830725da38a8c72890eb44fd49871605b67dd7753abc02aeb2a616486b5d" exitCode=0 Oct 30 19:52:36.822592 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.822578 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" event={"ID":"7ba6caa8-6ed3-40a3-bf9d-43d4008e9339","Type":"ContainerDied","Data":"fd44830725da38a8c72890eb44fd49871605b67dd7753abc02aeb2a616486b5d"} Oct 30 19:52:36.822664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.822622 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" event={"ID":"7ba6caa8-6ed3-40a3-bf9d-43d4008e9339","Type":"ContainerDied","Data":"c6753cec7ecb405f55f707565ae4d823a7937b0a3dfc03e227b8ef84aa9b7a55"} Oct 30 19:52:36.822664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.822644 5091 scope.go:122] "RemoveContainer" containerID="fd44830725da38a8c72890eb44fd49871605b67dd7753abc02aeb2a616486b5d" Oct 30 19:52:36.822664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.822656 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j" Oct 30 19:52:36.822838 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.822714 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ba6caa8-6ed3-40a3-bf9d-43d4008e9339-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba6caa8-6ed3-40a3-bf9d-43d4008e9339-plugins-config-volume\") pod \"7ba6caa8-6ed3-40a3-bf9d-43d4008e9339\" (UID: \"7ba6caa8-6ed3-40a3-bf9d-43d4008e9339\") " Oct 30 19:52:36.823213 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.823186 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ba6caa8-6ed3-40a3-bf9d-43d4008e9339-plugins-config-volume" pod "7ba6caa8-6ed3-40a3-bf9d-43d4008e9339" (UID: "7ba6caa8-6ed3-40a3-bf9d-43d4008e9339"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:36.823517 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.823492 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ba6caa8-6ed3-40a3-bf9d-43d4008e9339-kube-api-access-qnklc\" (UniqueName: \"kubernetes.io/projected/7ba6caa8-6ed3-40a3-bf9d-43d4008e9339-kube-api-access-qnklc\") pod \"7ba6caa8-6ed3-40a3-bf9d-43d4008e9339\" (UID: \"7ba6caa8-6ed3-40a3-bf9d-43d4008e9339\") " Oct 30 19:52:36.824337 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.824311 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ba6caa8-6ed3-40a3-bf9d-43d4008e9339-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:36.831334 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.831298 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ba6caa8-6ed3-40a3-bf9d-43d4008e9339-kube-api-access-qnklc" pod "7ba6caa8-6ed3-40a3-bf9d-43d4008e9339" (UID: "7ba6caa8-6ed3-40a3-bf9d-43d4008e9339"). InnerVolumeSpecName "kube-api-access-qnklc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:36.839287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.839263 5091 scope.go:122] "RemoveContainer" containerID="fd44830725da38a8c72890eb44fd49871605b67dd7753abc02aeb2a616486b5d" Oct 30 19:52:36.840320 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:52:36.839677 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd44830725da38a8c72890eb44fd49871605b67dd7753abc02aeb2a616486b5d\": container with ID starting with fd44830725da38a8c72890eb44fd49871605b67dd7753abc02aeb2a616486b5d not found: ID does not exist" containerID="fd44830725da38a8c72890eb44fd49871605b67dd7753abc02aeb2a616486b5d" Oct 30 19:52:36.840478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.840441 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd44830725da38a8c72890eb44fd49871605b67dd7753abc02aeb2a616486b5d"} err="failed to get container status \"fd44830725da38a8c72890eb44fd49871605b67dd7753abc02aeb2a616486b5d\": rpc error: code = NotFound desc = could not find container \"fd44830725da38a8c72890eb44fd49871605b67dd7753abc02aeb2a616486b5d\": container with ID starting with fd44830725da38a8c72890eb44fd49871605b67dd7753abc02aeb2a616486b5d not found: ID does not exist" Oct 30 19:52:36.926254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:36.926231 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnklc\" (UniqueName: \"kubernetes.io/projected/7ba6caa8-6ed3-40a3-bf9d-43d4008e9339-kube-api-access-qnklc\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:37.083623 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.083571 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw"] Oct 30 19:52:37.088450 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.088423 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e3138a2-sjcrw"] Oct 30 19:52:37.185733 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.185682 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:52:37.186039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.186002 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:52:37.186318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.186112 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:52:37.188909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.188872 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk"] Oct 30 19:52:37.189298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.189212 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:37.199517 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.199492 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j"] Oct 30 19:52:37.203299 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.203279 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4f2325-7cc7c79d6c-k9h2j"] Oct 30 19:52:37.381001 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.380548 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:52:37.381001 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.380685 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:52:37.381001 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.380732 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 19:52:37.841679 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.841635 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" event={"ID":"1ec5bb43-1784-459b-87b0-a7bff9143031","Type":"ContainerStarted","Data":"a7002857e0db2a31df496bacaa0b755a2b7029a85cc7edc7108d4cf5469309e2"} Oct 30 19:52:37.842294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.841698 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" event={"ID":"1ec5bb43-1784-459b-87b0-a7bff9143031","Type":"ContainerStarted","Data":"929120d4170d63379b74b8afc6d72801c5c763518bffd93ee0009f40c7e24d4d"} Oct 30 19:52:37.968407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.968340 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" podStartSLOduration=2.968317003 podStartE2EDuration="2.968317003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:37.89152052 +0000 UTC m=+1277.586728480" watchObservedRunningTime="2026-10-30 19:52:37.968317003 +0000 UTC m=+1277.663524964" Oct 30 19:52:37.972855 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.972822 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667"] Oct 30 19:52:37.976636 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.976607 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" podUID="92637c83-dfce-417e-99a2-2f433a1cfdf2" containerName="payload-pre-processing" containerID="cri-o://35d8cb8dd97f042f0ca05a0f0f62a0c058b51d53537c0136166acaec3753daa6" gracePeriod=30 Oct 30 19:52:37.977331 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:37.976231 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:38.155224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.152605 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:38.404225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.403326 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:52:38.404225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.403388 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 19:52:38.503169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.452140 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b8d321-8d7b-469f-994b-12f8a3b4f4b4" path="/var/lib/kubelet/pods/48b8d321-8d7b-469f-994b-12f8a3b4f4b4/volumes" Oct 30 19:52:38.515222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.503488 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ba6caa8-6ed3-40a3-bf9d-43d4008e9339" path="/var/lib/kubelet/pods/7ba6caa8-6ed3-40a3-bf9d-43d4008e9339/volumes" Oct 30 19:52:38.515222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.504199 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8f198c-ad87-454f-b7a4-19c194d4ac34" path="/var/lib/kubelet/pods/cc8f198c-ad87-454f-b7a4-19c194d4ac34/volumes" Oct 30 19:52:38.587601 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.587437 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:38.630203 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.630178 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" Oct 30 19:52:38.667266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.651038 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s"] Oct 30 19:52:38.667266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.665876 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-913351-zm97s"] Oct 30 19:52:38.684270 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.684230 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:38.691565 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.691544 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:38.718529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.718493 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92637c83-dfce-417e-99a2-2f433a1cfdf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92637c83-dfce-417e-99a2-2f433a1cfdf2-plugins-config-volume\") pod \"92637c83-dfce-417e-99a2-2f433a1cfdf2\" (UID: \"92637c83-dfce-417e-99a2-2f433a1cfdf2\") " Oct 30 19:52:38.718650 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.718626 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92637c83-dfce-417e-99a2-2f433a1cfdf2-kube-api-access-ncbwl\" (UniqueName: \"kubernetes.io/projected/92637c83-dfce-417e-99a2-2f433a1cfdf2-kube-api-access-ncbwl\") pod \"92637c83-dfce-417e-99a2-2f433a1cfdf2\" (UID: \"92637c83-dfce-417e-99a2-2f433a1cfdf2\") " Oct 30 19:52:38.718882 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.718858 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92637c83-dfce-417e-99a2-2f433a1cfdf2-plugins-config-volume" pod "92637c83-dfce-417e-99a2-2f433a1cfdf2" (UID: "92637c83-dfce-417e-99a2-2f433a1cfdf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:38.719399 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.719368 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92637c83-dfce-417e-99a2-2f433a1cfdf2-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:38.720967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.720939 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92637c83-dfce-417e-99a2-2f433a1cfdf2-kube-api-access-ncbwl" pod "92637c83-dfce-417e-99a2-2f433a1cfdf2" (UID: "92637c83-dfce-417e-99a2-2f433a1cfdf2"). InnerVolumeSpecName "kube-api-access-ncbwl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:38.760870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.760761 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:38.831968 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.827097 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncbwl\" (UniqueName: \"kubernetes.io/projected/92637c83-dfce-417e-99a2-2f433a1cfdf2-kube-api-access-ncbwl\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:38.884330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.884299 5091 generic.go:356] "Generic (PLEG): container finished" podID="92637c83-dfce-417e-99a2-2f433a1cfdf2" containerID="35d8cb8dd97f042f0ca05a0f0f62a0c058b51d53537c0136166acaec3753daa6" exitCode=0 Oct 30 19:52:38.884818 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.884436 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" event={"ID":"92637c83-dfce-417e-99a2-2f433a1cfdf2","Type":"ContainerDied","Data":"35d8cb8dd97f042f0ca05a0f0f62a0c058b51d53537c0136166acaec3753daa6"} Oct 30 19:52:38.885039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.885011 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" event={"ID":"92637c83-dfce-417e-99a2-2f433a1cfdf2","Type":"ContainerDied","Data":"d1a5b69f837c4598d6014fc7e360c21384476b881e88b7938d74ceff546f2e0e"} Oct 30 19:52:38.885186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.884577 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667" Oct 30 19:52:38.885362 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.885064 5091 scope.go:122] "RemoveContainer" containerID="35d8cb8dd97f042f0ca05a0f0f62a0c058b51d53537c0136166acaec3753daa6" Oct 30 19:52:38.886223 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.886206 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:38.886491 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.886478 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:52:38.900212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.900193 5091 scope.go:122] "RemoveContainer" containerID="35d8cb8dd97f042f0ca05a0f0f62a0c058b51d53537c0136166acaec3753daa6" Oct 30 19:52:38.900505 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:52:38.900486 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35d8cb8dd97f042f0ca05a0f0f62a0c058b51d53537c0136166acaec3753daa6\": container with ID starting with 35d8cb8dd97f042f0ca05a0f0f62a0c058b51d53537c0136166acaec3753daa6 not found: ID does not exist" containerID="35d8cb8dd97f042f0ca05a0f0f62a0c058b51d53537c0136166acaec3753daa6" Oct 30 19:52:38.900558 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.900514 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35d8cb8dd97f042f0ca05a0f0f62a0c058b51d53537c0136166acaec3753daa6"} err="failed to get container status \"35d8cb8dd97f042f0ca05a0f0f62a0c058b51d53537c0136166acaec3753daa6\": rpc error: code = NotFound desc = could not find container \"35d8cb8dd97f042f0ca05a0f0f62a0c058b51d53537c0136166acaec3753daa6\": container with ID starting with 35d8cb8dd97f042f0ca05a0f0f62a0c058b51d53537c0136166acaec3753daa6 not found: ID does not exist" Oct 30 19:52:38.915022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.914998 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:38.983994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.983911 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667"] Oct 30 19:52:38.987857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:38.987805 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-913351-69b8cfd859-pv667"] Oct 30 19:52:39.381446 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.381409 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:52:39.381651 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.381476 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 19:52:39.712567 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.712542 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.841931 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-envoy\") pod \"79cd91a7-7d69-46a4-b213-b36fed53e50b\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.842103 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79cd91a7-7d69-46a4-b213-b36fed53e50b-kube-api-access-v4sdf\" (UniqueName: \"kubernetes.io/projected/79cd91a7-7d69-46a4-b213-b36fed53e50b-kube-api-access-v4sdf\") pod \"79cd91a7-7d69-46a4-b213-b36fed53e50b\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.842241 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79cd91a7-7d69-46a4-b213-b36fed53e50b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cd91a7-7d69-46a4-b213-b36fed53e50b-istiod-ca-cert\") pod \"79cd91a7-7d69-46a4-b213-b36fed53e50b\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.842322 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-podinfo\") pod \"79cd91a7-7d69-46a4-b213-b36fed53e50b\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.842349 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-token\" (UniqueName: \"kubernetes.io/projected/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-token\") pod \"79cd91a7-7d69-46a4-b213-b36fed53e50b\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.842404 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-workload-certs\") pod \"79cd91a7-7d69-46a4-b213-b36fed53e50b\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.842502 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-credential-socket\") pod \"79cd91a7-7d69-46a4-b213-b36fed53e50b\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.842533 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-workload-socket\") pod \"79cd91a7-7d69-46a4-b213-b36fed53e50b\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.842572 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-data\") pod \"79cd91a7-7d69-46a4-b213-b36fed53e50b\" (UID: \"79cd91a7-7d69-46a4-b213-b36fed53e50b\") " Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.842779 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-credential-socket" pod "79cd91a7-7d69-46a4-b213-b36fed53e50b" (UID: "79cd91a7-7d69-46a4-b213-b36fed53e50b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.842892 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-workload-socket" pod "79cd91a7-7d69-46a4-b213-b36fed53e50b" (UID: "79cd91a7-7d69-46a4-b213-b36fed53e50b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.842998 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-data" pod "79cd91a7-7d69-46a4-b213-b36fed53e50b" (UID: "79cd91a7-7d69-46a4-b213-b36fed53e50b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.843200 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-workload-certs" pod "79cd91a7-7d69-46a4-b213-b36fed53e50b" (UID: "79cd91a7-7d69-46a4-b213-b36fed53e50b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.843311 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79cd91a7-7d69-46a4-b213-b36fed53e50b-istiod-ca-cert" pod "79cd91a7-7d69-46a4-b213-b36fed53e50b" (UID: "79cd91a7-7d69-46a4-b213-b36fed53e50b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.843421 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.843455 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.843472 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:39.843863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.843485 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:39.890166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.888908 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cd91a7-7d69-46a4-b213-b36fed53e50b-kube-api-access-v4sdf" pod "79cd91a7-7d69-46a4-b213-b36fed53e50b" (UID: "79cd91a7-7d69-46a4-b213-b36fed53e50b"). InnerVolumeSpecName "kube-api-access-v4sdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:39.892854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.891293 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-token" pod "79cd91a7-7d69-46a4-b213-b36fed53e50b" (UID: "79cd91a7-7d69-46a4-b213-b36fed53e50b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:39.892854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.891452 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-podinfo" pod "79cd91a7-7d69-46a4-b213-b36fed53e50b" (UID: "79cd91a7-7d69-46a4-b213-b36fed53e50b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:52:39.892854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.892695 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-envoy" pod "79cd91a7-7d69-46a4-b213-b36fed53e50b" (UID: "79cd91a7-7d69-46a4-b213-b36fed53e50b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:39.910497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.910467 5091 generic.go:356] "Generic (PLEG): container finished" podID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerID="2ea5cb7bc2f3a13e656c3b092879fd351a471b9c5e7716760c35c04d945abdff" exitCode=0 Oct 30 19:52:39.910624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.910522 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" Oct 30 19:52:39.910624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.910547 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" event={"ID":"79cd91a7-7d69-46a4-b213-b36fed53e50b","Type":"ContainerDied","Data":"2ea5cb7bc2f3a13e656c3b092879fd351a471b9c5e7716760c35c04d945abdff"} Oct 30 19:52:39.910624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.910606 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj" event={"ID":"79cd91a7-7d69-46a4-b213-b36fed53e50b","Type":"ContainerDied","Data":"881218d92faff6a3bd86e343355a669d8f850c574eba4f3c64c70f37d32f7180"} Oct 30 19:52:39.910784 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.910630 5091 scope.go:122] "RemoveContainer" containerID="2ea5cb7bc2f3a13e656c3b092879fd351a471b9c5e7716760c35c04d945abdff" Oct 30 19:52:39.926394 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.926375 5091 scope.go:122] "RemoveContainer" containerID="b573b8cfcb3098eec6c22107f9d9a1bb8d6b8a941996207387539fde58fe554d" Oct 30 19:52:39.937080 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.937058 5091 scope.go:122] "RemoveContainer" containerID="2ea5cb7bc2f3a13e656c3b092879fd351a471b9c5e7716760c35c04d945abdff" Oct 30 19:52:39.937382 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:52:39.937356 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ea5cb7bc2f3a13e656c3b092879fd351a471b9c5e7716760c35c04d945abdff\": container with ID starting with 2ea5cb7bc2f3a13e656c3b092879fd351a471b9c5e7716760c35c04d945abdff not found: ID does not exist" containerID="2ea5cb7bc2f3a13e656c3b092879fd351a471b9c5e7716760c35c04d945abdff" Oct 30 19:52:39.937468 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.937388 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ea5cb7bc2f3a13e656c3b092879fd351a471b9c5e7716760c35c04d945abdff"} err="failed to get container status \"2ea5cb7bc2f3a13e656c3b092879fd351a471b9c5e7716760c35c04d945abdff\": rpc error: code = NotFound desc = could not find container \"2ea5cb7bc2f3a13e656c3b092879fd351a471b9c5e7716760c35c04d945abdff\": container with ID starting with 2ea5cb7bc2f3a13e656c3b092879fd351a471b9c5e7716760c35c04d945abdff not found: ID does not exist" Oct 30 19:52:39.937468 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.937407 5091 scope.go:122] "RemoveContainer" containerID="b573b8cfcb3098eec6c22107f9d9a1bb8d6b8a941996207387539fde58fe554d" Oct 30 19:52:39.937685 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:52:39.937669 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b573b8cfcb3098eec6c22107f9d9a1bb8d6b8a941996207387539fde58fe554d\": container with ID starting with b573b8cfcb3098eec6c22107f9d9a1bb8d6b8a941996207387539fde58fe554d not found: ID does not exist" containerID="b573b8cfcb3098eec6c22107f9d9a1bb8d6b8a941996207387539fde58fe554d" Oct 30 19:52:39.937732 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.937694 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b573b8cfcb3098eec6c22107f9d9a1bb8d6b8a941996207387539fde58fe554d"} err="failed to get container status \"b573b8cfcb3098eec6c22107f9d9a1bb8d6b8a941996207387539fde58fe554d\": rpc error: code = NotFound desc = could not find container \"b573b8cfcb3098eec6c22107f9d9a1bb8d6b8a941996207387539fde58fe554d\": container with ID starting with b573b8cfcb3098eec6c22107f9d9a1bb8d6b8a941996207387539fde58fe554d not found: ID does not exist" Oct 30 19:52:39.944357 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.944330 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:39.944357 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.944359 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4sdf\" (UniqueName: \"kubernetes.io/projected/79cd91a7-7d69-46a4-b213-b36fed53e50b-kube-api-access-v4sdf\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:39.944547 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.944369 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79cd91a7-7d69-46a4-b213-b36fed53e50b-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:39.944547 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.944379 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:39.944547 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.944387 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79cd91a7-7d69-46a4-b213-b36fed53e50b-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:39.944804 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.944787 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj"] Oct 30 19:52:39.978182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:39.978122 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f61e4b-openshift-default-787f77d447-5dkcj"] Oct 30 19:52:40.277610 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:40.277578 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:40.380587 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:40.380490 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:52:40.380835 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:40.380597 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 19:52:40.484020 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:40.483971 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6769a587-f6fd-4b20-bebb-273b8f774b19" path="/var/lib/kubelet/pods/6769a587-f6fd-4b20-bebb-273b8f774b19/volumes" Oct 30 19:52:40.484831 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:40.484804 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79cd91a7-7d69-46a4-b213-b36fed53e50b" path="/var/lib/kubelet/pods/79cd91a7-7d69-46a4-b213-b36fed53e50b/volumes" Oct 30 19:52:40.485535 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:40.485509 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92637c83-dfce-417e-99a2-2f433a1cfdf2" path="/var/lib/kubelet/pods/92637c83-dfce-417e-99a2-2f433a1cfdf2/volumes" Oct 30 19:52:40.495560 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:40.495529 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:40.714251 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:40.707871 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:41.380887 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:41.380856 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:52:41.381464 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:41.381117 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 19:52:41.521167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:41.518234 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:42.081917 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.081877 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:42.087797 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.087760 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn"] Oct 30 19:52:42.089938 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.089910 5091 cpu_manager.go:414] "containerMap: removing container" podUID="92637c83-dfce-417e-99a2-2f433a1cfdf2" containerName="payload-pre-processing" Oct 30 19:52:42.090077 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.089951 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92637c83-dfce-417e-99a2-2f433a1cfdf2" containerName="payload-pre-processing" Oct 30 19:52:42.090077 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.089984 5091 cpu_manager.go:414] "containerMap: removing container" podUID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerName="istio-proxy" Oct 30 19:52:42.090077 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.089993 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerName="istio-proxy" Oct 30 19:52:42.090269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.090124 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7ba6caa8-6ed3-40a3-bf9d-43d4008e9339" containerName="payload-pre-processing" Oct 30 19:52:42.090269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.090134 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ba6caa8-6ed3-40a3-bf9d-43d4008e9339" containerName="payload-pre-processing" Oct 30 19:52:42.090589 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.090569 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerName="istio-proxy" Oct 30 19:52:42.090688 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.090610 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ba6caa8-6ed3-40a3-bf9d-43d4008e9339" containerName="payload-pre-processing" Oct 30 19:52:42.090688 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.090664 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerName="istio-proxy" Oct 30 19:52:42.090797 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.090684 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="92637c83-dfce-417e-99a2-2f433a1cfdf2" containerName="payload-pre-processing" Oct 30 19:52:42.109841 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.109715 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn"] Oct 30 19:52:42.109991 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.109852 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" Oct 30 19:52:42.221176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.196501 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltkjz\" (UniqueName: \"kubernetes.io/projected/4c812a3d-79c6-4060-be64-0da4dd1aa4d9-kube-api-access-ltkjz\") pod \"payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn\" (UID: \"4c812a3d-79c6-4060-be64-0da4dd1aa4d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" Oct 30 19:52:42.221653 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.221619 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c812a3d-79c6-4060-be64-0da4dd1aa4d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn\" (UID: \"4c812a3d-79c6-4060-be64-0da4dd1aa4d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" Oct 30 19:52:42.228353 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.212677 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-426bab\"" Oct 30 19:52:42.232171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.212580 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-426bab-dockercfg-2hkjt\"" Oct 30 19:52:42.348216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.335087 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c812a3d-79c6-4060-be64-0da4dd1aa4d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn\" (UID: \"4c812a3d-79c6-4060-be64-0da4dd1aa4d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" Oct 30 19:52:42.348820 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.348770 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltkjz\" (UniqueName: \"kubernetes.io/projected/4c812a3d-79c6-4060-be64-0da4dd1aa4d9-kube-api-access-ltkjz\") pod \"payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn\" (UID: \"4c812a3d-79c6-4060-be64-0da4dd1aa4d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" Oct 30 19:52:42.373247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.349634 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:42.393595 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.343985 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c812a3d-79c6-4060-be64-0da4dd1aa4d9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn\" (UID: \"4c812a3d-79c6-4060-be64-0da4dd1aa4d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" Oct 30 19:52:42.417241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.384033 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" start-of-body= Oct 30 19:52:42.417508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.417477 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.145:15021/healthz/ready\": dial tcp 10.132.0.145:15021: connect: connection refused" Oct 30 19:52:42.524134 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.524106 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltkjz\" (UniqueName: \"kubernetes.io/projected/4c812a3d-79c6-4060-be64-0da4dd1aa4d9-kube-api-access-ltkjz\") pod \"payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn\" (UID: \"4c812a3d-79c6-4060-be64-0da4dd1aa4d9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" Oct 30 19:52:42.736210 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.736119 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" Oct 30 19:52:42.964174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:42.959864 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:43.094182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:43.081175 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn"] Oct 30 19:52:43.117921 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:43.117886 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" event={"ID":"4c812a3d-79c6-4060-be64-0da4dd1aa4d9","Type":"ContainerStarted","Data":"93210eff2fc8f6e638738400c584bb616ee0e3ecb626251b56dd8cab93489e07"} Oct 30 19:52:43.384631 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:43.384550 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:52:43.385033 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:43.385015 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:52:43.385794 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:43.385774 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:52:43.981502 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:43.981468 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:44.127555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:44.127514 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" event={"ID":"4c812a3d-79c6-4060-be64-0da4dd1aa4d9","Type":"ContainerStarted","Data":"4efd3a4198e1878507cfab6c9ef8bc3f5963d44c9585607d59af258fa55830ab"} Oct 30 19:52:44.127730 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:44.127677 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" Oct 30 19:52:44.150791 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:44.150753 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7"] Oct 30 19:52:44.172777 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:44.172715 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" podStartSLOduration=2.172696509 podStartE2EDuration="2.172696509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:44.165513069 +0000 UTC m=+1283.860721028" watchObservedRunningTime="2026-10-30 19:52:44.172696509 +0000 UTC m=+1283.867904468" Oct 30 19:52:44.330169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:44.327429 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:45.475698 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:45.475660 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:45.978398 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:52:45.978358 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:52:46.360880 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:46.360849 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:46.527979 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:46.527903 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:46.708992 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:46.708914 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq"] Oct 30 19:52:46.709180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:46.709123 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" podUID="111283d7-9f88-4151-beee-283607f0e13b" containerName="istio-proxy" containerID="cri-o://a40e8721d7da58ff2741d9694c332d0ff9ee53afbd898303adeee76d76ea9d7e" gracePeriod=30 Oct 30 19:52:46.838818 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:46.838773 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:47.283169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.283107 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr"] Oct 30 19:52:47.283892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.283874 5091 cpu_manager.go:414] "containerMap: removing container" podUID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerName="istio-proxy" Oct 30 19:52:47.283949 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.283895 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79cd91a7-7d69-46a4-b213-b36fed53e50b" containerName="istio-proxy" Oct 30 19:52:47.288824 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.288803 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" Oct 30 19:52:47.310327 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.310255 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:52:47.310327 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.310292 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:52:47.329000 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.328967 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr"] Oct 30 19:52:47.430631 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.430598 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpn5w\" (UniqueName: \"kubernetes.io/projected/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-kube-api-access-zpn5w\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr\" (UID: \"63c5e14e-4ed6-44a7-901a-8f78bbe593b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" Oct 30 19:52:47.430824 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.430654 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr\" (UID: \"63c5e14e-4ed6-44a7-901a-8f78bbe593b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" Oct 30 19:52:47.430824 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.430784 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr\" (UID: \"63c5e14e-4ed6-44a7-901a-8f78bbe593b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" Oct 30 19:52:47.532380 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.532333 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpn5w\" (UniqueName: \"kubernetes.io/projected/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-kube-api-access-zpn5w\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr\" (UID: \"63c5e14e-4ed6-44a7-901a-8f78bbe593b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" Oct 30 19:52:47.532863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.532425 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr\" (UID: \"63c5e14e-4ed6-44a7-901a-8f78bbe593b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" Oct 30 19:52:47.532863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.532473 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr\" (UID: \"63c5e14e-4ed6-44a7-901a-8f78bbe593b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" Oct 30 19:52:47.533241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.533188 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr\" (UID: \"63c5e14e-4ed6-44a7-901a-8f78bbe593b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" Oct 30 19:52:47.534803 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.534783 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr\" (UID: \"63c5e14e-4ed6-44a7-901a-8f78bbe593b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" Oct 30 19:52:47.554539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.554509 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpn5w\" (UniqueName: \"kubernetes.io/projected/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-kube-api-access-zpn5w\") pod \"maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr\" (UID: \"63c5e14e-4ed6-44a7-901a-8f78bbe593b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" Oct 30 19:52:47.605891 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.605865 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" Oct 30 19:52:47.802326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.802234 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:47.962305 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:47.962277 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr"] Oct 30 19:52:48.163930 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:48.163897 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" event={"ID":"63c5e14e-4ed6-44a7-901a-8f78bbe593b5","Type":"ContainerStarted","Data":"a3116af6d144cc0c4cf2a5573e7425f3ea09ccefa4ad2ab00835bd637eaad0b3"} Oct 30 19:52:48.885756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:48.885670 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:49.177612 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:49.177579 5091 generic.go:356] "Generic (PLEG): container finished" podID="63c5e14e-4ed6-44a7-901a-8f78bbe593b5" containerID="bcdc8b20752c0a4838b6dd7f8cf85fa48cd70fd6a11b2551e9081b39a4496a1b" exitCode=0 Oct 30 19:52:49.177837 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:49.177663 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" event={"ID":"63c5e14e-4ed6-44a7-901a-8f78bbe593b5","Type":"ContainerDied","Data":"bcdc8b20752c0a4838b6dd7f8cf85fa48cd70fd6a11b2551e9081b39a4496a1b"} Oct 30 19:52:49.794950 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:49.794908 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:50.128220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:50.126803 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:50.379808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:50.379745 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" Oct 30 19:52:50.509429 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:50.509399 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-kube-api-access-zpn5w\" (UniqueName: \"kubernetes.io/projected/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-kube-api-access-zpn5w\") pod \"63c5e14e-4ed6-44a7-901a-8f78bbe593b5\" (UID: \"63c5e14e-4ed6-44a7-901a-8f78bbe593b5\") " Oct 30 19:52:50.509612 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:50.509470 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-maas-api-openshift-ca\") pod \"63c5e14e-4ed6-44a7-901a-8f78bbe593b5\" (UID: \"63c5e14e-4ed6-44a7-901a-8f78bbe593b5\") " Oct 30 19:52:50.509612 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:50.509603 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-maas-api-xks-ca\") pod \"63c5e14e-4ed6-44a7-901a-8f78bbe593b5\" (UID: \"63c5e14e-4ed6-44a7-901a-8f78bbe593b5\") " Oct 30 19:52:50.509824 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:50.509799 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-maas-api-openshift-ca" pod "63c5e14e-4ed6-44a7-901a-8f78bbe593b5" (UID: "63c5e14e-4ed6-44a7-901a-8f78bbe593b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:50.510122 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:50.510103 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:50.511666 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:50.511645 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-maas-api-xks-ca" pod "63c5e14e-4ed6-44a7-901a-8f78bbe593b5" (UID: "63c5e14e-4ed6-44a7-901a-8f78bbe593b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:52:50.511734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:50.511717 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-kube-api-access-zpn5w" pod "63c5e14e-4ed6-44a7-901a-8f78bbe593b5" (UID: "63c5e14e-4ed6-44a7-901a-8f78bbe593b5"). InnerVolumeSpecName "kube-api-access-zpn5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:50.611772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:50.611728 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:50.611772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:50.611766 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpn5w\" (UniqueName: \"kubernetes.io/projected/63c5e14e-4ed6-44a7-901a-8f78bbe593b5-kube-api-access-zpn5w\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:51.196661 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:51.196633 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" Oct 30 19:52:51.197083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:51.196673 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr" event={"ID":"63c5e14e-4ed6-44a7-901a-8f78bbe593b5","Type":"ContainerDied","Data":"a3116af6d144cc0c4cf2a5573e7425f3ea09ccefa4ad2ab00835bd637eaad0b3"} Oct 30 19:52:51.197083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:51.196725 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3116af6d144cc0c4cf2a5573e7425f3ea09ccefa4ad2ab00835bd637eaad0b3" Oct 30 19:52:51.380173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:51.377942 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:51.611968 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:51.611940 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:52:51.612308 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:51.612276 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" podUID="111283d7-9f88-4151-beee-283607f0e13b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:52:51.862242 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:51.860483 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:52.075180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.075130 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:52.219921 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.219887 5091 generic.go:356] "Generic (PLEG): container finished" podID="111283d7-9f88-4151-beee-283607f0e13b" containerID="a40e8721d7da58ff2741d9694c332d0ff9ee53afbd898303adeee76d76ea9d7e" exitCode=0 Oct 30 19:52:52.220380 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.220040 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" event={"ID":"111283d7-9f88-4151-beee-283607f0e13b","Type":"ContainerDied","Data":"a40e8721d7da58ff2741d9694c332d0ff9ee53afbd898303adeee76d76ea9d7e"} Oct 30 19:52:52.220380 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.220087 5091 scope.go:122] "RemoveContainer" containerID="c6dc220bc9f42c5e2bac62efc5eda85423de21fecc13cb7c8d66fda483812b67" Oct 30 19:52:52.507342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.507241 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:52.611170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.610328 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:52:52.715164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.715105 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-workload-socket\") pod \"111283d7-9f88-4151-beee-283607f0e13b\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " Oct 30 19:52:52.715465 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.715444 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-workload-certs\") pod \"111283d7-9f88-4151-beee-283607f0e13b\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " Oct 30 19:52:52.715586 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.715572 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-istio-data\") pod \"111283d7-9f88-4151-beee-283607f0e13b\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " Oct 30 19:52:52.715709 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.715694 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/111283d7-9f88-4151-beee-283607f0e13b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111283d7-9f88-4151-beee-283607f0e13b-istiod-ca-cert\") pod \"111283d7-9f88-4151-beee-283607f0e13b\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " Oct 30 19:52:52.715846 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.715831 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/111283d7-9f88-4151-beee-283607f0e13b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111283d7-9f88-4151-beee-283607f0e13b-istio-podinfo\") pod \"111283d7-9f88-4151-beee-283607f0e13b\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " Oct 30 19:52:52.715969 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.715954 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111283d7-9f88-4151-beee-283607f0e13b-istio-token\" (UniqueName: \"kubernetes.io/projected/111283d7-9f88-4151-beee-283607f0e13b-istio-token\") pod \"111283d7-9f88-4151-beee-283607f0e13b\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " Oct 30 19:52:52.716284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.716254 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-credential-socket\") pod \"111283d7-9f88-4151-beee-283607f0e13b\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " Oct 30 19:52:52.716537 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.716509 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/111283d7-9f88-4151-beee-283607f0e13b-kube-api-access-8xjlx\" (UniqueName: \"kubernetes.io/projected/111283d7-9f88-4151-beee-283607f0e13b-kube-api-access-8xjlx\") pod \"111283d7-9f88-4151-beee-283607f0e13b\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " Oct 30 19:52:52.725404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.725328 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-istio-envoy\") pod \"111283d7-9f88-4151-beee-283607f0e13b\" (UID: \"111283d7-9f88-4151-beee-283607f0e13b\") " Oct 30 19:52:52.725566 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.717661 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-workload-certs" pod "111283d7-9f88-4151-beee-283607f0e13b" (UID: "111283d7-9f88-4151-beee-283607f0e13b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:52.725645 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.717835 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-workload-socket" pod "111283d7-9f88-4151-beee-283607f0e13b" (UID: "111283d7-9f88-4151-beee-283607f0e13b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:52.725645 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.724358 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111283d7-9f88-4151-beee-283607f0e13b-istiod-ca-cert" pod "111283d7-9f88-4151-beee-283607f0e13b" (UID: "111283d7-9f88-4151-beee-283607f0e13b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:52:52.725645 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.724696 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-istio-data" pod "111283d7-9f88-4151-beee-283607f0e13b" (UID: "111283d7-9f88-4151-beee-283607f0e13b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:52.725645 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.725531 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-credential-socket" pod "111283d7-9f88-4151-beee-283607f0e13b" (UID: "111283d7-9f88-4151-beee-283607f0e13b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:52.726646 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.726620 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:52.726786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.726770 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:52.726883 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.726871 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:52.726966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.726955 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:52.727046 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.727035 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/111283d7-9f88-4151-beee-283607f0e13b-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:52.731084 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.731040 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-istio-envoy" pod "111283d7-9f88-4151-beee-283607f0e13b" (UID: "111283d7-9f88-4151-beee-283607f0e13b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:52.731605 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.731575 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/111283d7-9f88-4151-beee-283607f0e13b-istio-podinfo" pod "111283d7-9f88-4151-beee-283607f0e13b" (UID: "111283d7-9f88-4151-beee-283607f0e13b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:52:52.731767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.731738 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111283d7-9f88-4151-beee-283607f0e13b-istio-token" pod "111283d7-9f88-4151-beee-283607f0e13b" (UID: "111283d7-9f88-4151-beee-283607f0e13b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:52.731837 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.731788 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111283d7-9f88-4151-beee-283607f0e13b-kube-api-access-8xjlx" pod "111283d7-9f88-4151-beee-283607f0e13b" (UID: "111283d7-9f88-4151-beee-283607f0e13b"). InnerVolumeSpecName "kube-api-access-8xjlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:52.828496 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.828404 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/111283d7-9f88-4151-beee-283607f0e13b-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:52.828496 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.828443 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/111283d7-9f88-4151-beee-283607f0e13b-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:52.828496 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.828458 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xjlx\" (UniqueName: \"kubernetes.io/projected/111283d7-9f88-4151-beee-283607f0e13b-kube-api-access-8xjlx\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:52.828496 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:52.828472 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/111283d7-9f88-4151-beee-283607f0e13b-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:53.335518 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:53.335468 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" event={"ID":"111283d7-9f88-4151-beee-283607f0e13b","Type":"ContainerDied","Data":"1221847e7f54fc546f72fd434fb8aef0e82b26f0ff7dca51c80a9e0d5c3f0866"} Oct 30 19:52:53.335994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:53.335600 5091 scope.go:122] "RemoveContainer" containerID="a40e8721d7da58ff2741d9694c332d0ff9ee53afbd898303adeee76d76ea9d7e" Oct 30 19:52:53.335994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:53.335820 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq" Oct 30 19:52:53.449307 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:53.449260 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:53.504255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:53.504207 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq"] Oct 30 19:52:53.511286 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:53.511256 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-4f2325-openshift-default-8569d95fc5-sz2sq"] Oct 30 19:52:54.524169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:54.510912 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:54.639027 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:54.638993 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="111283d7-9f88-4151-beee-283607f0e13b" path="/var/lib/kubelet/pods/111283d7-9f88-4151-beee-283607f0e13b/volumes" Oct 30 19:52:54.711670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:54.711608 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:54.970460 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:54.970366 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:55.174179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:55.173373 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" Oct 30 19:52:55.616162 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:55.616116 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:55.869985 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:55.869726 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:56.077171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:56.076697 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:57.122306 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.122244 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8"] Oct 30 19:52:57.124054 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.123954 5091 cpu_manager.go:414] "containerMap: removing container" podUID="111283d7-9f88-4151-beee-283607f0e13b" containerName="istio-proxy" Oct 30 19:52:57.124733 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.124708 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111283d7-9f88-4151-beee-283607f0e13b" containerName="istio-proxy" Oct 30 19:52:57.124918 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.124905 5091 cpu_manager.go:414] "containerMap: removing container" podUID="63c5e14e-4ed6-44a7-901a-8f78bbe593b5" containerName="revoke-keys" Oct 30 19:52:57.125019 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.125006 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c5e14e-4ed6-44a7-901a-8f78bbe593b5" containerName="revoke-keys" Oct 30 19:52:57.125129 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.125117 5091 cpu_manager.go:414] "containerMap: removing container" podUID="111283d7-9f88-4151-beee-283607f0e13b" containerName="istio-proxy" Oct 30 19:52:57.125600 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.125582 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="111283d7-9f88-4151-beee-283607f0e13b" containerName="istio-proxy" Oct 30 19:52:57.125893 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.125878 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="111283d7-9f88-4151-beee-283607f0e13b" containerName="istio-proxy" Oct 30 19:52:57.126026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.126014 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="111283d7-9f88-4151-beee-283607f0e13b" containerName="istio-proxy" Oct 30 19:52:57.126125 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.126113 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c5e14e-4ed6-44a7-901a-8f78bbe593b5" containerName="revoke-keys" Oct 30 19:52:57.181170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.169681 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:57.181170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.174113 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8"] Oct 30 19:52:57.181170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.174184 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b"] Oct 30 19:52:57.231989 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.231958 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" Oct 30 19:52:57.239168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.237622 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a44dd5b-7eb5-418c-a1ee-99d44fc907d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8\" (UID: \"2a44dd5b-7eb5-418c-a1ee-99d44fc907d1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" Oct 30 19:52:57.239168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.237750 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnnwv\" (UniqueName: \"kubernetes.io/projected/2a44dd5b-7eb5-418c-a1ee-99d44fc907d1-kube-api-access-vnnwv\") pod \"payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8\" (UID: \"2a44dd5b-7eb5-418c-a1ee-99d44fc907d1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" Oct 30 19:52:57.240802 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.240780 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" Oct 30 19:52:57.252169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.247078 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b"] Oct 30 19:52:57.309456 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.253529 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ee4af9-dockercfg-nw5nq\"" Oct 30 19:52:57.611247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.367965 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a44dd5b-7eb5-418c-a1ee-99d44fc907d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8\" (UID: \"2a44dd5b-7eb5-418c-a1ee-99d44fc907d1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" Oct 30 19:52:57.611247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.606826 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnnwv\" (UniqueName: \"kubernetes.io/projected/2a44dd5b-7eb5-418c-a1ee-99d44fc907d1-kube-api-access-vnnwv\") pod \"payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8\" (UID: \"2a44dd5b-7eb5-418c-a1ee-99d44fc907d1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" Oct 30 19:52:57.611247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.393562 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a44dd5b-7eb5-418c-a1ee-99d44fc907d1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8\" (UID: \"2a44dd5b-7eb5-418c-a1ee-99d44fc907d1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" Oct 30 19:52:57.611247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.465477 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:57.611247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.391868 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ee4af9\"" Oct 30 19:52:57.782177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.776211 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21207314-dcb4-4ae2-b76c-54781df4f643-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b\" (UID: \"21207314-dcb4-4ae2-b76c-54781df4f643\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" Oct 30 19:52:57.782177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.776607 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gnvw\" (UniqueName: \"kubernetes.io/projected/21207314-dcb4-4ae2-b76c-54781df4f643-kube-api-access-7gnvw\") pod \"payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b\" (UID: \"21207314-dcb4-4ae2-b76c-54781df4f643\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" Oct 30 19:52:57.837334 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.828300 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:57.840204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.839879 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnnwv\" (UniqueName: \"kubernetes.io/projected/2a44dd5b-7eb5-418c-a1ee-99d44fc907d1-kube-api-access-vnnwv\") pod \"payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8\" (UID: \"2a44dd5b-7eb5-418c-a1ee-99d44fc907d1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" Oct 30 19:52:57.878574 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.878487 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21207314-dcb4-4ae2-b76c-54781df4f643-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b\" (UID: \"21207314-dcb4-4ae2-b76c-54781df4f643\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" Oct 30 19:52:57.895164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.879027 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gnvw\" (UniqueName: \"kubernetes.io/projected/21207314-dcb4-4ae2-b76c-54781df4f643-kube-api-access-7gnvw\") pod \"payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b\" (UID: \"21207314-dcb4-4ae2-b76c-54781df4f643\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" Oct 30 19:52:57.895879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.880394 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21207314-dcb4-4ae2-b76c-54781df4f643-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b\" (UID: \"21207314-dcb4-4ae2-b76c-54781df4f643\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" Oct 30 19:52:57.920241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.920218 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" Oct 30 19:52:57.921995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:57.921891 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gnvw\" (UniqueName: \"kubernetes.io/projected/21207314-dcb4-4ae2-b76c-54781df4f643-kube-api-access-7gnvw\") pod \"payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b\" (UID: \"21207314-dcb4-4ae2-b76c-54781df4f643\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" Oct 30 19:52:58.122409 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:58.118558 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" Oct 30 19:52:58.423372 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:58.423341 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8"] Oct 30 19:52:58.573894 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:58.573862 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b"] Oct 30 19:52:58.842120 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:58.842079 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" event={"ID":"21207314-dcb4-4ae2-b76c-54781df4f643","Type":"ContainerStarted","Data":"4dfb06fca71bbf96c83cb3ddcceeea0f0e230b9aa513b53371056510db36d491"} Oct 30 19:52:58.842293 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:58.842183 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" event={"ID":"21207314-dcb4-4ae2-b76c-54781df4f643","Type":"ContainerStarted","Data":"b2a5ec01d8c42dc1962d079011d96f5c2ac688c7de308c28ba53ffee8df07c4b"} Oct 30 19:52:58.842677 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:58.842641 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" Oct 30 19:52:58.844765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:58.844739 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" event={"ID":"2a44dd5b-7eb5-418c-a1ee-99d44fc907d1","Type":"ContainerStarted","Data":"a9877f9d2cdc3ff29375ca3107c1c6f042630d3fef63d85b1dfe09858e0e51bf"} Oct 30 19:52:58.844884 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:58.844793 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" Oct 30 19:52:58.844884 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:58.844820 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" event={"ID":"2a44dd5b-7eb5-418c-a1ee-99d44fc907d1","Type":"ContainerStarted","Data":"701f7e3205f1b5bbc0073a42f08db646898fbc30795921058b84d7be8813462a"} Oct 30 19:52:58.887996 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:58.887925 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" podStartSLOduration=1.887906291 podStartE2EDuration="1.887906291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:58.88138818 +0000 UTC m=+1298.576596138" watchObservedRunningTime="2026-10-30 19:52:58.887906291 +0000 UTC m=+1298.583114254" Oct 30 19:52:59.120125 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.120001 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:59.174640 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.174484 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" podUID="5faa3220-d981-43af-a6d2-19d841026802" containerName="main" containerID="cri-o://e11344a9458299258f6751fbb4bb79a1e8700fc25ff755dafbf588c7818384d8" gracePeriod=45 Oct 30 19:52:59.525109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.525074 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:52:59.560527 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.560465 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" podStartSLOduration=2.560444745 podStartE2EDuration="2.560444745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:52:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:52:58.936164056 +0000 UTC m=+1298.631372015" watchObservedRunningTime="2026-10-30 19:52:59.560444745 +0000 UTC m=+1299.255652704" Oct 30 19:52:59.646330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.646294 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-tmp-dir\") pod \"5faa3220-d981-43af-a6d2-19d841026802\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " Oct 30 19:52:59.646522 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.646431 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-model-cache\") pod \"5faa3220-d981-43af-a6d2-19d841026802\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " Oct 30 19:52:59.646595 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.646529 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5faa3220-d981-43af-a6d2-19d841026802-tls-certs\" (UniqueName: \"kubernetes.io/secret/5faa3220-d981-43af-a6d2-19d841026802-tls-certs\") pod \"5faa3220-d981-43af-a6d2-19d841026802\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " Oct 30 19:52:59.646652 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.646591 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-dshm\") pod \"5faa3220-d981-43af-a6d2-19d841026802\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " Oct 30 19:52:59.646703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.646660 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-home\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-home\") pod \"5faa3220-d981-43af-a6d2-19d841026802\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " Oct 30 19:52:59.648016 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.646772 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5faa3220-d981-43af-a6d2-19d841026802-kube-api-access-j8tqx\" (UniqueName: \"kubernetes.io/projected/5faa3220-d981-43af-a6d2-19d841026802-kube-api-access-j8tqx\") pod \"5faa3220-d981-43af-a6d2-19d841026802\" (UID: \"5faa3220-d981-43af-a6d2-19d841026802\") " Oct 30 19:52:59.648016 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.647127 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-tmp-dir" pod "5faa3220-d981-43af-a6d2-19d841026802" (UID: "5faa3220-d981-43af-a6d2-19d841026802"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:59.648016 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.647434 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-tmp-dir\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:59.648016 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.647684 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-home" pod "5faa3220-d981-43af-a6d2-19d841026802" (UID: "5faa3220-d981-43af-a6d2-19d841026802"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:59.648016 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.647987 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-model-cache" pod "5faa3220-d981-43af-a6d2-19d841026802" (UID: "5faa3220-d981-43af-a6d2-19d841026802"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:59.650292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.650269 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5faa3220-d981-43af-a6d2-19d841026802-kube-api-access-j8tqx" pod "5faa3220-d981-43af-a6d2-19d841026802" (UID: "5faa3220-d981-43af-a6d2-19d841026802"). InnerVolumeSpecName "kube-api-access-j8tqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:52:59.650709 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.650675 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-dshm" pod "5faa3220-d981-43af-a6d2-19d841026802" (UID: "5faa3220-d981-43af-a6d2-19d841026802"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:52:59.650819 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.650756 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5faa3220-d981-43af-a6d2-19d841026802-tls-certs" pod "5faa3220-d981-43af-a6d2-19d841026802" (UID: "5faa3220-d981-43af-a6d2-19d841026802"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:52:59.751267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.749897 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-model-cache\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:59.751267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.749933 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5faa3220-d981-43af-a6d2-19d841026802-tls-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:59.751267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.749947 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-dshm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:59.751267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.749961 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5faa3220-d981-43af-a6d2-19d841026802-home\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:59.751267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.749975 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8tqx\" (UniqueName: \"kubernetes.io/projected/5faa3220-d981-43af-a6d2-19d841026802-kube-api-access-j8tqx\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:52:59.852322 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.852216 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l"] Oct 30 19:52:59.852743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.852708 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" podUID="be3e710b-55f4-4cd6-8872-f7db69725afc" containerName="payload-processing" containerID="cri-o://8227ceee5ca652df67bf3a3118fa741fde4208cedeb130146cbda24054463b56" gracePeriod=30 Oct 30 19:52:59.864105 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.860780 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:52:59.874110 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.874076 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff"] Oct 30 19:52:59.874521 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.874488 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" podUID="433ca046-a713-4f04-a152-027d1bdeee27" containerName="payload-pre-processing" containerID="cri-o://69fa1c39bffae983c8a2940dbcaa62afe338ca72593c7b4c4e53187a38a3285b" gracePeriod=30 Oct 30 19:52:59.912863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.910689 5091 generic.go:356] "Generic (PLEG): container finished" podID="5faa3220-d981-43af-a6d2-19d841026802" containerID="e11344a9458299258f6751fbb4bb79a1e8700fc25ff755dafbf588c7818384d8" exitCode=0 Oct 30 19:52:59.912863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.911662 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" Oct 30 19:52:59.912863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.911886 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" event={"ID":"5faa3220-d981-43af-a6d2-19d841026802","Type":"ContainerDied","Data":"e11344a9458299258f6751fbb4bb79a1e8700fc25ff755dafbf588c7818384d8"} Oct 30 19:52:59.912863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.911924 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7" event={"ID":"5faa3220-d981-43af-a6d2-19d841026802","Type":"ContainerDied","Data":"ea950d6fa54054303b7bd6e6b7d7959c3638a4d736f8a5f4a82da173fa8f0b2a"} Oct 30 19:52:59.912863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:52:59.911945 5091 scope.go:122] "RemoveContainer" containerID="e11344a9458299258f6751fbb4bb79a1e8700fc25ff755dafbf588c7818384d8" Oct 30 19:53:00.026607 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.026578 5091 scope.go:122] "RemoveContainer" containerID="e11344a9458299258f6751fbb4bb79a1e8700fc25ff755dafbf588c7818384d8" Oct 30 19:53:00.026994 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:53:00.026965 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e11344a9458299258f6751fbb4bb79a1e8700fc25ff755dafbf588c7818384d8\": container with ID starting with e11344a9458299258f6751fbb4bb79a1e8700fc25ff755dafbf588c7818384d8 not found: ID does not exist" containerID="e11344a9458299258f6751fbb4bb79a1e8700fc25ff755dafbf588c7818384d8" Oct 30 19:53:00.027089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.027006 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e11344a9458299258f6751fbb4bb79a1e8700fc25ff755dafbf588c7818384d8"} err="failed to get container status \"e11344a9458299258f6751fbb4bb79a1e8700fc25ff755dafbf588c7818384d8\": rpc error: code = NotFound desc = could not find container \"e11344a9458299258f6751fbb4bb79a1e8700fc25ff755dafbf588c7818384d8\": container with ID starting with e11344a9458299258f6751fbb4bb79a1e8700fc25ff755dafbf588c7818384d8 not found: ID does not exist" Oct 30 19:53:00.047737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.047460 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:00.051118 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.051057 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7"] Oct 30 19:53:00.057348 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.057314 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-913351/ipp-route-913351-kserve-66b7cb8c7-w9fg7"] Oct 30 19:53:00.276961 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.276854 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:00.535565 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.535329 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:00.583608 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.583574 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5faa3220-d981-43af-a6d2-19d841026802" path="/var/lib/kubelet/pods/5faa3220-d981-43af-a6d2-19d841026802/volumes" Oct 30 19:53:00.584354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.584329 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr"] Oct 30 19:53:00.584562 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.584546 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ec5e0f-cfcpr"] Oct 30 19:53:00.652934 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.652564 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" Oct 30 19:53:00.687953 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.687886 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:00.750230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.749121 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/433ca046-a713-4f04-a152-027d1bdeee27-kube-api-access-pvhgr\" (UniqueName: \"kubernetes.io/projected/433ca046-a713-4f04-a152-027d1bdeee27-kube-api-access-pvhgr\") pod \"433ca046-a713-4f04-a152-027d1bdeee27\" (UID: \"433ca046-a713-4f04-a152-027d1bdeee27\") " Oct 30 19:53:00.750230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.749206 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/433ca046-a713-4f04-a152-027d1bdeee27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433ca046-a713-4f04-a152-027d1bdeee27-plugins-config-volume\") pod \"433ca046-a713-4f04-a152-027d1bdeee27\" (UID: \"433ca046-a713-4f04-a152-027d1bdeee27\") " Oct 30 19:53:00.750684 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.750642 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/433ca046-a713-4f04-a152-027d1bdeee27-plugins-config-volume" pod "433ca046-a713-4f04-a152-027d1bdeee27" (UID: "433ca046-a713-4f04-a152-027d1bdeee27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:00.762921 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.762877 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/433ca046-a713-4f04-a152-027d1bdeee27-kube-api-access-pvhgr" pod "433ca046-a713-4f04-a152-027d1bdeee27" (UID: "433ca046-a713-4f04-a152-027d1bdeee27"). InnerVolumeSpecName "kube-api-access-pvhgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:00.764230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.763707 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" Oct 30 19:53:00.851670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.851003 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be3e710b-55f4-4cd6-8872-f7db69725afc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3e710b-55f4-4cd6-8872-f7db69725afc-plugins-config-volume\") pod \"be3e710b-55f4-4cd6-8872-f7db69725afc\" (UID: \"be3e710b-55f4-4cd6-8872-f7db69725afc\") " Oct 30 19:53:00.851670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.851214 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3e710b-55f4-4cd6-8872-f7db69725afc-kube-api-access-tt5cj\" (UniqueName: \"kubernetes.io/projected/be3e710b-55f4-4cd6-8872-f7db69725afc-kube-api-access-tt5cj\") pod \"be3e710b-55f4-4cd6-8872-f7db69725afc\" (UID: \"be3e710b-55f4-4cd6-8872-f7db69725afc\") " Oct 30 19:53:00.853392 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.851988 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvhgr\" (UniqueName: \"kubernetes.io/projected/433ca046-a713-4f04-a152-027d1bdeee27-kube-api-access-pvhgr\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:00.853392 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.852011 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/433ca046-a713-4f04-a152-027d1bdeee27-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:00.853392 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.852550 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3e710b-55f4-4cd6-8872-f7db69725afc-plugins-config-volume" pod "be3e710b-55f4-4cd6-8872-f7db69725afc" (UID: "be3e710b-55f4-4cd6-8872-f7db69725afc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:00.866055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.865939 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3e710b-55f4-4cd6-8872-f7db69725afc-kube-api-access-tt5cj" pod "be3e710b-55f4-4cd6-8872-f7db69725afc" (UID: "be3e710b-55f4-4cd6-8872-f7db69725afc"). InnerVolumeSpecName "kube-api-access-tt5cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:00.921813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.921761 5091 generic.go:356] "Generic (PLEG): container finished" podID="be3e710b-55f4-4cd6-8872-f7db69725afc" containerID="8227ceee5ca652df67bf3a3118fa741fde4208cedeb130146cbda24054463b56" exitCode=0 Oct 30 19:53:00.922033 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.921938 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" event={"ID":"be3e710b-55f4-4cd6-8872-f7db69725afc","Type":"ContainerDied","Data":"8227ceee5ca652df67bf3a3118fa741fde4208cedeb130146cbda24054463b56"} Oct 30 19:53:00.922033 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.922006 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" event={"ID":"be3e710b-55f4-4cd6-8872-f7db69725afc","Type":"ContainerDied","Data":"09760b90891f3825911f233da6abd05cfc9dab94231c2fb5c2d79b97d02f5c23"} Oct 30 19:53:00.922112 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.922040 5091 scope.go:122] "RemoveContainer" containerID="8227ceee5ca652df67bf3a3118fa741fde4208cedeb130146cbda24054463b56" Oct 30 19:53:00.922193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.922180 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l" Oct 30 19:53:00.924205 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.924176 5091 generic.go:356] "Generic (PLEG): container finished" podID="433ca046-a713-4f04-a152-027d1bdeee27" containerID="69fa1c39bffae983c8a2940dbcaa62afe338ca72593c7b4c4e53187a38a3285b" exitCode=0 Oct 30 19:53:00.924346 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.924276 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" event={"ID":"433ca046-a713-4f04-a152-027d1bdeee27","Type":"ContainerDied","Data":"69fa1c39bffae983c8a2940dbcaa62afe338ca72593c7b4c4e53187a38a3285b"} Oct 30 19:53:00.924346 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.924324 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" event={"ID":"433ca046-a713-4f04-a152-027d1bdeee27","Type":"ContainerDied","Data":"89370883f2fcd8fa61aee61c10fd8f65e4f8f6e2e3f205f2775723845fea80db"} Oct 30 19:53:00.924465 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.924398 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" Oct 30 19:53:00.944442 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.942827 5091 scope.go:122] "RemoveContainer" containerID="8227ceee5ca652df67bf3a3118fa741fde4208cedeb130146cbda24054463b56" Oct 30 19:53:00.944442 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:53:00.943224 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8227ceee5ca652df67bf3a3118fa741fde4208cedeb130146cbda24054463b56\": container with ID starting with 8227ceee5ca652df67bf3a3118fa741fde4208cedeb130146cbda24054463b56 not found: ID does not exist" containerID="8227ceee5ca652df67bf3a3118fa741fde4208cedeb130146cbda24054463b56" Oct 30 19:53:00.944442 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.943260 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8227ceee5ca652df67bf3a3118fa741fde4208cedeb130146cbda24054463b56"} err="failed to get container status \"8227ceee5ca652df67bf3a3118fa741fde4208cedeb130146cbda24054463b56\": rpc error: code = NotFound desc = could not find container \"8227ceee5ca652df67bf3a3118fa741fde4208cedeb130146cbda24054463b56\": container with ID starting with 8227ceee5ca652df67bf3a3118fa741fde4208cedeb130146cbda24054463b56 not found: ID does not exist" Oct 30 19:53:00.944442 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.943288 5091 scope.go:122] "RemoveContainer" containerID="69fa1c39bffae983c8a2940dbcaa62afe338ca72593c7b4c4e53187a38a3285b" Oct 30 19:53:00.949097 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.947367 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:00.955469 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.955419 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt5cj\" (UniqueName: \"kubernetes.io/projected/be3e710b-55f4-4cd6-8872-f7db69725afc-kube-api-access-tt5cj\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:00.955469 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.955444 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be3e710b-55f4-4cd6-8872-f7db69725afc-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:00.979913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.972309 5091 scope.go:122] "RemoveContainer" containerID="69fa1c39bffae983c8a2940dbcaa62afe338ca72593c7b4c4e53187a38a3285b" Oct 30 19:53:00.979913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.975503 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l"] Oct 30 19:53:00.980230 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:53:00.980138 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69fa1c39bffae983c8a2940dbcaa62afe338ca72593c7b4c4e53187a38a3285b\": container with ID starting with 69fa1c39bffae983c8a2940dbcaa62afe338ca72593c7b4c4e53187a38a3285b not found: ID does not exist" containerID="69fa1c39bffae983c8a2940dbcaa62afe338ca72593c7b4c4e53187a38a3285b" Oct 30 19:53:00.980230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.980200 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69fa1c39bffae983c8a2940dbcaa62afe338ca72593c7b4c4e53187a38a3285b"} err="failed to get container status \"69fa1c39bffae983c8a2940dbcaa62afe338ca72593c7b4c4e53187a38a3285b\": rpc error: code = NotFound desc = could not find container \"69fa1c39bffae983c8a2940dbcaa62afe338ca72593c7b4c4e53187a38a3285b\": container with ID starting with 69fa1c39bffae983c8a2940dbcaa62afe338ca72593c7b4c4e53187a38a3285b not found: ID does not exist" Oct 30 19:53:00.983858 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:00.983824 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ec5e0f-6744b4d9b-5h64l"] Oct 30 19:53:01.002832 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:01.002795 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff"] Oct 30 19:53:01.008464 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:01.008430 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff"] Oct 30 19:53:01.057493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:01.057456 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk"] Oct 30 19:53:01.058089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:01.057781 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" podUID="0c9adda6-46b7-42fa-9230-eeb60648a305" containerName="istio-proxy" containerID="cri-o://f827f1d08677fa255340d862901e564746d6a37b0a147f266f36236b8494933b" gracePeriod=30 Oct 30 19:53:01.500955 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:01.500912 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ec5e0f-7d456d9f7f-g74ff" podUID="433ca046-a713-4f04-a152-027d1bdeee27" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.147:9004: i/o timeout" Oct 30 19:53:01.789343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:01.789140 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:02.265426 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:02.265377 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:02.445683 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:02.445643 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="433ca046-a713-4f04-a152-027d1bdeee27" path="/var/lib/kubelet/pods/433ca046-a713-4f04-a152-027d1bdeee27/volumes" Oct 30 19:53:02.446360 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:02.446336 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c5e14e-4ed6-44a7-901a-8f78bbe593b5" path="/var/lib/kubelet/pods/63c5e14e-4ed6-44a7-901a-8f78bbe593b5/volumes" Oct 30 19:53:02.446939 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:02.446918 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be3e710b-55f4-4cd6-8872-f7db69725afc" path="/var/lib/kubelet/pods/be3e710b-55f4-4cd6-8872-f7db69725afc/volumes" Oct 30 19:53:02.751753 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:02.751722 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:03.370233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:03.369915 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": read tcp 10.132.0.2:43630->10.132.0.153:15021: read: connection reset by peer" start-of-body= Oct 30 19:53:03.370233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:03.369975 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": read tcp 10.132.0.2:43630->10.132.0.153:15021: read: connection reset by peer" Oct 30 19:53:03.869011 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:03.868977 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:03.966700 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:03.966662 5091 generic.go:356] "Generic (PLEG): container finished" podID="2bff8956-093c-4a78-a281-8571789127a7" containerID="daaa5ef9b5bc570c59b69658ec8d3fc45424a35ba2b8d89db66a46dbb8df6743" exitCode=0 Oct 30 19:53:03.966868 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:03.966727 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" event={"ID":"2bff8956-093c-4a78-a281-8571789127a7","Type":"ContainerDied","Data":"daaa5ef9b5bc570c59b69658ec8d3fc45424a35ba2b8d89db66a46dbb8df6743"} Oct 30 19:53:03.967308 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:03.967283 5091 scope.go:122] "RemoveContainer" containerID="daaa5ef9b5bc570c59b69658ec8d3fc45424a35ba2b8d89db66a46dbb8df6743" Oct 30 19:53:03.968796 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:03.968776 5091 generic.go:356] "Generic (PLEG): container finished" podID="0b09a84c-ae99-434b-8562-2a921c923e33" containerID="282c799af4a39cb8f9aab241a18179721065caeb52aa98b75b7f49c8abdf7621" exitCode=0 Oct 30 19:53:03.968907 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:03.968846 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" event={"ID":"0b09a84c-ae99-434b-8562-2a921c923e33","Type":"ContainerDied","Data":"282c799af4a39cb8f9aab241a18179721065caeb52aa98b75b7f49c8abdf7621"} Oct 30 19:53:03.969299 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:03.969280 5091 scope.go:122] "RemoveContainer" containerID="282c799af4a39cb8f9aab241a18179721065caeb52aa98b75b7f49c8abdf7621" Oct 30 19:53:03.969413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:03.969362 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:53:03.969472 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:03.969453 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:53:03.969525 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:03.969503 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:53:03.971014 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:03.970982 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:53:03.971115 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:03.971061 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:53:03.971115 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:03.971106 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:53:04.931171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:04.923488 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:05.039305 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:05.039267 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" event={"ID":"2bff8956-093c-4a78-a281-8571789127a7","Type":"ContainerStarted","Data":"8704e0ac2dd4bd7d9cc142cfdf13918342a82975233e1e14c43ff18bf852d989"} Oct 30 19:53:05.042123 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:05.042095 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" event={"ID":"0b09a84c-ae99-434b-8562-2a921c923e33","Type":"ContainerStarted","Data":"507e988d2c08dcc131c57776eb8ea5a11f3ff5807057dc6b9400759936ac5f40"} Oct 30 19:53:05.561358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:05.561322 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:53:05.561712 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:05.561686 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 19:53:05.561881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:05.561857 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 19:53:05.679936 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:05.679902 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:53:05.680307 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:05.680281 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:53:05.685246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:05.685208 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" podUID="2bff8956-093c-4a78-a281-8571789127a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 19:53:05.801171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:05.800810 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:06.574035 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.573983 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 19:53:06.574596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.574075 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 19:53:06.598414 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.598383 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:53:06.642064 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.642025 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-token\" (UniqueName: \"kubernetes.io/projected/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-token\") pod \"0c9adda6-46b7-42fa-9230-eeb60648a305\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " Oct 30 19:53:06.642273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.642247 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-podinfo\") pod \"0c9adda6-46b7-42fa-9230-eeb60648a305\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " Oct 30 19:53:06.642357 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.642297 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-workload-certs\") pod \"0c9adda6-46b7-42fa-9230-eeb60648a305\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " Oct 30 19:53:06.642401 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.642356 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-envoy\") pod \"0c9adda6-46b7-42fa-9230-eeb60648a305\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " Oct 30 19:53:06.642401 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.642389 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c9adda6-46b7-42fa-9230-eeb60648a305-kube-api-access-vwdwv\" (UniqueName: \"kubernetes.io/projected/0c9adda6-46b7-42fa-9230-eeb60648a305-kube-api-access-vwdwv\") pod \"0c9adda6-46b7-42fa-9230-eeb60648a305\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " Oct 30 19:53:06.642473 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.642446 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-data\") pod \"0c9adda6-46b7-42fa-9230-eeb60648a305\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " Oct 30 19:53:06.642509 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.642487 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c9adda6-46b7-42fa-9230-eeb60648a305-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9adda6-46b7-42fa-9230-eeb60648a305-istiod-ca-cert\") pod \"0c9adda6-46b7-42fa-9230-eeb60648a305\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " Oct 30 19:53:06.642586 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.642568 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-workload-socket\") pod \"0c9adda6-46b7-42fa-9230-eeb60648a305\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " Oct 30 19:53:06.642660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.642637 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-credential-socket\") pod \"0c9adda6-46b7-42fa-9230-eeb60648a305\" (UID: \"0c9adda6-46b7-42fa-9230-eeb60648a305\") " Oct 30 19:53:06.643481 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.643044 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-workload-certs" pod "0c9adda6-46b7-42fa-9230-eeb60648a305" (UID: "0c9adda6-46b7-42fa-9230-eeb60648a305"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:06.643481 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.643097 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-data" pod "0c9adda6-46b7-42fa-9230-eeb60648a305" (UID: "0c9adda6-46b7-42fa-9230-eeb60648a305"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:06.643481 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.643275 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-workload-socket" pod "0c9adda6-46b7-42fa-9230-eeb60648a305" (UID: "0c9adda6-46b7-42fa-9230-eeb60648a305"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:06.643481 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.643358 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-credential-socket" pod "0c9adda6-46b7-42fa-9230-eeb60648a305" (UID: "0c9adda6-46b7-42fa-9230-eeb60648a305"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:06.643481 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.643417 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:06.643481 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.643437 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:06.643481 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.643453 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:06.643796 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.643561 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c9adda6-46b7-42fa-9230-eeb60648a305-istiod-ca-cert" pod "0c9adda6-46b7-42fa-9230-eeb60648a305" (UID: "0c9adda6-46b7-42fa-9230-eeb60648a305"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:06.678642 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.678351 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-token" pod "0c9adda6-46b7-42fa-9230-eeb60648a305" (UID: "0c9adda6-46b7-42fa-9230-eeb60648a305"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:06.710242 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.689113 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:53:06.710242 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.689196 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" podUID="2bff8956-093c-4a78-a281-8571789127a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 19:53:06.712188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.710803 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-envoy" pod "0c9adda6-46b7-42fa-9230-eeb60648a305" (UID: "0c9adda6-46b7-42fa-9230-eeb60648a305"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:06.718733 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.716737 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-podinfo" pod "0c9adda6-46b7-42fa-9230-eeb60648a305" (UID: "0c9adda6-46b7-42fa-9230-eeb60648a305"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:53:06.718733 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.716791 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9adda6-46b7-42fa-9230-eeb60648a305-kube-api-access-vwdwv" pod "0c9adda6-46b7-42fa-9230-eeb60648a305" (UID: "0c9adda6-46b7-42fa-9230-eeb60648a305"). InnerVolumeSpecName "kube-api-access-vwdwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:06.745340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.745222 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:06.745340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.745257 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:06.745340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.745272 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:06.745340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.745286 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c9adda6-46b7-42fa-9230-eeb60648a305-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:06.745340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.745302 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwdwv\" (UniqueName: \"kubernetes.io/projected/0c9adda6-46b7-42fa-9230-eeb60648a305-kube-api-access-vwdwv\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:06.745340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:06.745318 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c9adda6-46b7-42fa-9230-eeb60648a305-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:07.102173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:07.096669 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:07.169194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:07.166012 5091 generic.go:356] "Generic (PLEG): container finished" podID="0c9adda6-46b7-42fa-9230-eeb60648a305" containerID="f827f1d08677fa255340d862901e564746d6a37b0a147f266f36236b8494933b" exitCode=0 Oct 30 19:53:07.169194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:07.166093 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" event={"ID":"0c9adda6-46b7-42fa-9230-eeb60648a305","Type":"ContainerDied","Data":"f827f1d08677fa255340d862901e564746d6a37b0a147f266f36236b8494933b"} Oct 30 19:53:07.169194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:07.166172 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" event={"ID":"0c9adda6-46b7-42fa-9230-eeb60648a305","Type":"ContainerDied","Data":"8f4c09c89b1d307fe96a730540910c7fa88211b296f9aba7c5707b82765953f4"} Oct 30 19:53:07.169194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:07.166211 5091 scope.go:122] "RemoveContainer" containerID="f827f1d08677fa255340d862901e564746d6a37b0a147f266f36236b8494933b" Oct 30 19:53:07.169194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:07.166405 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk" Oct 30 19:53:07.237335 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:07.237254 5091 scope.go:122] "RemoveContainer" containerID="f827f1d08677fa255340d862901e564746d6a37b0a147f266f36236b8494933b" Oct 30 19:53:07.237667 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:53:07.237637 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f827f1d08677fa255340d862901e564746d6a37b0a147f266f36236b8494933b\": container with ID starting with f827f1d08677fa255340d862901e564746d6a37b0a147f266f36236b8494933b not found: ID does not exist" containerID="f827f1d08677fa255340d862901e564746d6a37b0a147f266f36236b8494933b" Oct 30 19:53:07.237729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:07.237677 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f827f1d08677fa255340d862901e564746d6a37b0a147f266f36236b8494933b"} err="failed to get container status \"f827f1d08677fa255340d862901e564746d6a37b0a147f266f36236b8494933b\": rpc error: code = NotFound desc = could not find container \"f827f1d08677fa255340d862901e564746d6a37b0a147f266f36236b8494933b\": container with ID starting with f827f1d08677fa255340d862901e564746d6a37b0a147f266f36236b8494933b not found: ID does not exist" Oct 30 19:53:07.309256 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:07.309208 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk"] Oct 30 19:53:07.315036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:07.313826 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-k4ppk"] Oct 30 19:53:07.560169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:07.557565 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 19:53:07.560169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:07.557642 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 19:53:07.677259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:07.676553 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:53:07.677259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:07.676625 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" podUID="2bff8956-093c-4a78-a281-8571789127a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 19:53:08.292466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:08.292433 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:08.442390 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:08.442351 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c9adda6-46b7-42fa-9230-eeb60648a305" path="/var/lib/kubelet/pods/0c9adda6-46b7-42fa-9230-eeb60648a305/volumes" Oct 30 19:53:08.522533 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:08.522502 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:08.565224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:08.564555 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 19:53:08.565224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:08.564624 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 19:53:08.686169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:08.685549 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:53:08.686169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:08.685613 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" podUID="2bff8956-093c-4a78-a281-8571789127a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 19:53:09.056180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.055572 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:09.277169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.276336 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:09.342528 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.340063 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn"] Oct 30 19:53:09.342528 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.342292 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl"] Oct 30 19:53:09.344184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.343129 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" podUID="192dd6ce-3921-41dc-ba51-be3b5535cb51" containerName="istio-proxy" containerID="cri-o://1d9ef5ec373924f04fed2748b9ed3ce54de13b437716e306dd6d1006bfc4c510" gracePeriod=30 Oct 30 19:53:09.368867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.368065 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" podUID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" containerName="istio-proxy" containerID="cri-o://c0b37806ac0863fbf43a3ac740d75b21e3d206f6f0614f1b3cee6a5a1ace469d" gracePeriod=30 Oct 30 19:53:09.515168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.514831 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn"] Oct 30 19:53:09.515600 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.515576 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5faa3220-d981-43af-a6d2-19d841026802" containerName="main" Oct 30 19:53:09.515764 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.515746 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5faa3220-d981-43af-a6d2-19d841026802" containerName="main" Oct 30 19:53:09.515877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.515866 5091 cpu_manager.go:414] "containerMap: removing container" podUID="433ca046-a713-4f04-a152-027d1bdeee27" containerName="payload-pre-processing" Oct 30 19:53:09.515959 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.515948 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="433ca046-a713-4f04-a152-027d1bdeee27" containerName="payload-pre-processing" Oct 30 19:53:09.516066 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.516056 5091 cpu_manager.go:414] "containerMap: removing container" podUID="be3e710b-55f4-4cd6-8872-f7db69725afc" containerName="payload-processing" Oct 30 19:53:09.516154 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.516132 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3e710b-55f4-4cd6-8872-f7db69725afc" containerName="payload-processing" Oct 30 19:53:09.516282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.516252 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0c9adda6-46b7-42fa-9230-eeb60648a305" containerName="istio-proxy" Oct 30 19:53:09.516357 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.516348 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c9adda6-46b7-42fa-9230-eeb60648a305" containerName="istio-proxy" Oct 30 19:53:09.516653 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.516638 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c9adda6-46b7-42fa-9230-eeb60648a305" containerName="istio-proxy" Oct 30 19:53:09.516774 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.516764 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="433ca046-a713-4f04-a152-027d1bdeee27" containerName="payload-pre-processing" Oct 30 19:53:09.516854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.516843 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5faa3220-d981-43af-a6d2-19d841026802" containerName="main" Oct 30 19:53:09.516931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.516923 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3e710b-55f4-4cd6-8872-f7db69725afc" containerName="payload-processing" Oct 30 19:53:09.524968 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.524946 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn"] Oct 30 19:53:09.525120 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.525095 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.529160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.529021 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:09.537636 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.537618 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-b17e94\"/\"openshift-service-ca.crt\"" Oct 30 19:53:09.537802 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.537757 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-b17e94\"/\"praxis-route-b17e94-kserve-self-signed-certs\"" Oct 30 19:53:09.606249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.593674 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-b17e94\"/\"default-dockercfg-9hfsw\"" Oct 30 19:53:09.606249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.601284 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-b17e94\"/\"kube-root-ca.crt\"" Oct 30 19:53:09.606249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.605439 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 19:53:09.606249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.605590 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 19:53:09.613808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.613783 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-home\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.613915 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.613832 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-tmp-dir\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.613915 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.613852 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-model-cache\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.613915 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.613871 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdcct\" (UniqueName: \"kubernetes.io/projected/0025d83b-0a0d-455d-9634-76492646d579-kube-api-access-gdcct\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.613915 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.613888 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-dshm\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.614063 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.613906 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0025d83b-0a0d-455d-9634-76492646d579-tls-certs\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.683168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.679655 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:53:09.692505 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.692435 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" podUID="2bff8956-093c-4a78-a281-8571789127a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 19:53:09.726088 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.717130 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-home\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.726545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.726512 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-tmp-dir\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.726856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.726807 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-model-cache\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.728042 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.728002 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdcct\" (UniqueName: \"kubernetes.io/projected/0025d83b-0a0d-455d-9634-76492646d579-kube-api-access-gdcct\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.728638 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.728618 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-dshm\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.728902 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.728881 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0025d83b-0a0d-455d-9634-76492646d579-tls-certs\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.730457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.729048 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-model-cache\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.735615 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.727786 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-tmp-dir\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.735785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.717583 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-home\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.762188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.753441 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-dshm\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.852170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.833176 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0025d83b-0a0d-455d-9634-76492646d579-tls-certs\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:09.853956 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.853915 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:09.945536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.945447 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" Oct 30 19:53:09.965233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:09.965205 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" Oct 30 19:53:10.037867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:10.037832 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdcct\" (UniqueName: \"kubernetes.io/projected/0025d83b-0a0d-455d-9634-76492646d579-kube-api-access-gdcct\") pod \"praxis-route-b17e94-kserve-6c5f5d757-4pszn\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:10.144319 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:10.143856 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:10.236183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:10.235317 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:53:10.575178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:10.570430 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Oct 30 19:53:10.575178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:10.570501 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Oct 30 19:53:10.683882 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:10.683845 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Oct 30 19:53:10.684134 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:10.684108 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" podUID="2bff8956-093c-4a78-a281-8571789127a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Oct 30 19:53:10.845962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:10.772758 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn"] Oct 30 19:53:10.865844 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:10.865809 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:10.874063 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:53:10.874004 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0025d83b_0a0d_455d_9634_76492646d579.slice/crio-84f6c9f8fd171de81d2047ce4c9411dd9d8ec4d9cddc3b572a0d527ad4be353a WatchSource:0}: Error finding container 84f6c9f8fd171de81d2047ce4c9411dd9d8ec4d9cddc3b572a0d527ad4be353a: Status 404 returned error can't find the container with id 84f6c9f8fd171de81d2047ce4c9411dd9d8ec4d9cddc3b572a0d527ad4be353a Oct 30 19:53:11.441114 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:11.441075 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" event={"ID":"0025d83b-0a0d-455d-9634-76492646d579","Type":"ContainerStarted","Data":"a53abbc1383d9480daecf2c2ec0c95e5f24df3f27b103ff6b78e94cb91cc7375"} Oct 30 19:53:11.441360 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:11.441124 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" event={"ID":"0025d83b-0a0d-455d-9634-76492646d579","Type":"ContainerStarted","Data":"84f6c9f8fd171de81d2047ce4c9411dd9d8ec4d9cddc3b572a0d527ad4be353a"} Oct 30 19:53:11.468525 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:11.468481 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" podStartSLOduration=2.468467812 podStartE2EDuration="2.468467812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:11.464678191 +0000 UTC m=+1311.159886139" watchObservedRunningTime="2026-10-30 19:53:11.468467812 +0000 UTC m=+1311.163675769" Oct 30 19:53:11.562205 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:11.562132 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:53:11.562631 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:11.562601 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:53:11.563318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:11.563302 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:53:11.682798 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:11.682773 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:53:11.684977 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:11.684954 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:53:11.759567 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:11.759523 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:11.950773 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:11.950737 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:12.146357 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:12.146276 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:12.164067 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:12.164033 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:12.805785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:12.805699 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:13.534934 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.534903 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb"] Oct 30 19:53:13.539442 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.539424 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" Oct 30 19:53:13.546849 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.546216 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:53:13.546849 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.546282 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:53:13.548586 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.548562 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb"] Oct 30 19:53:13.709767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.709735 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jsrq\" (UniqueName: \"kubernetes.io/projected/1897963b-68eb-4959-b391-16b595122405-kube-api-access-9jsrq\") pod \"maas-api-revoke-sub-f08a3c84-k9mqb\" (UID: \"1897963b-68eb-4959-b391-16b595122405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" Oct 30 19:53:13.709927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.709877 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1897963b-68eb-4959-b391-16b595122405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08a3c84-k9mqb\" (UID: \"1897963b-68eb-4959-b391-16b595122405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" Oct 30 19:53:13.710026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.709998 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1897963b-68eb-4959-b391-16b595122405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08a3c84-k9mqb\" (UID: \"1897963b-68eb-4959-b391-16b595122405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" Oct 30 19:53:13.811881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.811786 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jsrq\" (UniqueName: \"kubernetes.io/projected/1897963b-68eb-4959-b391-16b595122405-kube-api-access-9jsrq\") pod \"maas-api-revoke-sub-f08a3c84-k9mqb\" (UID: \"1897963b-68eb-4959-b391-16b595122405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" Oct 30 19:53:13.812259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.812231 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1897963b-68eb-4959-b391-16b595122405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08a3c84-k9mqb\" (UID: \"1897963b-68eb-4959-b391-16b595122405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" Oct 30 19:53:13.812458 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.812439 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1897963b-68eb-4959-b391-16b595122405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08a3c84-k9mqb\" (UID: \"1897963b-68eb-4959-b391-16b595122405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" Oct 30 19:53:13.813538 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.813515 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1897963b-68eb-4959-b391-16b595122405-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f08a3c84-k9mqb\" (UID: \"1897963b-68eb-4959-b391-16b595122405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" Oct 30 19:53:13.818953 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.818859 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1897963b-68eb-4959-b391-16b595122405-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f08a3c84-k9mqb\" (UID: \"1897963b-68eb-4959-b391-16b595122405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" Oct 30 19:53:13.837709 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.837657 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jsrq\" (UniqueName: \"kubernetes.io/projected/1897963b-68eb-4959-b391-16b595122405-kube-api-access-9jsrq\") pod \"maas-api-revoke-sub-f08a3c84-k9mqb\" (UID: \"1897963b-68eb-4959-b391-16b595122405\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" Oct 30 19:53:13.854061 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.854023 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:13.858575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:13.858546 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" Oct 30 19:53:14.155017 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:53:14.154967 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1897963b_68eb_4959_b391_16b595122405.slice/crio-271209d6de5a1d92216b8da5f149a8bfcd5c78037590bffe9f96a4cc68917432 WatchSource:0}: Error finding container 271209d6de5a1d92216b8da5f149a8bfcd5c78037590bffe9f96a4cc68917432: Status 404 returned error can't find the container with id 271209d6de5a1d92216b8da5f149a8bfcd5c78037590bffe9f96a4cc68917432 Oct 30 19:53:14.158269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.157990 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb"] Oct 30 19:53:14.473519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.473481 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" event={"ID":"1897963b-68eb-4959-b391-16b595122405","Type":"ContainerStarted","Data":"271209d6de5a1d92216b8da5f149a8bfcd5c78037590bffe9f96a4cc68917432"} Oct 30 19:53:14.680417 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.680392 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:53:14.732193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.727253 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:53:14.834197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.833652 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-workload-certs\") pod \"192dd6ce-3921-41dc-ba51-be3b5535cb51\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " Oct 30 19:53:14.834197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.833804 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-workload-socket\") pod \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " Oct 30 19:53:14.834197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.833839 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-envoy\") pod \"192dd6ce-3921-41dc-ba51-be3b5535cb51\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " Oct 30 19:53:14.834197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.833899 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-kube-api-access-ddwlm\" (UniqueName: \"kubernetes.io/projected/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-kube-api-access-ddwlm\") pod \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " Oct 30 19:53:14.834197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.833975 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192dd6ce-3921-41dc-ba51-be3b5535cb51-kube-api-access-q22tg\" (UniqueName: \"kubernetes.io/projected/192dd6ce-3921-41dc-ba51-be3b5535cb51-kube-api-access-q22tg\") pod \"192dd6ce-3921-41dc-ba51-be3b5535cb51\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " Oct 30 19:53:14.834197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.834003 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-workload-certs\") pod \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " Oct 30 19:53:14.834197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.834037 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-credential-socket\") pod \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " Oct 30 19:53:14.834197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.834079 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-data\") pod \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " Oct 30 19:53:14.834863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.834221 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-podinfo\") pod \"192dd6ce-3921-41dc-ba51-be3b5535cb51\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " Oct 30 19:53:14.834863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.834292 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-envoy\") pod \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " Oct 30 19:53:14.834863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.834319 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/192dd6ce-3921-41dc-ba51-be3b5535cb51-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192dd6ce-3921-41dc-ba51-be3b5535cb51-istiod-ca-cert\") pod \"192dd6ce-3921-41dc-ba51-be3b5535cb51\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " Oct 30 19:53:14.834863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.834371 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-token\" (UniqueName: \"kubernetes.io/projected/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-token\") pod \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " Oct 30 19:53:14.834863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.834401 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-workload-socket\") pod \"192dd6ce-3921-41dc-ba51-be3b5535cb51\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " Oct 30 19:53:14.834863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.834430 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-token\" (UniqueName: \"kubernetes.io/projected/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-token\") pod \"192dd6ce-3921-41dc-ba51-be3b5535cb51\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " Oct 30 19:53:14.834863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.834490 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-data\") pod \"192dd6ce-3921-41dc-ba51-be3b5535cb51\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " Oct 30 19:53:14.834863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.834566 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-podinfo\") pod \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " Oct 30 19:53:14.834863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.834617 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-credential-socket\") pod \"192dd6ce-3921-41dc-ba51-be3b5535cb51\" (UID: \"192dd6ce-3921-41dc-ba51-be3b5535cb51\") " Oct 30 19:53:14.834863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.834688 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istiod-ca-cert\") pod \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\" (UID: \"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b\") " Oct 30 19:53:14.880267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.879064 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istiod-ca-cert" pod "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" (UID: "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:14.881561 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.881526 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:14.881804 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.881781 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-credential-socket" pod "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" (UID: "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:14.882004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.881978 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-data" pod "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" (UID: "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:14.882004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.881996 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-workload-certs" pod "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" (UID: "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:14.886718 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.886517 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-kube-api-access-ddwlm" pod "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" (UID: "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b"). InnerVolumeSpecName "kube-api-access-ddwlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:14.886718 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.886524 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-envoy" pod "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" (UID: "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:14.886885 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.886804 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-credential-socket" pod "192dd6ce-3921-41dc-ba51-be3b5535cb51" (UID: "192dd6ce-3921-41dc-ba51-be3b5535cb51"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:14.887201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.887005 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-workload-certs" pod "192dd6ce-3921-41dc-ba51-be3b5535cb51" (UID: "192dd6ce-3921-41dc-ba51-be3b5535cb51"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:14.887201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.887040 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-workload-socket" pod "192dd6ce-3921-41dc-ba51-be3b5535cb51" (UID: "192dd6ce-3921-41dc-ba51-be3b5535cb51"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:14.887577 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.887403 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-workload-socket" pod "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" (UID: "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:14.887733 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.887684 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-data" pod "192dd6ce-3921-41dc-ba51-be3b5535cb51" (UID: "192dd6ce-3921-41dc-ba51-be3b5535cb51"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:14.887915 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.887891 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-token" pod "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" (UID: "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:14.889463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.889409 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/192dd6ce-3921-41dc-ba51-be3b5535cb51-istiod-ca-cert" pod "192dd6ce-3921-41dc-ba51-be3b5535cb51" (UID: "192dd6ce-3921-41dc-ba51-be3b5535cb51"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:14.896269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.896243 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-podinfo" pod "192dd6ce-3921-41dc-ba51-be3b5535cb51" (UID: "192dd6ce-3921-41dc-ba51-be3b5535cb51"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:53:14.896666 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.896644 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-token" pod "192dd6ce-3921-41dc-ba51-be3b5535cb51" (UID: "192dd6ce-3921-41dc-ba51-be3b5535cb51"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:14.897051 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.897031 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-podinfo" pod "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" (UID: "ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:53:14.898334 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.898252 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-envoy" pod "192dd6ce-3921-41dc-ba51-be3b5535cb51" (UID: "192dd6ce-3921-41dc-ba51-be3b5535cb51"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:14.898334 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.898269 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/192dd6ce-3921-41dc-ba51-be3b5535cb51-kube-api-access-q22tg" pod "192dd6ce-3921-41dc-ba51-be3b5535cb51" (UID: "192dd6ce-3921-41dc-ba51-be3b5535cb51"). InnerVolumeSpecName "kube-api-access-q22tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:14.937278 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937215 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937278 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937240 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937278 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937249 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/192dd6ce-3921-41dc-ba51-be3b5535cb51-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937278 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937258 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937278 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937267 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937278 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937276 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937284 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937292 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937301 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937309 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937318 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937326 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937334 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/192dd6ce-3921-41dc-ba51-be3b5535cb51-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937342 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddwlm\" (UniqueName: \"kubernetes.io/projected/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-kube-api-access-ddwlm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937350 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q22tg\" (UniqueName: \"kubernetes.io/projected/192dd6ce-3921-41dc-ba51-be3b5535cb51-kube-api-access-q22tg\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937358 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937367 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:14.937549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:14.937376 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:15.486035 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.485995 5091 generic.go:356] "Generic (PLEG): container finished" podID="1897963b-68eb-4959-b391-16b595122405" containerID="873c58bd568ed4f4c113c8d45a4592a0a256723e1c31bb80e56af442d36cfc80" exitCode=0 Oct 30 19:53:15.486253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.486137 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" event={"ID":"1897963b-68eb-4959-b391-16b595122405","Type":"ContainerDied","Data":"873c58bd568ed4f4c113c8d45a4592a0a256723e1c31bb80e56af442d36cfc80"} Oct 30 19:53:15.488097 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.488062 5091 generic.go:356] "Generic (PLEG): container finished" podID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" containerID="c0b37806ac0863fbf43a3ac740d75b21e3d206f6f0614f1b3cee6a5a1ace469d" exitCode=0 Oct 30 19:53:15.488284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.488123 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" Oct 30 19:53:15.488284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.488141 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" event={"ID":"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b","Type":"ContainerDied","Data":"c0b37806ac0863fbf43a3ac740d75b21e3d206f6f0614f1b3cee6a5a1ace469d"} Oct 30 19:53:15.488284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.488225 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn" event={"ID":"ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b","Type":"ContainerDied","Data":"69adc3cebe7c0b49efdea38e9e0c7b527c997ed3db5b0d597d9127300b6047ee"} Oct 30 19:53:15.488284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.488258 5091 scope.go:122] "RemoveContainer" containerID="c0b37806ac0863fbf43a3ac740d75b21e3d206f6f0614f1b3cee6a5a1ace469d" Oct 30 19:53:15.489737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.489684 5091 generic.go:356] "Generic (PLEG): container finished" podID="192dd6ce-3921-41dc-ba51-be3b5535cb51" containerID="1d9ef5ec373924f04fed2748b9ed3ce54de13b437716e306dd6d1006bfc4c510" exitCode=0 Oct 30 19:53:15.489737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.489714 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" event={"ID":"192dd6ce-3921-41dc-ba51-be3b5535cb51","Type":"ContainerDied","Data":"1d9ef5ec373924f04fed2748b9ed3ce54de13b437716e306dd6d1006bfc4c510"} Oct 30 19:53:15.489869 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.489745 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" Oct 30 19:53:15.489869 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.489777 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl" event={"ID":"192dd6ce-3921-41dc-ba51-be3b5535cb51","Type":"ContainerDied","Data":"eeb5d1669ff836a427b78ee3e05bd96ed5cd5ce0dcc7a5cec8b474fcf87eaa02"} Oct 30 19:53:15.512324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.512302 5091 scope.go:122] "RemoveContainer" containerID="fd94dbb84e8ddf338e641fe05490cfe8c052f8b7e98e29073513c6bdf7c361b0" Oct 30 19:53:15.522891 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.522868 5091 scope.go:122] "RemoveContainer" containerID="c0b37806ac0863fbf43a3ac740d75b21e3d206f6f0614f1b3cee6a5a1ace469d" Oct 30 19:53:15.523183 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:53:15.523129 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0b37806ac0863fbf43a3ac740d75b21e3d206f6f0614f1b3cee6a5a1ace469d\": container with ID starting with c0b37806ac0863fbf43a3ac740d75b21e3d206f6f0614f1b3cee6a5a1ace469d not found: ID does not exist" containerID="c0b37806ac0863fbf43a3ac740d75b21e3d206f6f0614f1b3cee6a5a1ace469d" Oct 30 19:53:15.523183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.523167 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0b37806ac0863fbf43a3ac740d75b21e3d206f6f0614f1b3cee6a5a1ace469d"} err="failed to get container status \"c0b37806ac0863fbf43a3ac740d75b21e3d206f6f0614f1b3cee6a5a1ace469d\": rpc error: code = NotFound desc = could not find container \"c0b37806ac0863fbf43a3ac740d75b21e3d206f6f0614f1b3cee6a5a1ace469d\": container with ID starting with c0b37806ac0863fbf43a3ac740d75b21e3d206f6f0614f1b3cee6a5a1ace469d not found: ID does not exist" Oct 30 19:53:15.523283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.523185 5091 scope.go:122] "RemoveContainer" containerID="fd94dbb84e8ddf338e641fe05490cfe8c052f8b7e98e29073513c6bdf7c361b0" Oct 30 19:53:15.523421 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:53:15.523406 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd94dbb84e8ddf338e641fe05490cfe8c052f8b7e98e29073513c6bdf7c361b0\": container with ID starting with fd94dbb84e8ddf338e641fe05490cfe8c052f8b7e98e29073513c6bdf7c361b0 not found: ID does not exist" containerID="fd94dbb84e8ddf338e641fe05490cfe8c052f8b7e98e29073513c6bdf7c361b0" Oct 30 19:53:15.523466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.523424 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd94dbb84e8ddf338e641fe05490cfe8c052f8b7e98e29073513c6bdf7c361b0"} err="failed to get container status \"fd94dbb84e8ddf338e641fe05490cfe8c052f8b7e98e29073513c6bdf7c361b0\": rpc error: code = NotFound desc = could not find container \"fd94dbb84e8ddf338e641fe05490cfe8c052f8b7e98e29073513c6bdf7c361b0\": container with ID starting with fd94dbb84e8ddf338e641fe05490cfe8c052f8b7e98e29073513c6bdf7c361b0 not found: ID does not exist" Oct 30 19:53:15.523466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.523436 5091 scope.go:122] "RemoveContainer" containerID="1d9ef5ec373924f04fed2748b9ed3ce54de13b437716e306dd6d1006bfc4c510" Oct 30 19:53:15.533032 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.533009 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn"] Oct 30 19:53:15.534962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.534942 5091 scope.go:122] "RemoveContainer" containerID="1d9ef5ec373924f04fed2748b9ed3ce54de13b437716e306dd6d1006bfc4c510" Oct 30 19:53:15.535258 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:53:15.535238 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d9ef5ec373924f04fed2748b9ed3ce54de13b437716e306dd6d1006bfc4c510\": container with ID starting with 1d9ef5ec373924f04fed2748b9ed3ce54de13b437716e306dd6d1006bfc4c510 not found: ID does not exist" containerID="1d9ef5ec373924f04fed2748b9ed3ce54de13b437716e306dd6d1006bfc4c510" Oct 30 19:53:15.535322 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.535266 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d9ef5ec373924f04fed2748b9ed3ce54de13b437716e306dd6d1006bfc4c510"} err="failed to get container status \"1d9ef5ec373924f04fed2748b9ed3ce54de13b437716e306dd6d1006bfc4c510\": rpc error: code = NotFound desc = could not find container \"1d9ef5ec373924f04fed2748b9ed3ce54de13b437716e306dd6d1006bfc4c510\": container with ID starting with 1d9ef5ec373924f04fed2748b9ed3ce54de13b437716e306dd6d1006bfc4c510 not found: ID does not exist" Oct 30 19:53:15.536879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.536861 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-57lgn"] Oct 30 19:53:15.553848 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.553747 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl"] Oct 30 19:53:15.559824 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.559802 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ec5e0f-openshift-default-c6576c9bf-x8csl"] Oct 30 19:53:15.928895 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.928859 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn"] Oct 30 19:53:15.982894 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:15.982361 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:16.005189 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:53:16.003538 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:53:16.140963 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:16.140925 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:16.332117 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:16.332083 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:16.607900 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:16.607868 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="192dd6ce-3921-41dc-ba51-be3b5535cb51" path="/var/lib/kubelet/pods/192dd6ce-3921-41dc-ba51-be3b5535cb51/volumes" Oct 30 19:53:16.608655 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:16.608634 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" path="/var/lib/kubelet/pods/ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b/volumes" Oct 30 19:53:16.798492 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:16.798469 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" Oct 30 19:53:16.891932 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:16.891845 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1897963b-68eb-4959-b391-16b595122405-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1897963b-68eb-4959-b391-16b595122405-maas-api-xks-ca\") pod \"1897963b-68eb-4959-b391-16b595122405\" (UID: \"1897963b-68eb-4959-b391-16b595122405\") " Oct 30 19:53:16.892066 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:16.891933 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1897963b-68eb-4959-b391-16b595122405-kube-api-access-9jsrq\" (UniqueName: \"kubernetes.io/projected/1897963b-68eb-4959-b391-16b595122405-kube-api-access-9jsrq\") pod \"1897963b-68eb-4959-b391-16b595122405\" (UID: \"1897963b-68eb-4959-b391-16b595122405\") " Oct 30 19:53:16.892615 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:16.892590 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1897963b-68eb-4959-b391-16b595122405-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1897963b-68eb-4959-b391-16b595122405-maas-api-openshift-ca\") pod \"1897963b-68eb-4959-b391-16b595122405\" (UID: \"1897963b-68eb-4959-b391-16b595122405\") " Oct 30 19:53:16.893230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:16.893201 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1897963b-68eb-4959-b391-16b595122405-maas-api-openshift-ca" pod "1897963b-68eb-4959-b391-16b595122405" (UID: "1897963b-68eb-4959-b391-16b595122405"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:16.894793 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:16.894771 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1897963b-68eb-4959-b391-16b595122405-kube-api-access-9jsrq" pod "1897963b-68eb-4959-b391-16b595122405" (UID: "1897963b-68eb-4959-b391-16b595122405"). InnerVolumeSpecName "kube-api-access-9jsrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:16.895990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:16.895970 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1897963b-68eb-4959-b391-16b595122405-maas-api-xks-ca" pod "1897963b-68eb-4959-b391-16b595122405" (UID: "1897963b-68eb-4959-b391-16b595122405"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:16.994724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:16.994689 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1897963b-68eb-4959-b391-16b595122405-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:16.994724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:16.994725 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1897963b-68eb-4959-b391-16b595122405-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:16.995328 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:16.994739 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jsrq\" (UniqueName: \"kubernetes.io/projected/1897963b-68eb-4959-b391-16b595122405-kube-api-access-9jsrq\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:17.116167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:17.113907 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:17.612130 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:17.612100 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" Oct 30 19:53:17.612332 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:17.612129 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb" event={"ID":"1897963b-68eb-4959-b391-16b595122405","Type":"ContainerDied","Data":"271209d6de5a1d92216b8da5f149a8bfcd5c78037590bffe9f96a4cc68917432"} Oct 30 19:53:17.612332 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:17.612212 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="271209d6de5a1d92216b8da5f149a8bfcd5c78037590bffe9f96a4cc68917432" Oct 30 19:53:18.328355 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:18.328319 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:19.465050 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:19.464987 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:20.145755 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:20.145716 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:20.158915 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:20.158890 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:20.746175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:20.745871 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:53:21.004262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:20.947132 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" podUID="4c812a3d-79c6-4060-be64-0da4dd1aa4d9" containerName="payload-processing" containerID="cri-o://4efd3a4198e1878507cfab6c9ef8bc3f5963d44c9585607d59af258fa55830ab" gracePeriod=25 Oct 30 19:53:21.071877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.071712 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:21.402122 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.402096 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" Oct 30 19:53:21.480241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.480211 5091 scope.go:122] "RemoveContainer" containerID="61dbd4c26185be9a20036a9ce58d702f1b554c9ec819693f03947678a2403409" Oct 30 19:53:21.492285 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.492264 5091 scope.go:122] "RemoveContainer" containerID="75db0e52e84f3ab2394e163581ae569cb89baff19cda66c57afab04cc338a02b" Oct 30 19:53:21.502765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.502743 5091 scope.go:122] "RemoveContainer" containerID="4707cb545a69b1f9d2ba2519dfeeb9ac64ce29d6df2e59f3e3eb331e68ab3066" Oct 30 19:53:21.578462 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.578439 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c812a3d-79c6-4060-be64-0da4dd1aa4d9-kube-api-access-ltkjz\" (UniqueName: \"kubernetes.io/projected/4c812a3d-79c6-4060-be64-0da4dd1aa4d9-kube-api-access-ltkjz\") pod \"4c812a3d-79c6-4060-be64-0da4dd1aa4d9\" (UID: \"4c812a3d-79c6-4060-be64-0da4dd1aa4d9\") " Oct 30 19:53:21.578593 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.578575 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c812a3d-79c6-4060-be64-0da4dd1aa4d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c812a3d-79c6-4060-be64-0da4dd1aa4d9-plugins-config-volume\") pod \"4c812a3d-79c6-4060-be64-0da4dd1aa4d9\" (UID: \"4c812a3d-79c6-4060-be64-0da4dd1aa4d9\") " Oct 30 19:53:21.578921 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.578894 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c812a3d-79c6-4060-be64-0da4dd1aa4d9-plugins-config-volume" pod "4c812a3d-79c6-4060-be64-0da4dd1aa4d9" (UID: "4c812a3d-79c6-4060-be64-0da4dd1aa4d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:21.579458 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.579220 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4c812a3d-79c6-4060-be64-0da4dd1aa4d9-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:21.580927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.580901 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c812a3d-79c6-4060-be64-0da4dd1aa4d9-kube-api-access-ltkjz" pod "4c812a3d-79c6-4060-be64-0da4dd1aa4d9" (UID: "4c812a3d-79c6-4060-be64-0da4dd1aa4d9"). InnerVolumeSpecName "kube-api-access-ltkjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:21.680760 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.680727 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltkjz\" (UniqueName: \"kubernetes.io/projected/4c812a3d-79c6-4060-be64-0da4dd1aa4d9-kube-api-access-ltkjz\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:21.733919 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.733865 5091 generic.go:356] "Generic (PLEG): container finished" podID="4c812a3d-79c6-4060-be64-0da4dd1aa4d9" containerID="4efd3a4198e1878507cfab6c9ef8bc3f5963d44c9585607d59af258fa55830ab" exitCode=0 Oct 30 19:53:21.733919 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.733910 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" Oct 30 19:53:21.734225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.733967 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" event={"ID":"4c812a3d-79c6-4060-be64-0da4dd1aa4d9","Type":"ContainerDied","Data":"4efd3a4198e1878507cfab6c9ef8bc3f5963d44c9585607d59af258fa55830ab"} Oct 30 19:53:21.734225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.734032 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn" event={"ID":"4c812a3d-79c6-4060-be64-0da4dd1aa4d9","Type":"ContainerDied","Data":"93210eff2fc8f6e638738400c584bb616ee0e3ecb626251b56dd8cab93489e07"} Oct 30 19:53:21.734225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.734080 5091 scope.go:122] "RemoveContainer" containerID="4efd3a4198e1878507cfab6c9ef8bc3f5963d44c9585607d59af258fa55830ab" Oct 30 19:53:21.747089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.747069 5091 scope.go:122] "RemoveContainer" containerID="4efd3a4198e1878507cfab6c9ef8bc3f5963d44c9585607d59af258fa55830ab" Oct 30 19:53:21.748190 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:53:21.748162 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4efd3a4198e1878507cfab6c9ef8bc3f5963d44c9585607d59af258fa55830ab\": container with ID starting with 4efd3a4198e1878507cfab6c9ef8bc3f5963d44c9585607d59af258fa55830ab not found: ID does not exist" containerID="4efd3a4198e1878507cfab6c9ef8bc3f5963d44c9585607d59af258fa55830ab" Oct 30 19:53:21.748273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.748198 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4efd3a4198e1878507cfab6c9ef8bc3f5963d44c9585607d59af258fa55830ab"} err="failed to get container status \"4efd3a4198e1878507cfab6c9ef8bc3f5963d44c9585607d59af258fa55830ab\": rpc error: code = NotFound desc = could not find container \"4efd3a4198e1878507cfab6c9ef8bc3f5963d44c9585607d59af258fa55830ab\": container with ID starting with 4efd3a4198e1878507cfab6c9ef8bc3f5963d44c9585607d59af258fa55830ab not found: ID does not exist" Oct 30 19:53:21.782279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.782196 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn"] Oct 30 19:53:21.785718 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:21.785690 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-426bab-7c4d75cb74-6v8gn"] Oct 30 19:53:22.164301 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:22.161490 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:22.442529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:22.442458 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c812a3d-79c6-4060-be64-0da4dd1aa4d9" path="/var/lib/kubelet/pods/4c812a3d-79c6-4060-be64-0da4dd1aa4d9/volumes" Oct 30 19:53:22.742351 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:22.742253 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:23.476893 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:23.476853 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:24.265137 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.265093 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:24.437395 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.437358 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:24.506458 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.506427 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r"] Oct 30 19:53:24.507292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.507272 5091 cpu_manager.go:414] "containerMap: removing container" podUID="192dd6ce-3921-41dc-ba51-be3b5535cb51" containerName="istio-proxy" Oct 30 19:53:24.507292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.507293 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="192dd6ce-3921-41dc-ba51-be3b5535cb51" containerName="istio-proxy" Oct 30 19:53:24.507493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.507308 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4c812a3d-79c6-4060-be64-0da4dd1aa4d9" containerName="payload-processing" Oct 30 19:53:24.507493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.507317 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c812a3d-79c6-4060-be64-0da4dd1aa4d9" containerName="payload-processing" Oct 30 19:53:24.507493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.507338 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1897963b-68eb-4959-b391-16b595122405" containerName="revoke-keys" Oct 30 19:53:24.507493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.507346 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1897963b-68eb-4959-b391-16b595122405" containerName="revoke-keys" Oct 30 19:53:24.507493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.507378 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" containerName="istio-proxy" Oct 30 19:53:24.507493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.507385 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" containerName="istio-proxy" Oct 30 19:53:24.507493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.507400 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" containerName="istio-proxy" Oct 30 19:53:24.507493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.507407 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" containerName="istio-proxy" Oct 30 19:53:24.507872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.507592 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" containerName="istio-proxy" Oct 30 19:53:24.507872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.507608 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae1f3e96-fdc1-43c1-a0d0-21585d0ed24b" containerName="istio-proxy" Oct 30 19:53:24.507872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.507630 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="192dd6ce-3921-41dc-ba51-be3b5535cb51" containerName="istio-proxy" Oct 30 19:53:24.507872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.507642 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c812a3d-79c6-4060-be64-0da4dd1aa4d9" containerName="payload-processing" Oct 30 19:53:24.507872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.507654 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1897963b-68eb-4959-b391-16b595122405" containerName="revoke-keys" Oct 30 19:53:24.513458 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.513436 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" Oct 30 19:53:24.528196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.528138 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-426bab\"" Oct 30 19:53:24.546989 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.546956 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glh7r\" (UniqueName: \"kubernetes.io/projected/5b77699f-9a88-4dde-94f0-feee3d4f01e8-kube-api-access-glh7r\") pod \"payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r\" (UID: \"5b77699f-9a88-4dde-94f0-feee3d4f01e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" Oct 30 19:53:24.547089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.547069 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b77699f-9a88-4dde-94f0-feee3d4f01e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r\" (UID: \"5b77699f-9a88-4dde-94f0-feee3d4f01e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" Oct 30 19:53:24.553660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.553638 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r"] Oct 30 19:53:24.569773 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.569745 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72"] Oct 30 19:53:24.577057 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.577033 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" Oct 30 19:53:24.588013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.587195 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72"] Oct 30 19:53:24.588013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.587411 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-426bab-dockercfg-s54bp\"" Oct 30 19:53:24.665167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.655233 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glh7r\" (UniqueName: \"kubernetes.io/projected/5b77699f-9a88-4dde-94f0-feee3d4f01e8-kube-api-access-glh7r\") pod \"payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r\" (UID: \"5b77699f-9a88-4dde-94f0-feee3d4f01e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" Oct 30 19:53:24.665167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.655350 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b77699f-9a88-4dde-94f0-feee3d4f01e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r\" (UID: \"5b77699f-9a88-4dde-94f0-feee3d4f01e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" Oct 30 19:53:24.665167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.656920 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b77699f-9a88-4dde-94f0-feee3d4f01e8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r\" (UID: \"5b77699f-9a88-4dde-94f0-feee3d4f01e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" Oct 30 19:53:24.694167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.677374 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:24.741176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.738233 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glh7r\" (UniqueName: \"kubernetes.io/projected/5b77699f-9a88-4dde-94f0-feee3d4f01e8-kube-api-access-glh7r\") pod \"payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r\" (UID: \"5b77699f-9a88-4dde-94f0-feee3d4f01e8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" Oct 30 19:53:24.779250 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.776646 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnljw\" (UniqueName: \"kubernetes.io/projected/da240f1c-8dd4-4205-9985-92f3e42cb8e5-kube-api-access-tnljw\") pod \"payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72\" (UID: \"da240f1c-8dd4-4205-9985-92f3e42cb8e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" Oct 30 19:53:24.779250 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.776773 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da240f1c-8dd4-4205-9985-92f3e42cb8e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72\" (UID: \"da240f1c-8dd4-4205-9985-92f3e42cb8e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" Oct 30 19:53:24.837170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.835016 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" Oct 30 19:53:24.906167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.900491 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnljw\" (UniqueName: \"kubernetes.io/projected/da240f1c-8dd4-4205-9985-92f3e42cb8e5-kube-api-access-tnljw\") pod \"payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72\" (UID: \"da240f1c-8dd4-4205-9985-92f3e42cb8e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" Oct 30 19:53:24.906167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.900639 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da240f1c-8dd4-4205-9985-92f3e42cb8e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72\" (UID: \"da240f1c-8dd4-4205-9985-92f3e42cb8e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" Oct 30 19:53:24.906167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.901770 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da240f1c-8dd4-4205-9985-92f3e42cb8e5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72\" (UID: \"da240f1c-8dd4-4205-9985-92f3e42cb8e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" Oct 30 19:53:24.916981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:24.916957 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnljw\" (UniqueName: \"kubernetes.io/projected/da240f1c-8dd4-4205-9985-92f3e42cb8e5-kube-api-access-tnljw\") pod \"payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72\" (UID: \"da240f1c-8dd4-4205-9985-92f3e42cb8e5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" Oct 30 19:53:25.200521 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:25.200443 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" Oct 30 19:53:25.212909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:25.212884 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r"] Oct 30 19:53:25.214768 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:53:25.214742 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b77699f_9a88_4dde_94f0_feee3d4f01e8.slice/crio-df7142542cf9195341cbedba985ccd363ef5b85986cd0637faba1b68f76ad882 WatchSource:0}: Error finding container df7142542cf9195341cbedba985ccd363ef5b85986cd0637faba1b68f76ad882: Status 404 returned error can't find the container with id df7142542cf9195341cbedba985ccd363ef5b85986cd0637faba1b68f76ad882 Oct 30 19:53:25.357902 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:25.339187 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:25.505890 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:53:25.505849 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda240f1c_8dd4_4205_9985_92f3e42cb8e5.slice/crio-9b89cb2b086334f06c6700d041c98d88920fe108b3c7d9654ad928a99c5b77a7 WatchSource:0}: Error finding container 9b89cb2b086334f06c6700d041c98d88920fe108b3c7d9654ad928a99c5b77a7: Status 404 returned error can't find the container with id 9b89cb2b086334f06c6700d041c98d88920fe108b3c7d9654ad928a99c5b77a7 Oct 30 19:53:25.506507 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:25.506471 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72"] Oct 30 19:53:25.828793 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:25.828711 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" event={"ID":"5b77699f-9a88-4dde-94f0-feee3d4f01e8","Type":"ContainerStarted","Data":"3987717a8f9ef800a474b3828c76e4c22bc17df4b76967d077406855ac11a595"} Oct 30 19:53:25.828793 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:25.828774 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" event={"ID":"5b77699f-9a88-4dde-94f0-feee3d4f01e8","Type":"ContainerStarted","Data":"df7142542cf9195341cbedba985ccd363ef5b85986cd0637faba1b68f76ad882"} Oct 30 19:53:25.828988 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:25.828807 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" Oct 30 19:53:25.830651 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:25.830626 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" event={"ID":"da240f1c-8dd4-4205-9985-92f3e42cb8e5","Type":"ContainerStarted","Data":"4e2fdd00bbdfbe9c5e0449416f8231c40db1019b1c9476ba07d41c6f44f56c70"} Oct 30 19:53:25.830803 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:25.830665 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" event={"ID":"da240f1c-8dd4-4205-9985-92f3e42cb8e5","Type":"ContainerStarted","Data":"9b89cb2b086334f06c6700d041c98d88920fe108b3c7d9654ad928a99c5b77a7"} Oct 30 19:53:25.830803 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:25.830701 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" Oct 30 19:53:25.925113 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:25.924941 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:25.926970 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:25.926897 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" podStartSLOduration=1.926876761 podStartE2EDuration="1.926876761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:25.890883002 +0000 UTC m=+1325.586090961" watchObservedRunningTime="2026-10-30 19:53:25.926876761 +0000 UTC m=+1325.622084720" Oct 30 19:53:25.936451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:25.936392 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" podStartSLOduration=1.936372643 podStartE2EDuration="1.936372643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:25.926791806 +0000 UTC m=+1325.621999764" watchObservedRunningTime="2026-10-30 19:53:25.936372643 +0000 UTC m=+1325.631580602" Oct 30 19:53:26.579021 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:26.573447 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:27.782175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:27.769240 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q"] Oct 30 19:53:27.782175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:27.775473 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" Oct 30 19:53:27.886179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:27.885302 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:53:27.886179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:27.885427 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:53:27.914916 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:27.914890 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q"] Oct 30 19:53:27.915260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:27.915234 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:27.939959 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:27.939932 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9355fe16-8af0-468e-a815-5119d7405e24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q\" (UID: \"9355fe16-8af0-468e-a815-5119d7405e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" Oct 30 19:53:27.940089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:27.940066 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9355fe16-8af0-468e-a815-5119d7405e24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q\" (UID: \"9355fe16-8af0-468e-a815-5119d7405e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" Oct 30 19:53:27.940292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:27.940272 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c98b6\" (UniqueName: \"kubernetes.io/projected/9355fe16-8af0-468e-a815-5119d7405e24-kube-api-access-c98b6\") pod \"maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q\" (UID: \"9355fe16-8af0-468e-a815-5119d7405e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" Oct 30 19:53:28.041945 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.041861 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c98b6\" (UniqueName: \"kubernetes.io/projected/9355fe16-8af0-468e-a815-5119d7405e24-kube-api-access-c98b6\") pod \"maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q\" (UID: \"9355fe16-8af0-468e-a815-5119d7405e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" Oct 30 19:53:28.042104 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.041988 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv"] Oct 30 19:53:28.042219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.042099 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9355fe16-8af0-468e-a815-5119d7405e24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q\" (UID: \"9355fe16-8af0-468e-a815-5119d7405e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" Oct 30 19:53:28.042902 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.042880 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9355fe16-8af0-468e-a815-5119d7405e24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q\" (UID: \"9355fe16-8af0-468e-a815-5119d7405e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" Oct 30 19:53:28.043122 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.043101 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9355fe16-8af0-468e-a815-5119d7405e24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q\" (UID: \"9355fe16-8af0-468e-a815-5119d7405e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" Oct 30 19:53:28.060175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.045320 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9355fe16-8af0-468e-a815-5119d7405e24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q\" (UID: \"9355fe16-8af0-468e-a815-5119d7405e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" Oct 30 19:53:28.066165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.064528 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b00c018-hvtjv"] Oct 30 19:53:28.091978 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.091234 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c98b6\" (UniqueName: \"kubernetes.io/projected/9355fe16-8af0-468e-a815-5119d7405e24-kube-api-access-c98b6\") pod \"maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q\" (UID: \"9355fe16-8af0-468e-a815-5119d7405e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" Oct 30 19:53:28.236090 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.236054 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" Oct 30 19:53:28.443588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.443559 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="273c236a-1a5d-4313-a7dc-1836307afe35" path="/var/lib/kubelet/pods/273c236a-1a5d-4313-a7dc-1836307afe35/volumes" Oct 30 19:53:28.540981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.540949 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx"] Oct 30 19:53:28.575492 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.575410 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" Oct 30 19:53:28.595036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.595003 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx"] Oct 30 19:53:28.601110 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.601083 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q"] Oct 30 19:53:28.608346 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:53:28.607696 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9355fe16_8af0_468e_a815_5119d7405e24.slice/crio-7764305d244d9992ca59c71f1dc8c4bf16fb132a3ab398646a2a6793f233659c WatchSource:0}: Error finding container 7764305d244d9992ca59c71f1dc8c4bf16fb132a3ab398646a2a6793f233659c: Status 404 returned error can't find the container with id 7764305d244d9992ca59c71f1dc8c4bf16fb132a3ab398646a2a6793f233659c Oct 30 19:53:28.694513 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.694466 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f72cc0-a860-47ae-90ae-2b15bbcb4135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6786e6d-6zznx\" (UID: \"90f72cc0-a860-47ae-90ae-2b15bbcb4135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" Oct 30 19:53:28.694709 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.694619 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f72cc0-a860-47ae-90ae-2b15bbcb4135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6786e6d-6zznx\" (UID: \"90f72cc0-a860-47ae-90ae-2b15bbcb4135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" Oct 30 19:53:28.694798 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.694778 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84kjg\" (UniqueName: \"kubernetes.io/projected/90f72cc0-a860-47ae-90ae-2b15bbcb4135-kube-api-access-84kjg\") pod \"maas-api-revoke-sub-e6786e6d-6zznx\" (UID: \"90f72cc0-a860-47ae-90ae-2b15bbcb4135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" Oct 30 19:53:28.796606 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.796498 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84kjg\" (UniqueName: \"kubernetes.io/projected/90f72cc0-a860-47ae-90ae-2b15bbcb4135-kube-api-access-84kjg\") pod \"maas-api-revoke-sub-e6786e6d-6zznx\" (UID: \"90f72cc0-a860-47ae-90ae-2b15bbcb4135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" Oct 30 19:53:28.797081 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.796746 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f72cc0-a860-47ae-90ae-2b15bbcb4135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6786e6d-6zznx\" (UID: \"90f72cc0-a860-47ae-90ae-2b15bbcb4135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" Oct 30 19:53:28.797081 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.796823 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f72cc0-a860-47ae-90ae-2b15bbcb4135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6786e6d-6zznx\" (UID: \"90f72cc0-a860-47ae-90ae-2b15bbcb4135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" Oct 30 19:53:28.797744 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.797717 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f72cc0-a860-47ae-90ae-2b15bbcb4135-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e6786e6d-6zznx\" (UID: \"90f72cc0-a860-47ae-90ae-2b15bbcb4135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" Oct 30 19:53:28.879211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.877583 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f72cc0-a860-47ae-90ae-2b15bbcb4135-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e6786e6d-6zznx\" (UID: \"90f72cc0-a860-47ae-90ae-2b15bbcb4135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" Oct 30 19:53:28.885178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.880964 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-84kjg\" (UniqueName: \"kubernetes.io/projected/90f72cc0-a860-47ae-90ae-2b15bbcb4135-kube-api-access-84kjg\") pod \"maas-api-revoke-sub-e6786e6d-6zznx\" (UID: \"90f72cc0-a860-47ae-90ae-2b15bbcb4135\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" Oct 30 19:53:28.907243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.907210 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" Oct 30 19:53:28.930124 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.930092 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" event={"ID":"9355fe16-8af0-468e-a815-5119d7405e24","Type":"ContainerStarted","Data":"7764305d244d9992ca59c71f1dc8c4bf16fb132a3ab398646a2a6793f233659c"} Oct 30 19:53:28.984057 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:28.984017 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:29.342126 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:53:29.342085 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90f72cc0_a860_47ae_90ae_2b15bbcb4135.slice/crio-a9d194b1b3fb10caeac041920415245c39ccaf41b33ae2453427c6fe29ed3fad WatchSource:0}: Error finding container a9d194b1b3fb10caeac041920415245c39ccaf41b33ae2453427c6fe29ed3fad: Status 404 returned error can't find the container with id a9d194b1b3fb10caeac041920415245c39ccaf41b33ae2453427c6fe29ed3fad Oct 30 19:53:29.343097 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:29.343076 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx"] Oct 30 19:53:29.904116 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:29.902120 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:29.976916 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:29.976879 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" event={"ID":"90f72cc0-a860-47ae-90ae-2b15bbcb4135","Type":"ContainerStarted","Data":"a9d194b1b3fb10caeac041920415245c39ccaf41b33ae2453427c6fe29ed3fad"} Oct 30 19:53:29.985389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:29.985358 5091 generic.go:356] "Generic (PLEG): container finished" podID="9355fe16-8af0-468e-a815-5119d7405e24" containerID="8695f41b72b9de9d21ab6105adffc56d91ccda4a08cd29afc67bf457ad39bf9f" exitCode=0 Oct 30 19:53:29.985784 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:29.985765 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" event={"ID":"9355fe16-8af0-468e-a815-5119d7405e24","Type":"ContainerDied","Data":"8695f41b72b9de9d21ab6105adffc56d91ccda4a08cd29afc67bf457ad39bf9f"} Oct 30 19:53:30.888232 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:30.888105 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:30.997503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:30.997472 5091 generic.go:356] "Generic (PLEG): container finished" podID="90f72cc0-a860-47ae-90ae-2b15bbcb4135" containerID="2f6c13daaa37583a71a3a31386ee21021f9047d4b2ffb26a3d71b4e6c6b0f79e" exitCode=0 Oct 30 19:53:30.997980 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:30.997553 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" event={"ID":"90f72cc0-a860-47ae-90ae-2b15bbcb4135","Type":"ContainerDied","Data":"2f6c13daaa37583a71a3a31386ee21021f9047d4b2ffb26a3d71b4e6c6b0f79e"} Oct 30 19:53:31.172177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:31.172131 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" Oct 30 19:53:31.238421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:31.238377 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9355fe16-8af0-468e-a815-5119d7405e24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9355fe16-8af0-468e-a815-5119d7405e24-maas-api-openshift-ca\") pod \"9355fe16-8af0-468e-a815-5119d7405e24\" (UID: \"9355fe16-8af0-468e-a815-5119d7405e24\") " Oct 30 19:53:31.238598 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:31.238529 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9355fe16-8af0-468e-a815-5119d7405e24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9355fe16-8af0-468e-a815-5119d7405e24-maas-api-xks-ca\") pod \"9355fe16-8af0-468e-a815-5119d7405e24\" (UID: \"9355fe16-8af0-468e-a815-5119d7405e24\") " Oct 30 19:53:31.238840 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:31.238811 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9355fe16-8af0-468e-a815-5119d7405e24-maas-api-openshift-ca" pod "9355fe16-8af0-468e-a815-5119d7405e24" (UID: "9355fe16-8af0-468e-a815-5119d7405e24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:31.239208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:31.239184 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9355fe16-8af0-468e-a815-5119d7405e24-kube-api-access-c98b6\" (UniqueName: \"kubernetes.io/projected/9355fe16-8af0-468e-a815-5119d7405e24-kube-api-access-c98b6\") pod \"9355fe16-8af0-468e-a815-5119d7405e24\" (UID: \"9355fe16-8af0-468e-a815-5119d7405e24\") " Oct 30 19:53:31.239991 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:31.239969 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9355fe16-8af0-468e-a815-5119d7405e24-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:31.240830 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:31.240799 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9355fe16-8af0-468e-a815-5119d7405e24-maas-api-xks-ca" pod "9355fe16-8af0-468e-a815-5119d7405e24" (UID: "9355fe16-8af0-468e-a815-5119d7405e24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:31.241893 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:31.241870 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9355fe16-8af0-468e-a815-5119d7405e24-kube-api-access-c98b6" pod "9355fe16-8af0-468e-a815-5119d7405e24" (UID: "9355fe16-8af0-468e-a815-5119d7405e24"). InnerVolumeSpecName "kube-api-access-c98b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:31.341540 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:31.341506 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c98b6\" (UniqueName: \"kubernetes.io/projected/9355fe16-8af0-468e-a815-5119d7405e24-kube-api-access-c98b6\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:31.341540 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:31.341537 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9355fe16-8af0-468e-a815-5119d7405e24-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:31.899174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:31.897468 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:32.007091 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.007059 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" Oct 30 19:53:32.007548 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.007083 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q" event={"ID":"9355fe16-8af0-468e-a815-5119d7405e24","Type":"ContainerDied","Data":"7764305d244d9992ca59c71f1dc8c4bf16fb132a3ab398646a2a6793f233659c"} Oct 30 19:53:32.007548 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.007182 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7764305d244d9992ca59c71f1dc8c4bf16fb132a3ab398646a2a6793f233659c" Oct 30 19:53:32.199329 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.199300 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" Oct 30 19:53:32.255561 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.255520 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90f72cc0-a860-47ae-90ae-2b15bbcb4135-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f72cc0-a860-47ae-90ae-2b15bbcb4135-maas-api-xks-ca\") pod \"90f72cc0-a860-47ae-90ae-2b15bbcb4135\" (UID: \"90f72cc0-a860-47ae-90ae-2b15bbcb4135\") " Oct 30 19:53:32.255748 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.255685 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90f72cc0-a860-47ae-90ae-2b15bbcb4135-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f72cc0-a860-47ae-90ae-2b15bbcb4135-maas-api-openshift-ca\") pod \"90f72cc0-a860-47ae-90ae-2b15bbcb4135\" (UID: \"90f72cc0-a860-47ae-90ae-2b15bbcb4135\") " Oct 30 19:53:32.255951 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.255931 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90f72cc0-a860-47ae-90ae-2b15bbcb4135-kube-api-access-84kjg\" (UniqueName: \"kubernetes.io/projected/90f72cc0-a860-47ae-90ae-2b15bbcb4135-kube-api-access-84kjg\") pod \"90f72cc0-a860-47ae-90ae-2b15bbcb4135\" (UID: \"90f72cc0-a860-47ae-90ae-2b15bbcb4135\") " Oct 30 19:53:32.256717 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.256686 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90f72cc0-a860-47ae-90ae-2b15bbcb4135-maas-api-openshift-ca" pod "90f72cc0-a860-47ae-90ae-2b15bbcb4135" (UID: "90f72cc0-a860-47ae-90ae-2b15bbcb4135"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:32.260867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.260828 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90f72cc0-a860-47ae-90ae-2b15bbcb4135-maas-api-xks-ca" pod "90f72cc0-a860-47ae-90ae-2b15bbcb4135" (UID: "90f72cc0-a860-47ae-90ae-2b15bbcb4135"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:32.261194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.261161 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90f72cc0-a860-47ae-90ae-2b15bbcb4135-kube-api-access-84kjg" pod "90f72cc0-a860-47ae-90ae-2b15bbcb4135" (UID: "90f72cc0-a860-47ae-90ae-2b15bbcb4135"). InnerVolumeSpecName "kube-api-access-84kjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:32.268650 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.267410 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:32.358638 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.358599 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84kjg\" (UniqueName: \"kubernetes.io/projected/90f72cc0-a860-47ae-90ae-2b15bbcb4135-kube-api-access-84kjg\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:32.358638 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.358632 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90f72cc0-a860-47ae-90ae-2b15bbcb4135-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:32.358638 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.358643 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90f72cc0-a860-47ae-90ae-2b15bbcb4135-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:32.627176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.627116 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:32.774622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.774191 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf"] Oct 30 19:53:32.906179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:32.906061 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:33.057756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:33.054909 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" podUID="4b3c46ef-f755-4462-9dae-250f052538b5" containerName="istio-proxy" containerID="cri-o://9dc97d61a4bcdbf458a1bcb4cf618b3df493cef760db9c27b9d36f19a3470dc7" gracePeriod=30 Oct 30 19:53:33.057756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:33.055361 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" Oct 30 19:53:33.057756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:33.056129 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx" event={"ID":"90f72cc0-a860-47ae-90ae-2b15bbcb4135","Type":"ContainerDied","Data":"a9d194b1b3fb10caeac041920415245c39ccaf41b33ae2453427c6fe29ed3fad"} Oct 30 19:53:33.057756 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:33.056196 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9d194b1b3fb10caeac041920415245c39ccaf41b33ae2453427c6fe29ed3fad" Oct 30 19:53:33.085185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:33.083562 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:34.191474 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:34.190643 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:35.364084 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:35.361819 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:36.081931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:36.081899 5091 generic.go:356] "Generic (PLEG): container finished" podID="1ec5bb43-1784-459b-87b0-a7bff9143031" containerID="a7002857e0db2a31df496bacaa0b755a2b7029a85cc7edc7108d4cf5469309e2" exitCode=0 Oct 30 19:53:36.082124 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:36.081980 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" event={"ID":"1ec5bb43-1784-459b-87b0-a7bff9143031","Type":"ContainerDied","Data":"a7002857e0db2a31df496bacaa0b755a2b7029a85cc7edc7108d4cf5469309e2"} Oct 30 19:53:36.082580 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:36.082561 5091 scope.go:122] "RemoveContainer" containerID="a7002857e0db2a31df496bacaa0b755a2b7029a85cc7edc7108d4cf5469309e2" Oct 30 19:53:36.084207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:36.084179 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:53:36.084327 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:36.084247 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:53:36.084327 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:36.084284 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:53:36.325166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:36.317701 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:36.841857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:36.841827 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" Oct 30 19:53:36.842281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:36.841903 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" Oct 30 19:53:37.094722 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:37.094636 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" event={"ID":"1ec5bb43-1784-459b-87b0-a7bff9143031","Type":"ContainerStarted","Data":"7f151bda3ad0db35995ad38cacb1c3216ee1c29c9cb43a7c7f22aa4b612dd27e"} Oct 30 19:53:37.496268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:37.494535 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:37.673171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:37.672613 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:53:37.676183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:37.674229 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 19:53:37.676183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:37.674306 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" podUID="1ec5bb43-1784-459b-87b0-a7bff9143031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 19:53:37.878708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:37.878339 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:37.896681 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:37.896650 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m"] Oct 30 19:53:37.919972 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:37.919946 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z"] Oct 30 19:53:38.043109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.043077 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px"] Oct 30 19:53:38.043791 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.043774 5091 cpu_manager.go:414] "containerMap: removing container" podUID="90f72cc0-a860-47ae-90ae-2b15bbcb4135" containerName="revoke-keys" Oct 30 19:53:38.043849 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.043791 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90f72cc0-a860-47ae-90ae-2b15bbcb4135" containerName="revoke-keys" Oct 30 19:53:38.043849 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.043821 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9355fe16-8af0-468e-a815-5119d7405e24" containerName="revoke-keys" Oct 30 19:53:38.043849 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.043826 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9355fe16-8af0-468e-a815-5119d7405e24" containerName="revoke-keys" Oct 30 19:53:38.043947 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.043916 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9355fe16-8af0-468e-a815-5119d7405e24" containerName="revoke-keys" Oct 30 19:53:38.043947 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.043934 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="90f72cc0-a860-47ae-90ae-2b15bbcb4135" containerName="revoke-keys" Oct 30 19:53:38.082605 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.082561 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" Oct 30 19:53:38.083406 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.083377 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px"] Oct 30 19:53:38.092051 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.092017 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:53:38.092638 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.092603 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:53:38.133235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.131648 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:38.205354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.205321 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px\" (UID: \"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" Oct 30 19:53:38.205485 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.205415 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh6q9\" (UniqueName: \"kubernetes.io/projected/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-kube-api-access-wh6q9\") pod \"maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px\" (UID: \"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" Oct 30 19:53:38.205485 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.205467 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px\" (UID: \"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" Oct 30 19:53:38.292894 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.289984 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth"] Oct 30 19:53:38.295345 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.295320 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.310724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.310252 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-podinfo\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.310724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.310327 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-workload-certs\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.310724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.310376 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px\" (UID: \"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" Oct 30 19:53:38.310724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.310443 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-data\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.310724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.310473 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-workload-socket\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.310724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.310504 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh6q9\" (UniqueName: \"kubernetes.io/projected/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-kube-api-access-wh6q9\") pod \"maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px\" (UID: \"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" Oct 30 19:53:38.310724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.310535 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-token\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.310724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.310574 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px\" (UID: \"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" Oct 30 19:53:38.310724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.310673 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cnsk\" (UniqueName: \"kubernetes.io/projected/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-kube-api-access-2cnsk\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.311298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.310747 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istiod-ca-cert\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.311298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.310837 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-credential-socket\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.311298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.310863 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-envoy\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.317590 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.312181 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px\" (UID: \"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" Oct 30 19:53:38.322672 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.322217 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth"] Oct 30 19:53:38.330225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.330180 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px\" (UID: \"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" Oct 30 19:53:38.331008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.330965 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-55a8e6-openshift-default-dockercfg-rrpt4\"" Oct 30 19:53:38.390914 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.390829 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh6q9\" (UniqueName: \"kubernetes.io/projected/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-kube-api-access-wh6q9\") pod \"maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px\" (UID: \"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" Oct 30 19:53:38.434254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.419248 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-credential-socket\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.434254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.419292 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-envoy\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.434254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.419361 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-podinfo\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.434254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.419405 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-workload-certs\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.434254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.419484 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-data\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.434254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.419515 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-workload-socket\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.434254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.419555 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-token\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.434254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.419662 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cnsk\" (UniqueName: \"kubernetes.io/projected/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-kube-api-access-2cnsk\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.434254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.419735 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istiod-ca-cert\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.434254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.420555 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istiod-ca-cert\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.434254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.420830 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-credential-socket\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.434254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.421474 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-data\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.507785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.449844 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-workload-socket\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.507785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.463858 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:38.507785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.465721 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-workload-certs\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.507785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.482251 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" Oct 30 19:53:38.570585 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.551790 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-token\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.723866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.711507 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 19:53:38.723866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.711598 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" podUID="1ec5bb43-1784-459b-87b0-a7bff9143031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 19:53:38.742651 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.742569 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cnsk\" (UniqueName: \"kubernetes.io/projected/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-kube-api-access-2cnsk\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.760861 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.760255 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-podinfo\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.765547 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.765380 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:38.799257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.796436 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-envoy\") pod \"e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:38.830196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.829810 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:53:38.916729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.902904 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:53:38.916729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.905940 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3c46ef-f755-4462-9dae-250f052538b5-kube-api-access-tgrk8\" (UniqueName: \"kubernetes.io/projected/4b3c46ef-f755-4462-9dae-250f052538b5-kube-api-access-tgrk8\") pod \"4b3c46ef-f755-4462-9dae-250f052538b5\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " Oct 30 19:53:38.916729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.906041 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-istio-envoy\") pod \"4b3c46ef-f755-4462-9dae-250f052538b5\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " Oct 30 19:53:38.916729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.906125 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b3c46ef-f755-4462-9dae-250f052538b5-istio-token\" (UniqueName: \"kubernetes.io/projected/4b3c46ef-f755-4462-9dae-250f052538b5-istio-token\") pod \"4b3c46ef-f755-4462-9dae-250f052538b5\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " Oct 30 19:53:38.916729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.915372 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-istio-data\") pod \"4b3c46ef-f755-4462-9dae-250f052538b5\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " Oct 30 19:53:38.916729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.915449 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-credential-socket\") pod \"4b3c46ef-f755-4462-9dae-250f052538b5\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " Oct 30 19:53:38.916729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.915505 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b3c46ef-f755-4462-9dae-250f052538b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3c46ef-f755-4462-9dae-250f052538b5-istiod-ca-cert\") pod \"4b3c46ef-f755-4462-9dae-250f052538b5\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " Oct 30 19:53:38.916729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.915551 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4b3c46ef-f755-4462-9dae-250f052538b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3c46ef-f755-4462-9dae-250f052538b5-istio-podinfo\") pod \"4b3c46ef-f755-4462-9dae-250f052538b5\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " Oct 30 19:53:38.916729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.915614 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-workload-socket\") pod \"4b3c46ef-f755-4462-9dae-250f052538b5\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " Oct 30 19:53:38.916729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.915669 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-workload-certs\") pod \"4b3c46ef-f755-4462-9dae-250f052538b5\" (UID: \"4b3c46ef-f755-4462-9dae-250f052538b5\") " Oct 30 19:53:38.916729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.916460 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-istio-data" pod "4b3c46ef-f755-4462-9dae-250f052538b5" (UID: "4b3c46ef-f755-4462-9dae-250f052538b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:38.916729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.916470 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-workload-certs" pod "4b3c46ef-f755-4462-9dae-250f052538b5" (UID: "4b3c46ef-f755-4462-9dae-250f052538b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:38.916729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.916657 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-credential-socket" pod "4b3c46ef-f755-4462-9dae-250f052538b5" (UID: "4b3c46ef-f755-4462-9dae-250f052538b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:38.918323 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.916830 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-workload-socket" pod "4b3c46ef-f755-4462-9dae-250f052538b5" (UID: "4b3c46ef-f755-4462-9dae-250f052538b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:38.918323 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.917091 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b3c46ef-f755-4462-9dae-250f052538b5-istiod-ca-cert" pod "4b3c46ef-f755-4462-9dae-250f052538b5" (UID: "4b3c46ef-f755-4462-9dae-250f052538b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:38.918323 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.917918 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3c46ef-f755-4462-9dae-250f052538b5-kube-api-access-tgrk8" pod "4b3c46ef-f755-4462-9dae-250f052538b5" (UID: "4b3c46ef-f755-4462-9dae-250f052538b5"). InnerVolumeSpecName "kube-api-access-tgrk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:38.920532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.920502 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-istio-envoy" pod "4b3c46ef-f755-4462-9dae-250f052538b5" (UID: "4b3c46ef-f755-4462-9dae-250f052538b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:53:38.923685 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.923650 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b3c46ef-f755-4462-9dae-250f052538b5-istio-token" pod "4b3c46ef-f755-4462-9dae-250f052538b5" (UID: "4b3c46ef-f755-4462-9dae-250f052538b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:38.923954 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.923929 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4b3c46ef-f755-4462-9dae-250f052538b5-istio-podinfo" pod "4b3c46ef-f755-4462-9dae-250f052538b5" (UID: "4b3c46ef-f755-4462-9dae-250f052538b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:53:38.927475 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:38.927189 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:39.018870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.018829 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:39.018870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.018872 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:39.019071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.018888 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4b3c46ef-f755-4462-9dae-250f052538b5-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:39.019071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.018901 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4b3c46ef-f755-4462-9dae-250f052538b5-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:39.019071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.018915 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:39.019071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.018930 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:39.019071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.018944 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgrk8\" (UniqueName: \"kubernetes.io/projected/4b3c46ef-f755-4462-9dae-250f052538b5-kube-api-access-tgrk8\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:39.019071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.018959 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4b3c46ef-f755-4462-9dae-250f052538b5-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:39.019071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.018971 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4b3c46ef-f755-4462-9dae-250f052538b5-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:39.095554 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.095093 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85"] Oct 30 19:53:39.099762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.099683 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36edca7e-6xv85"] Oct 30 19:53:39.103429 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.103406 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px"] Oct 30 19:53:39.123386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.123177 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" event={"ID":"4b3c46ef-f755-4462-9dae-250f052538b5","Type":"ContainerDied","Data":"9dc97d61a4bcdbf458a1bcb4cf618b3df493cef760db9c27b9d36f19a3470dc7"} Oct 30 19:53:39.123386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.123255 5091 scope.go:122] "RemoveContainer" containerID="9dc97d61a4bcdbf458a1bcb4cf618b3df493cef760db9c27b9d36f19a3470dc7" Oct 30 19:53:39.123386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.123183 5091 generic.go:356] "Generic (PLEG): container finished" podID="4b3c46ef-f755-4462-9dae-250f052538b5" containerID="9dc97d61a4bcdbf458a1bcb4cf618b3df493cef760db9c27b9d36f19a3470dc7" exitCode=0 Oct 30 19:53:39.123386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.123357 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" event={"ID":"4b3c46ef-f755-4462-9dae-250f052538b5","Type":"ContainerDied","Data":"2e0efa3d75bb18cd67e617a23eae1250c366f185ee4214e919c8e14e32395387"} Oct 30 19:53:39.124726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.123734 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf" Oct 30 19:53:39.126377 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.126343 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" event={"ID":"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0","Type":"ContainerStarted","Data":"4ef2aef1b6337611b38e0ff21ca358c62c11c6c0c53b3ff21ababbe9f36b67d0"} Oct 30 19:53:39.235271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.183893 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:39.278732 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.278689 5091 scope.go:122] "RemoveContainer" containerID="0a674f1877120eb81273084b00af32f9b0af4bd6298e08853ded33b74a2e5af4" Oct 30 19:53:39.291125 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.290720 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth"] Oct 30 19:53:39.295311 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:53:39.295121 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed9893a_07bd_4fb4_8a22_2c55b0d122bc.slice/crio-af3e415e14c7227f2543c6aaec44fd92661891685fe319669219548d1b29c0e1 WatchSource:0}: Error finding container af3e415e14c7227f2543c6aaec44fd92661891685fe319669219548d1b29c0e1: Status 404 returned error can't find the container with id af3e415e14c7227f2543c6aaec44fd92661891685fe319669219548d1b29c0e1 Oct 30 19:53:39.298729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.298693 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:53:39.299199 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.299179 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:53:39.299343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.299328 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:53:39.305727 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.305691 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf"] Oct 30 19:53:39.311062 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.311042 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-913351-openshift-default-579dcdf795-xcgmf"] Oct 30 19:53:39.317710 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.317678 5091 scope.go:122] "RemoveContainer" containerID="9dc97d61a4bcdbf458a1bcb4cf618b3df493cef760db9c27b9d36f19a3470dc7" Oct 30 19:53:39.317960 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:53:39.317940 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc97d61a4bcdbf458a1bcb4cf618b3df493cef760db9c27b9d36f19a3470dc7\": container with ID starting with 9dc97d61a4bcdbf458a1bcb4cf618b3df493cef760db9c27b9d36f19a3470dc7 not found: ID does not exist" containerID="9dc97d61a4bcdbf458a1bcb4cf618b3df493cef760db9c27b9d36f19a3470dc7" Oct 30 19:53:39.318031 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.317968 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc97d61a4bcdbf458a1bcb4cf618b3df493cef760db9c27b9d36f19a3470dc7"} err="failed to get container status \"9dc97d61a4bcdbf458a1bcb4cf618b3df493cef760db9c27b9d36f19a3470dc7\": rpc error: code = NotFound desc = could not find container \"9dc97d61a4bcdbf458a1bcb4cf618b3df493cef760db9c27b9d36f19a3470dc7\": container with ID starting with 9dc97d61a4bcdbf458a1bcb4cf618b3df493cef760db9c27b9d36f19a3470dc7 not found: ID does not exist" Oct 30 19:53:39.318031 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.317987 5091 scope.go:122] "RemoveContainer" containerID="0a674f1877120eb81273084b00af32f9b0af4bd6298e08853ded33b74a2e5af4" Oct 30 19:53:39.318225 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:53:39.318209 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a674f1877120eb81273084b00af32f9b0af4bd6298e08853ded33b74a2e5af4\": container with ID starting with 0a674f1877120eb81273084b00af32f9b0af4bd6298e08853ded33b74a2e5af4 not found: ID does not exist" containerID="0a674f1877120eb81273084b00af32f9b0af4bd6298e08853ded33b74a2e5af4" Oct 30 19:53:39.318286 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.318228 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a674f1877120eb81273084b00af32f9b0af4bd6298e08853ded33b74a2e5af4"} err="failed to get container status \"0a674f1877120eb81273084b00af32f9b0af4bd6298e08853ded33b74a2e5af4\": rpc error: code = NotFound desc = could not find container \"0a674f1877120eb81273084b00af32f9b0af4bd6298e08853ded33b74a2e5af4\": container with ID starting with 0a674f1877120eb81273084b00af32f9b0af4bd6298e08853ded33b74a2e5af4 not found: ID does not exist" Oct 30 19:53:39.381168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.379853 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:39.673201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.673167 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 19:53:39.673464 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:39.673435 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" podUID="1ec5bb43-1784-459b-87b0-a7bff9143031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 19:53:40.140306 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:40.140268 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" event={"ID":"fed9893a-07bd-4fb4-8a22-2c55b0d122bc","Type":"ContainerStarted","Data":"462df2709f0731a0695283434955e2ac9857ad2062b17d2fdd08554e82865560"} Oct 30 19:53:40.140682 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:40.140364 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" event={"ID":"fed9893a-07bd-4fb4-8a22-2c55b0d122bc","Type":"ContainerStarted","Data":"af3e415e14c7227f2543c6aaec44fd92661891685fe319669219548d1b29c0e1"} Oct 30 19:53:40.142017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:40.141991 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" event={"ID":"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0","Type":"ContainerStarted","Data":"a373363934285a0ee559a6768851e533d4f61bfdde92622be57b03d57ae5e8cc"} Oct 30 19:53:40.223176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:40.223080 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" podStartSLOduration=2.223065263 podStartE2EDuration="2.223065263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:40.221367762 +0000 UTC m=+1339.916575721" watchObservedRunningTime="2026-10-30 19:53:40.223065263 +0000 UTC m=+1339.918273222" Oct 30 19:53:40.225509 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:40.225459 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" podStartSLOduration=2.225447659 podStartE2EDuration="2.225447659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:40.186921817 +0000 UTC m=+1339.882129769" watchObservedRunningTime="2026-10-30 19:53:40.225447659 +0000 UTC m=+1339.920655621" Oct 30 19:53:40.427104 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:40.426925 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:40.505619 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:40.505590 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e9f9dd1-39a8-419f-bccc-38a670d0786b" path="/var/lib/kubelet/pods/3e9f9dd1-39a8-419f-bccc-38a670d0786b/volumes" Oct 30 19:53:40.507632 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:40.507602 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b3c46ef-f755-4462-9dae-250f052538b5" path="/var/lib/kubelet/pods/4b3c46ef-f755-4462-9dae-250f052538b5/volumes" Oct 30 19:53:40.678185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:40.677604 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 19:53:40.678185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:40.677668 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" podUID="1ec5bb43-1784-459b-87b0-a7bff9143031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 19:53:40.927808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:40.927772 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:40.933216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:40.933131 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:41.179189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.178513 5091 generic.go:356] "Generic (PLEG): container finished" podID="ab201154-6cf4-4cf0-b3b3-fe58a3700ee0" containerID="a373363934285a0ee559a6768851e533d4f61bfdde92622be57b03d57ae5e8cc" exitCode=0 Oct 30 19:53:41.179580 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.179475 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" event={"ID":"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0","Type":"ContainerDied","Data":"a373363934285a0ee559a6768851e533d4f61bfdde92622be57b03d57ae5e8cc"} Oct 30 19:53:41.180112 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.180094 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:41.180963 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.180945 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:53:41.416963 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.400602 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:41.675223 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.673460 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Oct 30 19:53:41.675223 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.673518 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" podUID="1ec5bb43-1784-459b-87b0-a7bff9143031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Oct 30 19:53:41.794999 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.794963 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z"] Oct 30 19:53:41.795796 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.795776 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4b3c46ef-f755-4462-9dae-250f052538b5" containerName="istio-proxy" Oct 30 19:53:41.795796 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.795796 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3c46ef-f755-4462-9dae-250f052538b5" containerName="istio-proxy" Oct 30 19:53:41.795934 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.795897 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3c46ef-f755-4462-9dae-250f052538b5" containerName="istio-proxy" Oct 30 19:53:41.799842 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.799825 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" Oct 30 19:53:41.816180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.808231 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z"] Oct 30 19:53:41.904440 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.904404 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea59a6da-e105-4dd4-9e37-2041ca75cffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e1e28af-dmg8z\" (UID: \"ea59a6da-e105-4dd4-9e37-2041ca75cffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" Oct 30 19:53:41.904440 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.904442 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl7bv\" (UniqueName: \"kubernetes.io/projected/ea59a6da-e105-4dd4-9e37-2041ca75cffc-kube-api-access-zl7bv\") pod \"maas-api-revoke-sub-8e1e28af-dmg8z\" (UID: \"ea59a6da-e105-4dd4-9e37-2041ca75cffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" Oct 30 19:53:41.904708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:41.904688 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea59a6da-e105-4dd4-9e37-2041ca75cffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e1e28af-dmg8z\" (UID: \"ea59a6da-e105-4dd4-9e37-2041ca75cffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" Oct 30 19:53:42.006534 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.006488 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea59a6da-e105-4dd4-9e37-2041ca75cffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e1e28af-dmg8z\" (UID: \"ea59a6da-e105-4dd4-9e37-2041ca75cffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" Oct 30 19:53:42.007190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.007164 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea59a6da-e105-4dd4-9e37-2041ca75cffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e1e28af-dmg8z\" (UID: \"ea59a6da-e105-4dd4-9e37-2041ca75cffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" Oct 30 19:53:42.007261 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.007204 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl7bv\" (UniqueName: \"kubernetes.io/projected/ea59a6da-e105-4dd4-9e37-2041ca75cffc-kube-api-access-zl7bv\") pod \"maas-api-revoke-sub-8e1e28af-dmg8z\" (UID: \"ea59a6da-e105-4dd4-9e37-2041ca75cffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" Oct 30 19:53:42.007405 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.007383 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea59a6da-e105-4dd4-9e37-2041ca75cffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8e1e28af-dmg8z\" (UID: \"ea59a6da-e105-4dd4-9e37-2041ca75cffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" Oct 30 19:53:42.011787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.011757 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea59a6da-e105-4dd4-9e37-2041ca75cffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8e1e28af-dmg8z\" (UID: \"ea59a6da-e105-4dd4-9e37-2041ca75cffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" Oct 30 19:53:42.024553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.024519 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl7bv\" (UniqueName: \"kubernetes.io/projected/ea59a6da-e105-4dd4-9e37-2041ca75cffc-kube-api-access-zl7bv\") pod \"maas-api-revoke-sub-8e1e28af-dmg8z\" (UID: \"ea59a6da-e105-4dd4-9e37-2041ca75cffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" Oct 30 19:53:42.118401 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.118356 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" Oct 30 19:53:42.435345 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.435312 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" Oct 30 19:53:42.504714 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.504673 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z"] Oct 30 19:53:42.554467 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.554431 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:42.673591 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.673562 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-kube-api-access-wh6q9\" (UniqueName: \"kubernetes.io/projected/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-kube-api-access-wh6q9\") pod \"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0\" (UID: \"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0\") " Oct 30 19:53:42.673767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.673626 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-maas-api-openshift-ca\") pod \"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0\" (UID: \"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0\") " Oct 30 19:53:42.673767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.673662 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-maas-api-xks-ca\") pod \"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0\" (UID: \"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0\") " Oct 30 19:53:42.674059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.673998 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-maas-api-openshift-ca" pod "ab201154-6cf4-4cf0-b3b3-fe58a3700ee0" (UID: "ab201154-6cf4-4cf0-b3b3-fe58a3700ee0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:42.674430 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.674377 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:42.676203 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.676072 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-maas-api-xks-ca" pod "ab201154-6cf4-4cf0-b3b3-fe58a3700ee0" (UID: "ab201154-6cf4-4cf0-b3b3-fe58a3700ee0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:42.677945 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.677917 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:53:42.678325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.678296 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-kube-api-access-wh6q9" pod "ab201154-6cf4-4cf0-b3b3-fe58a3700ee0" (UID: "ab201154-6cf4-4cf0-b3b3-fe58a3700ee0"). InnerVolumeSpecName "kube-api-access-wh6q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:42.679986 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.679953 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:53:42.728072 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.728039 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:42.776867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.776831 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh6q9\" (UniqueName: \"kubernetes.io/projected/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-kube-api-access-wh6q9\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:42.776867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.776868 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:42.882943 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.882904 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:42.897479 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.897445 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" podUID="afb6f324-f820-459e-9419-5a89e5b59e6f" containerName="payload-processing" containerID="cri-o://7eda6418d319208bcdf7b372b303f5aca4ce90f70a1bc3d2d2cd0e5ee428ca10" gracePeriod=25 Oct 30 19:53:42.920931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:42.920898 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" podUID="fe2c9b06-2e9f-4a52-9f31-48e043063a1d" containerName="payload-pre-processing" containerID="cri-o://21c497742ea62d23a8a62bb517145505dc6f54aa7db646ce77b9d6330faf2a70" gracePeriod=25 Oct 30 19:53:43.203989 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.203953 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" event={"ID":"ab201154-6cf4-4cf0-b3b3-fe58a3700ee0","Type":"ContainerDied","Data":"4ef2aef1b6337611b38e0ff21ca358c62c11c6c0c53b3ff21ababbe9f36b67d0"} Oct 30 19:53:43.204195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.204008 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4ef2aef1b6337611b38e0ff21ca358c62c11c6c0c53b3ff21ababbe9f36b67d0" Oct 30 19:53:43.204195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.204082 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px" Oct 30 19:53:43.212097 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.212006 5091 generic.go:356] "Generic (PLEG): container finished" podID="afb6f324-f820-459e-9419-5a89e5b59e6f" containerID="7eda6418d319208bcdf7b372b303f5aca4ce90f70a1bc3d2d2cd0e5ee428ca10" exitCode=0 Oct 30 19:53:43.212497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.212139 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" event={"ID":"afb6f324-f820-459e-9419-5a89e5b59e6f","Type":"ContainerDied","Data":"7eda6418d319208bcdf7b372b303f5aca4ce90f70a1bc3d2d2cd0e5ee428ca10"} Oct 30 19:53:43.212497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.212269 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" event={"ID":"afb6f324-f820-459e-9419-5a89e5b59e6f","Type":"ContainerDied","Data":"b036e5ffcc019144dcd79b037a1edcfbe5e7b91fcf2a86fb3d7f4d6310690ed4"} Oct 30 19:53:43.212497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.212312 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b036e5ffcc019144dcd79b037a1edcfbe5e7b91fcf2a86fb3d7f4d6310690ed4" Oct 30 19:53:43.214452 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.214428 5091 generic.go:356] "Generic (PLEG): container finished" podID="fe2c9b06-2e9f-4a52-9f31-48e043063a1d" containerID="21c497742ea62d23a8a62bb517145505dc6f54aa7db646ce77b9d6330faf2a70" exitCode=0 Oct 30 19:53:43.214572 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.214506 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" event={"ID":"fe2c9b06-2e9f-4a52-9f31-48e043063a1d","Type":"ContainerDied","Data":"21c497742ea62d23a8a62bb517145505dc6f54aa7db646ce77b9d6330faf2a70"} Oct 30 19:53:43.218237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.216886 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" event={"ID":"ea59a6da-e105-4dd4-9e37-2041ca75cffc","Type":"ContainerStarted","Data":"9cfd0aaab7d69f0d22a7ef768663e3cc4723dff94f7bfc9361c8220daf9fa388"} Oct 30 19:53:43.218237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.216950 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" event={"ID":"ea59a6da-e105-4dd4-9e37-2041ca75cffc","Type":"ContainerStarted","Data":"07ae139c63b3effddb4706361ca469fab534960b2c9b45756920dacaf679841f"} Oct 30 19:53:43.223494 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.223475 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" Oct 30 19:53:43.243483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.243433 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" podStartSLOduration=2.243417683 podStartE2EDuration="2.243417683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:43.239527909 +0000 UTC m=+1342.934735868" watchObservedRunningTime="2026-10-30 19:53:43.243417683 +0000 UTC m=+1342.938625638" Oct 30 19:53:43.286225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.286184 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afb6f324-f820-459e-9419-5a89e5b59e6f-kube-api-access-whbmh\" (UniqueName: \"kubernetes.io/projected/afb6f324-f820-459e-9419-5a89e5b59e6f-kube-api-access-whbmh\") pod \"afb6f324-f820-459e-9419-5a89e5b59e6f\" (UID: \"afb6f324-f820-459e-9419-5a89e5b59e6f\") " Oct 30 19:53:43.286394 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.286242 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afb6f324-f820-459e-9419-5a89e5b59e6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb6f324-f820-459e-9419-5a89e5b59e6f-plugins-config-volume\") pod \"afb6f324-f820-459e-9419-5a89e5b59e6f\" (UID: \"afb6f324-f820-459e-9419-5a89e5b59e6f\") " Oct 30 19:53:43.287321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.287283 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb6f324-f820-459e-9419-5a89e5b59e6f-plugins-config-volume" pod "afb6f324-f820-459e-9419-5a89e5b59e6f" (UID: "afb6f324-f820-459e-9419-5a89e5b59e6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:43.288622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.288534 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afb6f324-f820-459e-9419-5a89e5b59e6f-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:43.289273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.289252 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" Oct 30 19:53:43.289457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.289433 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afb6f324-f820-459e-9419-5a89e5b59e6f-kube-api-access-whbmh" pod "afb6f324-f820-459e-9419-5a89e5b59e6f" (UID: "afb6f324-f820-459e-9419-5a89e5b59e6f"). InnerVolumeSpecName "kube-api-access-whbmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:43.389244 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.389158 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe2c9b06-2e9f-4a52-9f31-48e043063a1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2c9b06-2e9f-4a52-9f31-48e043063a1d-plugins-config-volume\") pod \"fe2c9b06-2e9f-4a52-9f31-48e043063a1d\" (UID: \"fe2c9b06-2e9f-4a52-9f31-48e043063a1d\") " Oct 30 19:53:43.389418 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.389246 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe2c9b06-2e9f-4a52-9f31-48e043063a1d-kube-api-access-9ht5c\" (UniqueName: \"kubernetes.io/projected/fe2c9b06-2e9f-4a52-9f31-48e043063a1d-kube-api-access-9ht5c\") pod \"fe2c9b06-2e9f-4a52-9f31-48e043063a1d\" (UID: \"fe2c9b06-2e9f-4a52-9f31-48e043063a1d\") " Oct 30 19:53:43.389549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.389521 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe2c9b06-2e9f-4a52-9f31-48e043063a1d-plugins-config-volume" pod "fe2c9b06-2e9f-4a52-9f31-48e043063a1d" (UID: "fe2c9b06-2e9f-4a52-9f31-48e043063a1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:43.389834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.389808 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe2c9b06-2e9f-4a52-9f31-48e043063a1d-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:43.389834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.389828 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whbmh\" (UniqueName: \"kubernetes.io/projected/afb6f324-f820-459e-9419-5a89e5b59e6f-kube-api-access-whbmh\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:43.391453 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.391428 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe2c9b06-2e9f-4a52-9f31-48e043063a1d-kube-api-access-9ht5c" pod "fe2c9b06-2e9f-4a52-9f31-48e043063a1d" (UID: "fe2c9b06-2e9f-4a52-9f31-48e043063a1d"). InnerVolumeSpecName "kube-api-access-9ht5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:43.491944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.491909 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ht5c\" (UniqueName: \"kubernetes.io/projected/fe2c9b06-2e9f-4a52-9f31-48e043063a1d-kube-api-access-9ht5c\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:43.595717 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.595667 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:43.934673 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.934639 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:43.968364 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.968333 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5"] Oct 30 19:53:43.969064 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.969049 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4b3c46ef-f755-4462-9dae-250f052538b5" containerName="istio-proxy" Oct 30 19:53:43.969116 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.969066 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b3c46ef-f755-4462-9dae-250f052538b5" containerName="istio-proxy" Oct 30 19:53:43.969191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.969137 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fe2c9b06-2e9f-4a52-9f31-48e043063a1d" containerName="payload-pre-processing" Oct 30 19:53:43.969191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.969155 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe2c9b06-2e9f-4a52-9f31-48e043063a1d" containerName="payload-pre-processing" Oct 30 19:53:43.969191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.969171 5091 cpu_manager.go:414] "containerMap: removing container" podUID="afb6f324-f820-459e-9419-5a89e5b59e6f" containerName="payload-processing" Oct 30 19:53:43.969191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.969176 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afb6f324-f820-459e-9419-5a89e5b59e6f" containerName="payload-processing" Oct 30 19:53:43.969191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.969185 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ab201154-6cf4-4cf0-b3b3-fe58a3700ee0" containerName="revoke-keys" Oct 30 19:53:43.969191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.969190 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab201154-6cf4-4cf0-b3b3-fe58a3700ee0" containerName="revoke-keys" Oct 30 19:53:43.969403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.969288 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe2c9b06-2e9f-4a52-9f31-48e043063a1d" containerName="payload-pre-processing" Oct 30 19:53:43.969403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.969297 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="afb6f324-f820-459e-9419-5a89e5b59e6f" containerName="payload-processing" Oct 30 19:53:43.969403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.969303 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab201154-6cf4-4cf0-b3b3-fe58a3700ee0" containerName="revoke-keys" Oct 30 19:53:43.969403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.969312 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b3c46ef-f755-4462-9dae-250f052538b5" containerName="istio-proxy" Oct 30 19:53:43.973332 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.973308 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" Oct 30 19:53:43.986235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.986208 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-55a8e6\"" Oct 30 19:53:43.992491 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.992466 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5"] Oct 30 19:53:43.998063 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:43.995366 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-55a8e6-dockercfg-88z4r\"" Oct 30 19:53:44.000230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.000200 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45941330-4133-4795-ae8b-cd3dea49ceb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5\" (UID: \"45941330-4133-4795-ae8b-cd3dea49ceb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" Oct 30 19:53:44.000512 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.000490 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drn6h\" (UniqueName: \"kubernetes.io/projected/45941330-4133-4795-ae8b-cd3dea49ceb9-kube-api-access-drn6h\") pod \"payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5\" (UID: \"45941330-4133-4795-ae8b-cd3dea49ceb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" Oct 30 19:53:44.027392 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.027365 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww"] Oct 30 19:53:44.033140 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.033119 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" Oct 30 19:53:44.040054 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.040031 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww"] Oct 30 19:53:44.092702 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.087911 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:44.151197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.150029 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drn6h\" (UniqueName: \"kubernetes.io/projected/45941330-4133-4795-ae8b-cd3dea49ceb9-kube-api-access-drn6h\") pod \"payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5\" (UID: \"45941330-4133-4795-ae8b-cd3dea49ceb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" Oct 30 19:53:44.151197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.150129 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45941330-4133-4795-ae8b-cd3dea49ceb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5\" (UID: \"45941330-4133-4795-ae8b-cd3dea49ceb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" Oct 30 19:53:44.151197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.150191 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdd4s\" (UniqueName: \"kubernetes.io/projected/ca4eff0e-4814-46a5-9822-92d297027447-kube-api-access-vdd4s\") pod \"payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww\" (UID: \"ca4eff0e-4814-46a5-9822-92d297027447\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" Oct 30 19:53:44.151197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.150277 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4eff0e-4814-46a5-9822-92d297027447-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww\" (UID: \"ca4eff0e-4814-46a5-9822-92d297027447\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" Oct 30 19:53:44.168402 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.151806 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45941330-4133-4795-ae8b-cd3dea49ceb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5\" (UID: \"45941330-4133-4795-ae8b-cd3dea49ceb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" Oct 30 19:53:44.237241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.225085 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drn6h\" (UniqueName: \"kubernetes.io/projected/45941330-4133-4795-ae8b-cd3dea49ceb9-kube-api-access-drn6h\") pod \"payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5\" (UID: \"45941330-4133-4795-ae8b-cd3dea49ceb9\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" Oct 30 19:53:44.297800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.297420 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" Oct 30 19:53:44.298883 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.298795 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdd4s\" (UniqueName: \"kubernetes.io/projected/ca4eff0e-4814-46a5-9822-92d297027447-kube-api-access-vdd4s\") pod \"payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww\" (UID: \"ca4eff0e-4814-46a5-9822-92d297027447\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" Oct 30 19:53:44.299575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.299549 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4eff0e-4814-46a5-9822-92d297027447-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww\" (UID: \"ca4eff0e-4814-46a5-9822-92d297027447\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" Oct 30 19:53:44.301000 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.300726 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4eff0e-4814-46a5-9822-92d297027447-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww\" (UID: \"ca4eff0e-4814-46a5-9822-92d297027447\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" Oct 30 19:53:44.335374 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.335308 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdd4s\" (UniqueName: \"kubernetes.io/projected/ca4eff0e-4814-46a5-9822-92d297027447-kube-api-access-vdd4s\") pod \"payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww\" (UID: \"ca4eff0e-4814-46a5-9822-92d297027447\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" Oct 30 19:53:44.345434 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.341682 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" event={"ID":"fe2c9b06-2e9f-4a52-9f31-48e043063a1d","Type":"ContainerDied","Data":"cf768937739fddcf83aadf3765dae30a184afc1fce6ca1879e78df850c6cc525"} Oct 30 19:53:44.345434 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.341750 5091 scope.go:122] "RemoveContainer" containerID="21c497742ea62d23a8a62bb517145505dc6f54aa7db646ce77b9d6330faf2a70" Oct 30 19:53:44.345434 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.341910 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z" Oct 30 19:53:44.352776 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.352746 5091 generic.go:356] "Generic (PLEG): container finished" podID="ea59a6da-e105-4dd4-9e37-2041ca75cffc" containerID="9cfd0aaab7d69f0d22a7ef768663e3cc4723dff94f7bfc9361c8220daf9fa388" exitCode=0 Oct 30 19:53:44.353051 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.353038 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m" Oct 30 19:53:44.353371 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.353337 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" event={"ID":"ea59a6da-e105-4dd4-9e37-2041ca75cffc","Type":"ContainerDied","Data":"9cfd0aaab7d69f0d22a7ef768663e3cc4723dff94f7bfc9361c8220daf9fa388"} Oct 30 19:53:44.361579 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.361551 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" Oct 30 19:53:44.413905 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.413877 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m"] Oct 30 19:53:44.422025 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.421997 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-490cdf-6d869df5dd-kqf6m"] Oct 30 19:53:44.479640 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.479597 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afb6f324-f820-459e-9419-5a89e5b59e6f" path="/var/lib/kubelet/pods/afb6f324-f820-459e-9419-5a89e5b59e6f/volumes" Oct 30 19:53:44.480112 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.480090 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z"] Oct 30 19:53:44.480241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.480133 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-490cdf-6cf955d547-b8s7z"] Oct 30 19:53:44.502330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.502265 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:44.634648 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.634613 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5"] Oct 30 19:53:44.717800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.717761 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:44.835244 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.835206 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww"] Oct 30 19:53:44.989912 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.989878 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q"] Oct 30 19:53:44.999639 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:44.999606 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-490cdf-nh72q"] Oct 30 19:53:45.070105 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.070028 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:45.236406 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.236370 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:45.364204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.364067 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" event={"ID":"45941330-4133-4795-ae8b-cd3dea49ceb9","Type":"ContainerStarted","Data":"36c7afdc0da2e88e3cddfdf028520f09fcdac90d0d92553b5f98c9920e595d28"} Oct 30 19:53:45.364204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.364136 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" event={"ID":"45941330-4133-4795-ae8b-cd3dea49ceb9","Type":"ContainerStarted","Data":"6657bdbc98444c57c4953b6d3ae2ad3923692939c19d6a55af5e45fa4a3cdb31"} Oct 30 19:53:45.364448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.364320 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" Oct 30 19:53:45.367319 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.367290 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" event={"ID":"ca4eff0e-4814-46a5-9822-92d297027447","Type":"ContainerStarted","Data":"d8759005fce25d08f63ac131e1232856e4218dcd420201d11f5a04270f2ba1e7"} Oct 30 19:53:45.367452 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.367355 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" event={"ID":"ca4eff0e-4814-46a5-9822-92d297027447","Type":"ContainerStarted","Data":"4da96060a319a26da351497af44e0a3aee9e63e13283c854a3efa445a6eba92e"} Oct 30 19:53:45.367495 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.367458 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" Oct 30 19:53:45.389664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.389609 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" podStartSLOduration=2.389591972 podStartE2EDuration="2.389591972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:45.387936381 +0000 UTC m=+1345.083144334" watchObservedRunningTime="2026-10-30 19:53:45.389591972 +0000 UTC m=+1345.084799931" Oct 30 19:53:45.420403 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.420355 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" podStartSLOduration=1.420341982 podStartE2EDuration="1.420341982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:45.417365348 +0000 UTC m=+1345.112573297" watchObservedRunningTime="2026-10-30 19:53:45.420341982 +0000 UTC m=+1345.115549984" Oct 30 19:53:45.510495 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.510465 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:45.598168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.594579 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" Oct 30 19:53:45.683680 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.683594 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea59a6da-e105-4dd4-9e37-2041ca75cffc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea59a6da-e105-4dd4-9e37-2041ca75cffc-maas-api-openshift-ca\") pod \"ea59a6da-e105-4dd4-9e37-2041ca75cffc\" (UID: \"ea59a6da-e105-4dd4-9e37-2041ca75cffc\") " Oct 30 19:53:45.683680 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.683631 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea59a6da-e105-4dd4-9e37-2041ca75cffc-kube-api-access-zl7bv\" (UniqueName: \"kubernetes.io/projected/ea59a6da-e105-4dd4-9e37-2041ca75cffc-kube-api-access-zl7bv\") pod \"ea59a6da-e105-4dd4-9e37-2041ca75cffc\" (UID: \"ea59a6da-e105-4dd4-9e37-2041ca75cffc\") " Oct 30 19:53:45.683680 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.683664 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea59a6da-e105-4dd4-9e37-2041ca75cffc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea59a6da-e105-4dd4-9e37-2041ca75cffc-maas-api-xks-ca\") pod \"ea59a6da-e105-4dd4-9e37-2041ca75cffc\" (UID: \"ea59a6da-e105-4dd4-9e37-2041ca75cffc\") " Oct 30 19:53:45.684024 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.683997 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea59a6da-e105-4dd4-9e37-2041ca75cffc-maas-api-openshift-ca" pod "ea59a6da-e105-4dd4-9e37-2041ca75cffc" (UID: "ea59a6da-e105-4dd4-9e37-2041ca75cffc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:45.684436 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.684416 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea59a6da-e105-4dd4-9e37-2041ca75cffc-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:45.686347 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.686320 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea59a6da-e105-4dd4-9e37-2041ca75cffc-maas-api-xks-ca" pod "ea59a6da-e105-4dd4-9e37-2041ca75cffc" (UID: "ea59a6da-e105-4dd4-9e37-2041ca75cffc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:45.686688 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.686662 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea59a6da-e105-4dd4-9e37-2041ca75cffc-kube-api-access-zl7bv" pod "ea59a6da-e105-4dd4-9e37-2041ca75cffc" (UID: "ea59a6da-e105-4dd4-9e37-2041ca75cffc"). InnerVolumeSpecName "kube-api-access-zl7bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:45.786022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.785982 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zl7bv\" (UniqueName: \"kubernetes.io/projected/ea59a6da-e105-4dd4-9e37-2041ca75cffc-kube-api-access-zl7bv\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:45.786022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:45.786022 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea59a6da-e105-4dd4-9e37-2041ca75cffc-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:45.978871 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:53:45.978672 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:53:46.149198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:46.149116 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:46.376712 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:46.376683 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" Oct 30 19:53:46.376891 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:46.376715 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z" event={"ID":"ea59a6da-e105-4dd4-9e37-2041ca75cffc","Type":"ContainerDied","Data":"07ae139c63b3effddb4706361ca469fab534960b2c9b45756920dacaf679841f"} Oct 30 19:53:46.376891 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:46.376796 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07ae139c63b3effddb4706361ca469fab534960b2c9b45756920dacaf679841f" Oct 30 19:53:46.443104 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:46.443072 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9355fe16-8af0-468e-a815-5119d7405e24" path="/var/lib/kubelet/pods/9355fe16-8af0-468e-a815-5119d7405e24/volumes" Oct 30 19:53:46.443641 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:46.443616 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe2c9b06-2e9f-4a52-9f31-48e043063a1d" path="/var/lib/kubelet/pods/fe2c9b06-2e9f-4a52-9f31-48e043063a1d/volumes" Oct 30 19:53:47.485770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:47.485736 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:49.023589 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:49.023543 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:50.041220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:50.041184 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:51.080339 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:51.080302 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:52.619190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:52.619138 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:53.886172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:53.880931 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:54.606784 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:54.606738 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:54.774883 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:54.774833 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:54.910237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:54.908384 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd"] Oct 30 19:53:54.910237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:54.908629 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" containerName="istio-proxy" containerID="cri-o://507e988d2c08dcc131c57776eb8ea5a11f3ff5807057dc6b9400759936ac5f40" gracePeriod=30 Oct 30 19:53:54.936530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:54.936494 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs"] Oct 30 19:53:54.937282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:54.937260 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ea59a6da-e105-4dd4-9e37-2041ca75cffc" containerName="revoke-keys" Oct 30 19:53:54.937282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:54.937281 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea59a6da-e105-4dd4-9e37-2041ca75cffc" containerName="revoke-keys" Oct 30 19:53:54.937525 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:54.937511 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea59a6da-e105-4dd4-9e37-2041ca75cffc" containerName="revoke-keys" Oct 30 19:53:54.942430 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:54.942408 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" Oct 30 19:53:54.948932 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:54.948910 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:53:54.948932 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:54.948920 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:53:54.959268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:54.959244 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs"] Oct 30 19:53:54.972871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:54.972830 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:55.046197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.016631 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6cdeda8-9cvrs\" (UID: \"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" Oct 30 19:53:55.046197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.016878 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vzbf\" (UniqueName: \"kubernetes.io/projected/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-kube-api-access-7vzbf\") pod \"maas-api-revoke-sub-c6cdeda8-9cvrs\" (UID: \"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" Oct 30 19:53:55.046197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.016954 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6cdeda8-9cvrs\" (UID: \"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" Oct 30 19:53:55.129174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.121103 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vzbf\" (UniqueName: \"kubernetes.io/projected/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-kube-api-access-7vzbf\") pod \"maas-api-revoke-sub-c6cdeda8-9cvrs\" (UID: \"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" Oct 30 19:53:55.151721 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.150654 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6cdeda8-9cvrs\" (UID: \"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" Oct 30 19:53:55.152074 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.151678 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6cdeda8-9cvrs\" (UID: \"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" Oct 30 19:53:55.152341 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.152320 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6cdeda8-9cvrs\" (UID: \"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" Oct 30 19:53:55.155910 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.155882 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:55.197993 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.188609 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6cdeda8-9cvrs\" (UID: \"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" Oct 30 19:53:55.197993 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.197196 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vzbf\" (UniqueName: \"kubernetes.io/projected/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-kube-api-access-7vzbf\") pod \"maas-api-revoke-sub-c6cdeda8-9cvrs\" (UID: \"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" Oct 30 19:53:55.261358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.261326 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" Oct 30 19:53:55.429090 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.429058 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b"] Oct 30 19:53:55.434577 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.434557 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" Oct 30 19:53:55.458408 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.444925 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b"] Oct 30 19:53:55.549141 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:53:55.549101 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8f83d61_a4f4_4a99_a021_2ca09dcb1dcf.slice/crio-d7f94bd1b67149cb1d97aecb8236e4b107a7b2e74edebb645265a9dc0c093ce0 WatchSource:0}: Error finding container d7f94bd1b67149cb1d97aecb8236e4b107a7b2e74edebb645265a9dc0c093ce0: Status 404 returned error can't find the container with id d7f94bd1b67149cb1d97aecb8236e4b107a7b2e74edebb645265a9dc0c093ce0 Oct 30 19:53:55.549318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.549203 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs"] Oct 30 19:53:55.561019 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.560991 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmn6z\" (UniqueName: \"kubernetes.io/projected/9b219df2-601d-4e3f-bf24-d32a34b67ee6-kube-api-access-lmn6z\") pod \"maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b\" (UID: \"9b219df2-601d-4e3f-bf24-d32a34b67ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" Oct 30 19:53:55.561207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.561190 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b219df2-601d-4e3f-bf24-d32a34b67ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b\" (UID: \"9b219df2-601d-4e3f-bf24-d32a34b67ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" Oct 30 19:53:55.561326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.561304 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b219df2-601d-4e3f-bf24-d32a34b67ee6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b\" (UID: \"9b219df2-601d-4e3f-bf24-d32a34b67ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" Oct 30 19:53:55.662913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.662880 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmn6z\" (UniqueName: \"kubernetes.io/projected/9b219df2-601d-4e3f-bf24-d32a34b67ee6-kube-api-access-lmn6z\") pod \"maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b\" (UID: \"9b219df2-601d-4e3f-bf24-d32a34b67ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" Oct 30 19:53:55.663063 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.662964 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b219df2-601d-4e3f-bf24-d32a34b67ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b\" (UID: \"9b219df2-601d-4e3f-bf24-d32a34b67ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" Oct 30 19:53:55.663063 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.663011 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b219df2-601d-4e3f-bf24-d32a34b67ee6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b\" (UID: \"9b219df2-601d-4e3f-bf24-d32a34b67ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" Oct 30 19:53:55.663802 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.663768 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b219df2-601d-4e3f-bf24-d32a34b67ee6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b\" (UID: \"9b219df2-601d-4e3f-bf24-d32a34b67ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" Oct 30 19:53:55.665366 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.665343 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b219df2-601d-4e3f-bf24-d32a34b67ee6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b\" (UID: \"9b219df2-601d-4e3f-bf24-d32a34b67ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" Oct 30 19:53:55.676981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.676955 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmn6z\" (UniqueName: \"kubernetes.io/projected/9b219df2-601d-4e3f-bf24-d32a34b67ee6-kube-api-access-lmn6z\") pod \"maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b\" (UID: \"9b219df2-601d-4e3f-bf24-d32a34b67ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" Oct 30 19:53:55.769248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:55.769221 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" Oct 30 19:53:56.066025 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:53:56.065996 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b219df2_601d_4e3f_bf24_d32a34b67ee6.slice/crio-31103b43929e6bcc18af629263c276ed0783d8e16a573a019ce551c738c1a68e WatchSource:0}: Error finding container 31103b43929e6bcc18af629263c276ed0783d8e16a573a019ce551c738c1a68e: Status 404 returned error can't find the container with id 31103b43929e6bcc18af629263c276ed0783d8e16a573a019ce551c738c1a68e Oct 30 19:53:56.067186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:56.067158 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b"] Oct 30 19:53:56.101085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:56.101044 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:56.318587 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:56.318491 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:56.380374 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:56.380349 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" Oct 30 19:53:56.380631 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:56.380616 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" Oct 30 19:53:56.495795 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:56.495645 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" event={"ID":"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf","Type":"ContainerStarted","Data":"9b71f996462a070b6c207d1672118d220a7325f45f9d409244761dda777fd1c5"} Oct 30 19:53:56.495795 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:56.495706 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" event={"ID":"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf","Type":"ContainerStarted","Data":"d7f94bd1b67149cb1d97aecb8236e4b107a7b2e74edebb645265a9dc0c093ce0"} Oct 30 19:53:56.501638 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:56.501606 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" event={"ID":"9b219df2-601d-4e3f-bf24-d32a34b67ee6","Type":"ContainerStarted","Data":"cee45743584c77e8a1873446ad66f99f5de40faaca1782e6d4b53e0dce13da23"} Oct 30 19:53:56.501828 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:56.501670 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" event={"ID":"9b219df2-601d-4e3f-bf24-d32a34b67ee6","Type":"ContainerStarted","Data":"31103b43929e6bcc18af629263c276ed0783d8e16a573a019ce551c738c1a68e"} Oct 30 19:53:56.523081 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:56.523037 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" podStartSLOduration=2.523024039 podStartE2EDuration="2.523024039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:56.520018764 +0000 UTC m=+1356.215226725" watchObservedRunningTime="2026-10-30 19:53:56.523024039 +0000 UTC m=+1356.218232019" Oct 30 19:53:56.549383 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:56.549333 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" podStartSLOduration=1.549316261 podStartE2EDuration="1.549316261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:53:56.543513254 +0000 UTC m=+1356.238721229" watchObservedRunningTime="2026-10-30 19:53:56.549316261 +0000 UTC m=+1356.244524220" Oct 30 19:53:56.566068 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:56.566042 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:53:56.566202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:56.566096 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:53:57.187208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:57.185759 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:57.521355 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:57.521320 5091 generic.go:356] "Generic (PLEG): container finished" podID="9b219df2-601d-4e3f-bf24-d32a34b67ee6" containerID="cee45743584c77e8a1873446ad66f99f5de40faaca1782e6d4b53e0dce13da23" exitCode=0 Oct 30 19:53:57.521536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:57.521398 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" event={"ID":"9b219df2-601d-4e3f-bf24-d32a34b67ee6","Type":"ContainerDied","Data":"cee45743584c77e8a1873446ad66f99f5de40faaca1782e6d4b53e0dce13da23"} Oct 30 19:53:57.523369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:57.523342 5091 generic.go:356] "Generic (PLEG): container finished" podID="a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf" containerID="9b71f996462a070b6c207d1672118d220a7325f45f9d409244761dda777fd1c5" exitCode=0 Oct 30 19:53:57.523485 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:57.523407 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" event={"ID":"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf","Type":"ContainerDied","Data":"9b71f996462a070b6c207d1672118d220a7325f45f9d409244761dda777fd1c5"} Oct 30 19:53:58.072813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.070914 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:58.756878 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.756856 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" Oct 30 19:53:58.759892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.759869 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" Oct 30 19:53:58.839192 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.839098 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-maas-api-openshift-ca\") pod \"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf\" (UID: \"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf\") " Oct 30 19:53:58.839192 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.839140 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b219df2-601d-4e3f-bf24-d32a34b67ee6-kube-api-access-lmn6z\" (UniqueName: \"kubernetes.io/projected/9b219df2-601d-4e3f-bf24-d32a34b67ee6-kube-api-access-lmn6z\") pod \"9b219df2-601d-4e3f-bf24-d32a34b67ee6\" (UID: \"9b219df2-601d-4e3f-bf24-d32a34b67ee6\") " Oct 30 19:53:58.839408 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.839279 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b219df2-601d-4e3f-bf24-d32a34b67ee6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b219df2-601d-4e3f-bf24-d32a34b67ee6-maas-api-xks-ca\") pod \"9b219df2-601d-4e3f-bf24-d32a34b67ee6\" (UID: \"9b219df2-601d-4e3f-bf24-d32a34b67ee6\") " Oct 30 19:53:58.839408 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.839369 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-maas-api-xks-ca\") pod \"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf\" (UID: \"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf\") " Oct 30 19:53:58.839530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.839438 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b219df2-601d-4e3f-bf24-d32a34b67ee6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b219df2-601d-4e3f-bf24-d32a34b67ee6-maas-api-openshift-ca\") pod \"9b219df2-601d-4e3f-bf24-d32a34b67ee6\" (UID: \"9b219df2-601d-4e3f-bf24-d32a34b67ee6\") " Oct 30 19:53:58.839530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.839495 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-maas-api-openshift-ca" pod "a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf" (UID: "a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:58.839530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.839510 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-kube-api-access-7vzbf\" (UniqueName: \"kubernetes.io/projected/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-kube-api-access-7vzbf\") pod \"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf\" (UID: \"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf\") " Oct 30 19:53:58.839910 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.839834 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b219df2-601d-4e3f-bf24-d32a34b67ee6-maas-api-openshift-ca" pod "9b219df2-601d-4e3f-bf24-d32a34b67ee6" (UID: "9b219df2-601d-4e3f-bf24-d32a34b67ee6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:53:58.840103 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.840084 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b219df2-601d-4e3f-bf24-d32a34b67ee6-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:58.840206 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.840109 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:58.841621 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.841595 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b219df2-601d-4e3f-bf24-d32a34b67ee6-maas-api-xks-ca" pod "9b219df2-601d-4e3f-bf24-d32a34b67ee6" (UID: "9b219df2-601d-4e3f-bf24-d32a34b67ee6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:58.841723 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.841706 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-kube-api-access-7vzbf" pod "a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf" (UID: "a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf"). InnerVolumeSpecName "kube-api-access-7vzbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:58.841797 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.841781 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-maas-api-xks-ca" pod "a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf" (UID: "a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:53:58.842011 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.841996 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b219df2-601d-4e3f-bf24-d32a34b67ee6-kube-api-access-lmn6z" pod "9b219df2-601d-4e3f-bf24-d32a34b67ee6" (UID: "9b219df2-601d-4e3f-bf24-d32a34b67ee6"). InnerVolumeSpecName "kube-api-access-lmn6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:53:58.955657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.951042 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vzbf\" (UniqueName: \"kubernetes.io/projected/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-kube-api-access-7vzbf\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:58.955657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.951091 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmn6z\" (UniqueName: \"kubernetes.io/projected/9b219df2-601d-4e3f-bf24-d32a34b67ee6-kube-api-access-lmn6z\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:58.955657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.951109 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b219df2-601d-4e3f-bf24-d32a34b67ee6-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:58.955657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.951126 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:53:58.982166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:58.974221 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:53:59.540106 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:59.540075 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" Oct 30 19:53:59.540320 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:59.540110 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b" event={"ID":"9b219df2-601d-4e3f-bf24-d32a34b67ee6","Type":"ContainerDied","Data":"31103b43929e6bcc18af629263c276ed0783d8e16a573a019ce551c738c1a68e"} Oct 30 19:53:59.540320 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:59.540174 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31103b43929e6bcc18af629263c276ed0783d8e16a573a019ce551c738c1a68e" Oct 30 19:53:59.541597 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:59.541575 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" event={"ID":"a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf","Type":"ContainerDied","Data":"d7f94bd1b67149cb1d97aecb8236e4b107a7b2e74edebb645265a9dc0c093ce0"} Oct 30 19:53:59.541743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:59.541609 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d7f94bd1b67149cb1d97aecb8236e4b107a7b2e74edebb645265a9dc0c093ce0" Oct 30 19:53:59.541743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:53:59.541647 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs" Oct 30 19:54:00.135787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.135745 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:00.213508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.213452 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:54:00.371925 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.371884 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-credential-socket\") pod \"0b09a84c-ae99-434b-8562-2a921c923e33\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " Oct 30 19:54:00.372103 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.371969 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b09a84c-ae99-434b-8562-2a921c923e33-kube-api-access-b4xf4\" (UniqueName: \"kubernetes.io/projected/0b09a84c-ae99-434b-8562-2a921c923e33-kube-api-access-b4xf4\") pod \"0b09a84c-ae99-434b-8562-2a921c923e33\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " Oct 30 19:54:00.372103 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.372094 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b09a84c-ae99-434b-8562-2a921c923e33-istio-token\" (UniqueName: \"kubernetes.io/projected/0b09a84c-ae99-434b-8562-2a921c923e33-istio-token\") pod \"0b09a84c-ae99-434b-8562-2a921c923e33\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " Oct 30 19:54:00.372264 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.372131 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-workload-certs\") pod \"0b09a84c-ae99-434b-8562-2a921c923e33\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " Oct 30 19:54:00.372264 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.372188 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0b09a84c-ae99-434b-8562-2a921c923e33-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b09a84c-ae99-434b-8562-2a921c923e33-istio-podinfo\") pod \"0b09a84c-ae99-434b-8562-2a921c923e33\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " Oct 30 19:54:00.372264 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.372220 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b09a84c-ae99-434b-8562-2a921c923e33-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b09a84c-ae99-434b-8562-2a921c923e33-istiod-ca-cert\") pod \"0b09a84c-ae99-434b-8562-2a921c923e33\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " Oct 30 19:54:00.372264 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.372235 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-credential-socket" pod "0b09a84c-ae99-434b-8562-2a921c923e33" (UID: "0b09a84c-ae99-434b-8562-2a921c923e33"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:00.372473 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.372315 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-istio-envoy\") pod \"0b09a84c-ae99-434b-8562-2a921c923e33\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " Oct 30 19:54:00.372473 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.372425 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-workload-socket\") pod \"0b09a84c-ae99-434b-8562-2a921c923e33\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " Oct 30 19:54:00.372473 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.372430 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-workload-certs" pod "0b09a84c-ae99-434b-8562-2a921c923e33" (UID: "0b09a84c-ae99-434b-8562-2a921c923e33"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:00.372634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.372560 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-istio-data\") pod \"0b09a84c-ae99-434b-8562-2a921c923e33\" (UID: \"0b09a84c-ae99-434b-8562-2a921c923e33\") " Oct 30 19:54:00.372687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.372647 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-workload-socket" pod "0b09a84c-ae99-434b-8562-2a921c923e33" (UID: "0b09a84c-ae99-434b-8562-2a921c923e33"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:00.372687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.372663 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b09a84c-ae99-434b-8562-2a921c923e33-istiod-ca-cert" pod "0b09a84c-ae99-434b-8562-2a921c923e33" (UID: "0b09a84c-ae99-434b-8562-2a921c923e33"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:00.372943 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.372921 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-istio-data" pod "0b09a84c-ae99-434b-8562-2a921c923e33" (UID: "0b09a84c-ae99-434b-8562-2a921c923e33"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:00.373207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.373187 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:00.373207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.373208 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b09a84c-ae99-434b-8562-2a921c923e33-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:00.373385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.373221 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:00.373385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.373235 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:00.373385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.373247 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:00.375348 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.375313 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-istio-envoy" pod "0b09a84c-ae99-434b-8562-2a921c923e33" (UID: "0b09a84c-ae99-434b-8562-2a921c923e33"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:54:00.375441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.375416 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b09a84c-ae99-434b-8562-2a921c923e33-istio-token" pod "0b09a84c-ae99-434b-8562-2a921c923e33" (UID: "0b09a84c-ae99-434b-8562-2a921c923e33"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:00.375594 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.375566 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b09a84c-ae99-434b-8562-2a921c923e33-istio-podinfo" pod "0b09a84c-ae99-434b-8562-2a921c923e33" (UID: "0b09a84c-ae99-434b-8562-2a921c923e33"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:54:00.375698 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.375610 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b09a84c-ae99-434b-8562-2a921c923e33-kube-api-access-b4xf4" pod "0b09a84c-ae99-434b-8562-2a921c923e33" (UID: "0b09a84c-ae99-434b-8562-2a921c923e33"). InnerVolumeSpecName "kube-api-access-b4xf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:00.474974 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.474899 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b09a84c-ae99-434b-8562-2a921c923e33-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:00.474974 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.474925 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4xf4\" (UniqueName: \"kubernetes.io/projected/0b09a84c-ae99-434b-8562-2a921c923e33-kube-api-access-b4xf4\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:00.474974 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.474936 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b09a84c-ae99-434b-8562-2a921c923e33-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:00.474974 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.474944 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b09a84c-ae99-434b-8562-2a921c923e33-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:00.551062 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.551029 5091 generic.go:356] "Generic (PLEG): container finished" podID="0b09a84c-ae99-434b-8562-2a921c923e33" containerID="507e988d2c08dcc131c57776eb8ea5a11f3ff5807057dc6b9400759936ac5f40" exitCode=0 Oct 30 19:54:00.551258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.551094 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" Oct 30 19:54:00.551258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.551117 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" event={"ID":"0b09a84c-ae99-434b-8562-2a921c923e33","Type":"ContainerDied","Data":"507e988d2c08dcc131c57776eb8ea5a11f3ff5807057dc6b9400759936ac5f40"} Oct 30 19:54:00.551258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.551229 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd" event={"ID":"0b09a84c-ae99-434b-8562-2a921c923e33","Type":"ContainerDied","Data":"2d1035ce0c174b8dd45c73e5e942cf0ff31b6dabdcce62f770e75dc595668096"} Oct 30 19:54:00.551385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.551263 5091 scope.go:122] "RemoveContainer" containerID="507e988d2c08dcc131c57776eb8ea5a11f3ff5807057dc6b9400759936ac5f40" Oct 30 19:54:00.563110 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.563091 5091 scope.go:122] "RemoveContainer" containerID="282c799af4a39cb8f9aab241a18179721065caeb52aa98b75b7f49c8abdf7621" Oct 30 19:54:00.574250 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.574221 5091 scope.go:122] "RemoveContainer" containerID="507e988d2c08dcc131c57776eb8ea5a11f3ff5807057dc6b9400759936ac5f40" Oct 30 19:54:00.574587 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:54:00.574558 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"507e988d2c08dcc131c57776eb8ea5a11f3ff5807057dc6b9400759936ac5f40\": container with ID starting with 507e988d2c08dcc131c57776eb8ea5a11f3ff5807057dc6b9400759936ac5f40 not found: ID does not exist" containerID="507e988d2c08dcc131c57776eb8ea5a11f3ff5807057dc6b9400759936ac5f40" Oct 30 19:54:00.574655 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.574596 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"507e988d2c08dcc131c57776eb8ea5a11f3ff5807057dc6b9400759936ac5f40"} err="failed to get container status \"507e988d2c08dcc131c57776eb8ea5a11f3ff5807057dc6b9400759936ac5f40\": rpc error: code = NotFound desc = could not find container \"507e988d2c08dcc131c57776eb8ea5a11f3ff5807057dc6b9400759936ac5f40\": container with ID starting with 507e988d2c08dcc131c57776eb8ea5a11f3ff5807057dc6b9400759936ac5f40 not found: ID does not exist" Oct 30 19:54:00.574655 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.574614 5091 scope.go:122] "RemoveContainer" containerID="282c799af4a39cb8f9aab241a18179721065caeb52aa98b75b7f49c8abdf7621" Oct 30 19:54:00.574924 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:54:00.574905 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"282c799af4a39cb8f9aab241a18179721065caeb52aa98b75b7f49c8abdf7621\": container with ID starting with 282c799af4a39cb8f9aab241a18179721065caeb52aa98b75b7f49c8abdf7621 not found: ID does not exist" containerID="282c799af4a39cb8f9aab241a18179721065caeb52aa98b75b7f49c8abdf7621" Oct 30 19:54:00.574987 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.574929 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"282c799af4a39cb8f9aab241a18179721065caeb52aa98b75b7f49c8abdf7621"} err="failed to get container status \"282c799af4a39cb8f9aab241a18179721065caeb52aa98b75b7f49c8abdf7621\": rpc error: code = NotFound desc = could not find container \"282c799af4a39cb8f9aab241a18179721065caeb52aa98b75b7f49c8abdf7621\": container with ID starting with 282c799af4a39cb8f9aab241a18179721065caeb52aa98b75b7f49c8abdf7621 not found: ID does not exist" Oct 30 19:54:00.578497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.578475 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd"] Oct 30 19:54:00.582424 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.582402 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-490cdf-openshift-default-7dd4bb56cc-v5vdd"] Oct 30 19:54:00.801799 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.801746 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:00.963348 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:00.963307 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:01.133714 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:01.133621 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:01.362907 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:01.362868 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px"] Oct 30 19:54:01.371900 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:01.371871 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-59e21a-2d8px"] Oct 30 19:54:01.408505 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:01.408406 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:01.586359 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:01.586328 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:01.918359 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:01.918325 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:02.427156 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:02.422180 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:02.485439 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:02.485411 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" path="/var/lib/kubelet/pods/0b09a84c-ae99-434b-8562-2a921c923e33/volumes" Oct 30 19:54:02.486191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:02.486171 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab201154-6cf4-4cf0-b3b3-fe58a3700ee0" path="/var/lib/kubelet/pods/ab201154-6cf4-4cf0-b3b3-fe58a3700ee0/volumes" Oct 30 19:54:03.728824 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:03.728784 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:05.330114 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.330075 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s"] Oct 30 19:54:05.330776 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.330760 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" containerName="istio-proxy" Oct 30 19:54:05.330822 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.330777 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" containerName="istio-proxy" Oct 30 19:54:05.330822 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.330788 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf" containerName="revoke-keys" Oct 30 19:54:05.330822 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.330793 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf" containerName="revoke-keys" Oct 30 19:54:05.330990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.330828 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9b219df2-601d-4e3f-bf24-d32a34b67ee6" containerName="revoke-keys" Oct 30 19:54:05.330990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.330834 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b219df2-601d-4e3f-bf24-d32a34b67ee6" containerName="revoke-keys" Oct 30 19:54:05.330990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.330929 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" containerName="istio-proxy" Oct 30 19:54:05.330990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.330935 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf" containerName="revoke-keys" Oct 30 19:54:05.330990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.330948 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" containerName="istio-proxy" Oct 30 19:54:05.330990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.330955 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b219df2-601d-4e3f-bf24-d32a34b67ee6" containerName="revoke-keys" Oct 30 19:54:05.335951 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.335931 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" Oct 30 19:54:05.341325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.341289 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:54:05.343434 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.343334 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s"] Oct 30 19:54:05.343434 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.343360 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:54:05.511720 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.510869 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shlml\" (UniqueName: \"kubernetes.io/projected/2b24b839-150f-467e-9114-a09bacf549d9-kube-api-access-shlml\") pod \"maas-api-revoke-sub-1d20fe46-f6f5s\" (UID: \"2b24b839-150f-467e-9114-a09bacf549d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" Oct 30 19:54:05.514126 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.514093 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b24b839-150f-467e-9114-a09bacf549d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d20fe46-f6f5s\" (UID: \"2b24b839-150f-467e-9114-a09bacf549d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" Oct 30 19:54:05.514428 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.513840 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:05.514661 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.514554 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b24b839-150f-467e-9114-a09bacf549d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d20fe46-f6f5s\" (UID: \"2b24b839-150f-467e-9114-a09bacf549d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" Oct 30 19:54:05.616389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.616298 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b24b839-150f-467e-9114-a09bacf549d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d20fe46-f6f5s\" (UID: \"2b24b839-150f-467e-9114-a09bacf549d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" Oct 30 19:54:05.616733 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.616671 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b24b839-150f-467e-9114-a09bacf549d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d20fe46-f6f5s\" (UID: \"2b24b839-150f-467e-9114-a09bacf549d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" Oct 30 19:54:05.616971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.616937 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shlml\" (UniqueName: \"kubernetes.io/projected/2b24b839-150f-467e-9114-a09bacf549d9-kube-api-access-shlml\") pod \"maas-api-revoke-sub-1d20fe46-f6f5s\" (UID: \"2b24b839-150f-467e-9114-a09bacf549d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" Oct 30 19:54:05.617340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.617318 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b24b839-150f-467e-9114-a09bacf549d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1d20fe46-f6f5s\" (UID: \"2b24b839-150f-467e-9114-a09bacf549d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" Oct 30 19:54:05.618848 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.618829 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b24b839-150f-467e-9114-a09bacf549d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1d20fe46-f6f5s\" (UID: \"2b24b839-150f-467e-9114-a09bacf549d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" Oct 30 19:54:05.634454 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.634430 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shlml\" (UniqueName: \"kubernetes.io/projected/2b24b839-150f-467e-9114-a09bacf549d9-kube-api-access-shlml\") pod \"maas-api-revoke-sub-1d20fe46-f6f5s\" (UID: \"2b24b839-150f-467e-9114-a09bacf549d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" Oct 30 19:54:05.655249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.655225 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b"] Oct 30 19:54:05.659344 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.659321 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" Oct 30 19:54:05.669747 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.669719 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8"] Oct 30 19:54:05.798377 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:05.798335 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:06.068537 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:06.067214 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s"] Oct 30 19:54:06.132706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:06.132476 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:06.639376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:06.639298 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" event={"ID":"2b24b839-150f-467e-9114-a09bacf549d9","Type":"ContainerStarted","Data":"73dff09f417534f2d65c40dfae97c863044c2faf50c4374943c9ac9b8f759caf"} Oct 30 19:54:06.639376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:06.639358 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" event={"ID":"2b24b839-150f-467e-9114-a09bacf549d9","Type":"ContainerStarted","Data":"19fd725874b77f83cd461886665bb5f7560092d58f723923be8218e66966ee76"} Oct 30 19:54:06.667487 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:06.667442 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" podStartSLOduration=1.667427105 podStartE2EDuration="1.667427105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:06.663841139 +0000 UTC m=+1366.359049097" watchObservedRunningTime="2026-10-30 19:54:06.667427105 +0000 UTC m=+1366.362635067" Oct 30 19:54:06.765029 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:06.764993 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:07.649775 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:07.649735 5091 generic.go:356] "Generic (PLEG): container finished" podID="2b24b839-150f-467e-9114-a09bacf549d9" containerID="73dff09f417534f2d65c40dfae97c863044c2faf50c4374943c9ac9b8f759caf" exitCode=0 Oct 30 19:54:07.650202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:07.649822 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" event={"ID":"2b24b839-150f-467e-9114-a09bacf549d9","Type":"ContainerDied","Data":"73dff09f417534f2d65c40dfae97c863044c2faf50c4374943c9ac9b8f759caf"} Oct 30 19:54:07.903570 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:07.903484 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:08.842198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:08.842136 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" Oct 30 19:54:08.865625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:08.865588 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b24b839-150f-467e-9114-a09bacf549d9-kube-api-access-shlml\" (UniqueName: \"kubernetes.io/projected/2b24b839-150f-467e-9114-a09bacf549d9-kube-api-access-shlml\") pod \"2b24b839-150f-467e-9114-a09bacf549d9\" (UID: \"2b24b839-150f-467e-9114-a09bacf549d9\") " Oct 30 19:54:08.865915 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:08.865672 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b24b839-150f-467e-9114-a09bacf549d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b24b839-150f-467e-9114-a09bacf549d9-maas-api-openshift-ca\") pod \"2b24b839-150f-467e-9114-a09bacf549d9\" (UID: \"2b24b839-150f-467e-9114-a09bacf549d9\") " Oct 30 19:54:08.866036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:08.865998 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b24b839-150f-467e-9114-a09bacf549d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b24b839-150f-467e-9114-a09bacf549d9-maas-api-xks-ca\") pod \"2b24b839-150f-467e-9114-a09bacf549d9\" (UID: \"2b24b839-150f-467e-9114-a09bacf549d9\") " Oct 30 19:54:08.866593 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:08.866563 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b24b839-150f-467e-9114-a09bacf549d9-maas-api-openshift-ca" pod "2b24b839-150f-467e-9114-a09bacf549d9" (UID: "2b24b839-150f-467e-9114-a09bacf549d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:08.868979 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:08.868949 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b24b839-150f-467e-9114-a09bacf549d9-maas-api-xks-ca" pod "2b24b839-150f-467e-9114-a09bacf549d9" (UID: "2b24b839-150f-467e-9114-a09bacf549d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:08.869385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:08.869361 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b24b839-150f-467e-9114-a09bacf549d9-kube-api-access-shlml" pod "2b24b839-150f-467e-9114-a09bacf549d9" (UID: "2b24b839-150f-467e-9114-a09bacf549d9"). InnerVolumeSpecName "kube-api-access-shlml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:08.981337 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:08.978666 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2b24b839-150f-467e-9114-a09bacf549d9-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:08.981337 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:08.978714 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shlml\" (UniqueName: \"kubernetes.io/projected/2b24b839-150f-467e-9114-a09bacf549d9-kube-api-access-shlml\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:08.981337 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:08.978731 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2b24b839-150f-467e-9114-a09bacf549d9-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:09.014172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:08.996003 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:09.672324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:09.672293 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" Oct 30 19:54:09.672528 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:09.672293 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s" event={"ID":"2b24b839-150f-467e-9114-a09bacf549d9","Type":"ContainerDied","Data":"19fd725874b77f83cd461886665bb5f7560092d58f723923be8218e66966ee76"} Oct 30 19:54:09.672528 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:09.672435 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19fd725874b77f83cd461886665bb5f7560092d58f723923be8218e66966ee76" Oct 30 19:54:10.408876 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:10.408825 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:10.584661 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:10.584631 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:10.656694 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:10.656617 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" podUID="21207314-dcb4-4ae2-b76c-54781df4f643" containerName="payload-processing" containerID="cri-o://4dfb06fca71bbf96c83cb3ddcceeea0f0e230b9aa513b53371056510db36d491" gracePeriod=25 Oct 30 19:54:10.676727 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:10.671292 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" podUID="2a44dd5b-7eb5-418c-a1ee-99d44fc907d1" containerName="payload-pre-processing" containerID="cri-o://a9877f9d2cdc3ff29375ca3107c1c6f042630d3fef63d85b1dfe09858e0e51bf" gracePeriod=25 Oct 30 19:54:10.833185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:10.815687 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:11.153596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.153564 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" Oct 30 19:54:11.182363 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.182338 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" Oct 30 19:54:11.221258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.221227 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21207314-dcb4-4ae2-b76c-54781df4f643-kube-api-access-7gnvw\" (UniqueName: \"kubernetes.io/projected/21207314-dcb4-4ae2-b76c-54781df4f643-kube-api-access-7gnvw\") pod \"21207314-dcb4-4ae2-b76c-54781df4f643\" (UID: \"21207314-dcb4-4ae2-b76c-54781df4f643\") " Oct 30 19:54:11.221419 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.221360 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21207314-dcb4-4ae2-b76c-54781df4f643-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21207314-dcb4-4ae2-b76c-54781df4f643-plugins-config-volume\") pod \"21207314-dcb4-4ae2-b76c-54781df4f643\" (UID: \"21207314-dcb4-4ae2-b76c-54781df4f643\") " Oct 30 19:54:11.221419 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.221398 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a44dd5b-7eb5-418c-a1ee-99d44fc907d1-kube-api-access-vnnwv\" (UniqueName: \"kubernetes.io/projected/2a44dd5b-7eb5-418c-a1ee-99d44fc907d1-kube-api-access-vnnwv\") pod \"2a44dd5b-7eb5-418c-a1ee-99d44fc907d1\" (UID: \"2a44dd5b-7eb5-418c-a1ee-99d44fc907d1\") " Oct 30 19:54:11.221545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.221478 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a44dd5b-7eb5-418c-a1ee-99d44fc907d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a44dd5b-7eb5-418c-a1ee-99d44fc907d1-plugins-config-volume\") pod \"2a44dd5b-7eb5-418c-a1ee-99d44fc907d1\" (UID: \"2a44dd5b-7eb5-418c-a1ee-99d44fc907d1\") " Oct 30 19:54:11.221870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.221843 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21207314-dcb4-4ae2-b76c-54781df4f643-plugins-config-volume" pod "21207314-dcb4-4ae2-b76c-54781df4f643" (UID: "21207314-dcb4-4ae2-b76c-54781df4f643"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:11.222088 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.222067 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21207314-dcb4-4ae2-b76c-54781df4f643-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:11.222189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.222068 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a44dd5b-7eb5-418c-a1ee-99d44fc907d1-plugins-config-volume" pod "2a44dd5b-7eb5-418c-a1ee-99d44fc907d1" (UID: "2a44dd5b-7eb5-418c-a1ee-99d44fc907d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:11.223760 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.223738 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21207314-dcb4-4ae2-b76c-54781df4f643-kube-api-access-7gnvw" pod "21207314-dcb4-4ae2-b76c-54781df4f643" (UID: "21207314-dcb4-4ae2-b76c-54781df4f643"). InnerVolumeSpecName "kube-api-access-7gnvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:11.223992 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.223975 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a44dd5b-7eb5-418c-a1ee-99d44fc907d1-kube-api-access-vnnwv" pod "2a44dd5b-7eb5-418c-a1ee-99d44fc907d1" (UID: "2a44dd5b-7eb5-418c-a1ee-99d44fc907d1"). InnerVolumeSpecName "kube-api-access-vnnwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:11.323379 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.323293 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnnwv\" (UniqueName: \"kubernetes.io/projected/2a44dd5b-7eb5-418c-a1ee-99d44fc907d1-kube-api-access-vnnwv\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:11.323379 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.323326 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a44dd5b-7eb5-418c-a1ee-99d44fc907d1-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:11.323379 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.323339 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gnvw\" (UniqueName: \"kubernetes.io/projected/21207314-dcb4-4ae2-b76c-54781df4f643-kube-api-access-7gnvw\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:11.710871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.710835 5091 generic.go:356] "Generic (PLEG): container finished" podID="2a44dd5b-7eb5-418c-a1ee-99d44fc907d1" containerID="a9877f9d2cdc3ff29375ca3107c1c6f042630d3fef63d85b1dfe09858e0e51bf" exitCode=0 Oct 30 19:54:11.711389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.710887 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" Oct 30 19:54:11.711389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.710920 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" event={"ID":"2a44dd5b-7eb5-418c-a1ee-99d44fc907d1","Type":"ContainerDied","Data":"a9877f9d2cdc3ff29375ca3107c1c6f042630d3fef63d85b1dfe09858e0e51bf"} Oct 30 19:54:11.711389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.710981 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8" event={"ID":"2a44dd5b-7eb5-418c-a1ee-99d44fc907d1","Type":"ContainerDied","Data":"701f7e3205f1b5bbc0073a42f08db646898fbc30795921058b84d7be8813462a"} Oct 30 19:54:11.711389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.711012 5091 scope.go:122] "RemoveContainer" containerID="a9877f9d2cdc3ff29375ca3107c1c6f042630d3fef63d85b1dfe09858e0e51bf" Oct 30 19:54:11.712840 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.712819 5091 generic.go:356] "Generic (PLEG): container finished" podID="21207314-dcb4-4ae2-b76c-54781df4f643" containerID="4dfb06fca71bbf96c83cb3ddcceeea0f0e230b9aa513b53371056510db36d491" exitCode=0 Oct 30 19:54:11.713016 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.712876 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" Oct 30 19:54:11.713016 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.712907 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" event={"ID":"21207314-dcb4-4ae2-b76c-54781df4f643","Type":"ContainerDied","Data":"4dfb06fca71bbf96c83cb3ddcceeea0f0e230b9aa513b53371056510db36d491"} Oct 30 19:54:11.713016 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.712962 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b" event={"ID":"21207314-dcb4-4ae2-b76c-54781df4f643","Type":"ContainerDied","Data":"b2a5ec01d8c42dc1962d079011d96f5c2ac688c7de308c28ba53ffee8df07c4b"} Oct 30 19:54:11.724022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.724004 5091 scope.go:122] "RemoveContainer" containerID="a9877f9d2cdc3ff29375ca3107c1c6f042630d3fef63d85b1dfe09858e0e51bf" Oct 30 19:54:11.724323 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:54:11.724305 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9877f9d2cdc3ff29375ca3107c1c6f042630d3fef63d85b1dfe09858e0e51bf\": container with ID starting with a9877f9d2cdc3ff29375ca3107c1c6f042630d3fef63d85b1dfe09858e0e51bf not found: ID does not exist" containerID="a9877f9d2cdc3ff29375ca3107c1c6f042630d3fef63d85b1dfe09858e0e51bf" Oct 30 19:54:11.724417 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.724329 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9877f9d2cdc3ff29375ca3107c1c6f042630d3fef63d85b1dfe09858e0e51bf"} err="failed to get container status \"a9877f9d2cdc3ff29375ca3107c1c6f042630d3fef63d85b1dfe09858e0e51bf\": rpc error: code = NotFound desc = could not find container \"a9877f9d2cdc3ff29375ca3107c1c6f042630d3fef63d85b1dfe09858e0e51bf\": container with ID starting with a9877f9d2cdc3ff29375ca3107c1c6f042630d3fef63d85b1dfe09858e0e51bf not found: ID does not exist" Oct 30 19:54:11.724417 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.724346 5091 scope.go:122] "RemoveContainer" containerID="4dfb06fca71bbf96c83cb3ddcceeea0f0e230b9aa513b53371056510db36d491" Oct 30 19:54:11.739115 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.739089 5091 scope.go:122] "RemoveContainer" containerID="4dfb06fca71bbf96c83cb3ddcceeea0f0e230b9aa513b53371056510db36d491" Oct 30 19:54:11.739471 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:54:11.739441 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dfb06fca71bbf96c83cb3ddcceeea0f0e230b9aa513b53371056510db36d491\": container with ID starting with 4dfb06fca71bbf96c83cb3ddcceeea0f0e230b9aa513b53371056510db36d491 not found: ID does not exist" containerID="4dfb06fca71bbf96c83cb3ddcceeea0f0e230b9aa513b53371056510db36d491" Oct 30 19:54:11.739589 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.739481 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dfb06fca71bbf96c83cb3ddcceeea0f0e230b9aa513b53371056510db36d491"} err="failed to get container status \"4dfb06fca71bbf96c83cb3ddcceeea0f0e230b9aa513b53371056510db36d491\": rpc error: code = NotFound desc = could not find container \"4dfb06fca71bbf96c83cb3ddcceeea0f0e230b9aa513b53371056510db36d491\": container with ID starting with 4dfb06fca71bbf96c83cb3ddcceeea0f0e230b9aa513b53371056510db36d491 not found: ID does not exist" Oct 30 19:54:11.756220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.756192 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b"] Oct 30 19:54:11.761902 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.761880 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-ee4af9-7fd89f99b6-bcw9b"] Oct 30 19:54:11.772510 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.772482 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8"] Oct 30 19:54:11.778177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.778137 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ee4af9-57577b479f-zw8j8"] Oct 30 19:54:11.840116 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:11.839633 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:12.013808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.013767 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:12.251026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.250985 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:12.441996 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.441919 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21207314-dcb4-4ae2-b76c-54781df4f643" path="/var/lib/kubelet/pods/21207314-dcb4-4ae2-b76c-54781df4f643/volumes" Oct 30 19:54:12.442381 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.442365 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a44dd5b-7eb5-418c-a1ee-99d44fc907d1" path="/var/lib/kubelet/pods/2a44dd5b-7eb5-418c-a1ee-99d44fc907d1/volumes" Oct 30 19:54:12.479171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.479115 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:12.944236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.944207 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf"] Oct 30 19:54:12.944861 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.944845 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" containerName="istio-proxy" Oct 30 19:54:12.944908 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.944862 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b09a84c-ae99-434b-8562-2a921c923e33" containerName="istio-proxy" Oct 30 19:54:12.944908 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.944883 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2a44dd5b-7eb5-418c-a1ee-99d44fc907d1" containerName="payload-pre-processing" Oct 30 19:54:12.944908 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.944888 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a44dd5b-7eb5-418c-a1ee-99d44fc907d1" containerName="payload-pre-processing" Oct 30 19:54:12.944908 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.944909 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2b24b839-150f-467e-9114-a09bacf549d9" containerName="revoke-keys" Oct 30 19:54:12.945029 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.944914 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b24b839-150f-467e-9114-a09bacf549d9" containerName="revoke-keys" Oct 30 19:54:12.945029 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.944931 5091 cpu_manager.go:414] "containerMap: removing container" podUID="21207314-dcb4-4ae2-b76c-54781df4f643" containerName="payload-processing" Oct 30 19:54:12.945029 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.944936 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21207314-dcb4-4ae2-b76c-54781df4f643" containerName="payload-processing" Oct 30 19:54:12.945123 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.945035 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b24b839-150f-467e-9114-a09bacf549d9" containerName="revoke-keys" Oct 30 19:54:12.945123 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.945045 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="21207314-dcb4-4ae2-b76c-54781df4f643" containerName="payload-processing" Oct 30 19:54:12.945123 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.945058 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a44dd5b-7eb5-418c-a1ee-99d44fc907d1" containerName="payload-pre-processing" Oct 30 19:54:12.951371 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.951348 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:12.963124 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.963098 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-d2dd78-openshift-default-dockercfg-nq2tq\"" Oct 30 19:54:12.977762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:12.977523 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf"] Oct 30 19:54:13.160169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.156937 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:13.182358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.182303 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww"] Oct 30 19:54:13.183488 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.183450 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5"] Oct 30 19:54:13.224380 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.224285 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-data\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.228404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.224546 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldltb\" (UniqueName: \"kubernetes.io/projected/8d15b25d-07cf-4e6c-8446-60915a92b92e-kube-api-access-ldltb\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.228404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.225134 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-workload-certs\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.228404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.225732 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-envoy\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.228404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.225894 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-credential-socket\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.228404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.226665 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-podinfo\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.228404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.226744 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-workload-socket\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.228404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.226838 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-token\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.228404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.227014 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d15b25d-07cf-4e6c-8446-60915a92b92e-istiod-ca-cert\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.329018 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.328980 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-podinfo\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.329253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.329045 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-workload-socket\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.329253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.329103 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-token\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.329253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.329208 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d15b25d-07cf-4e6c-8446-60915a92b92e-istiod-ca-cert\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.329463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.329273 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-data\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.329463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.329298 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldltb\" (UniqueName: \"kubernetes.io/projected/8d15b25d-07cf-4e6c-8446-60915a92b92e-kube-api-access-ldltb\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.329463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.329410 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-workload-certs\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.329625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.329456 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-workload-socket\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.329625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.329531 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-envoy\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.329625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.329563 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-credential-socket\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.329922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.329900 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-credential-socket\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.329982 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.329938 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d15b25d-07cf-4e6c-8446-60915a92b92e-istiod-ca-cert\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.330184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.330135 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-data\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.330367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.330351 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-workload-certs\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.332369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.332344 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-envoy\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.332937 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.332917 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-podinfo\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.347269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.347241 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:13.348965 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.348941 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-token\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.351216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.351193 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldltb\" (UniqueName: \"kubernetes.io/projected/8d15b25d-07cf-4e6c-8446-60915a92b92e-kube-api-access-ldltb\") pod \"e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.573961 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.573923 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:13.574955 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.574929 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:13.747167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.743664 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:13.995457 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:54:13.995422 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d15b25d_07cf_4e6c_8446_60915a92b92e.slice/crio-1ecb1e5429d128cccc25dad50c681abb13e75c867c955ed16fec18d6e5641c5f WatchSource:0}: Error finding container 1ecb1e5429d128cccc25dad50c681abb13e75c867c955ed16fec18d6e5641c5f: Status 404 returned error can't find the container with id 1ecb1e5429d128cccc25dad50c681abb13e75c867c955ed16fec18d6e5641c5f Oct 30 19:54:13.996232 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.996202 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf"] Oct 30 19:54:13.997905 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.997872 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:54:13.998013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.997948 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:54:13.998078 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:13.998011 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:54:14.697167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:14.696430 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:14.761237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:14.761202 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" event={"ID":"8d15b25d-07cf-4e6c-8446-60915a92b92e","Type":"ContainerStarted","Data":"1244a685b6d3d0480d68cff2d53cc40d24045899cd770f19021a2b9c170cd42b"} Oct 30 19:54:14.761415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:14.761262 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" event={"ID":"8d15b25d-07cf-4e6c-8446-60915a92b92e","Type":"ContainerStarted","Data":"1ecb1e5429d128cccc25dad50c681abb13e75c867c955ed16fec18d6e5641c5f"} Oct 30 19:54:14.797365 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:14.797312 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" podStartSLOduration=2.797298277 podStartE2EDuration="2.797298277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:14.796615658 +0000 UTC m=+1374.491823617" watchObservedRunningTime="2026-10-30 19:54:14.797298277 +0000 UTC m=+1374.492506239" Oct 30 19:54:15.577018 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:15.576978 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:15.583666 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:15.583637 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:15.728892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:15.728855 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:15.782367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:15.782107 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:15.782999 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:15.782974 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:54:15.982834 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:54:15.982736 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:54:16.660194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:16.660126 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:16.953288 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:16.931727 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:17.118292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:17.118257 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b"] Oct 30 19:54:17.145411 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:17.145378 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-ee4af9-zff5b"] Oct 30 19:54:17.194262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:17.194228 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:17.523168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:17.522520 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:17.964762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:17.964674 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:17.996967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:17.996937 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9"] Oct 30 19:54:18.001557 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.001540 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.024599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.024571 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9"] Oct 30 19:54:18.059067 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.059037 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.059263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.059118 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.059263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.059182 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1babb797-d60b-4d9c-8589-27a6877d4478-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.059263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.059242 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.059460 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.059358 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scf9h\" (UniqueName: \"kubernetes.io/projected/1babb797-d60b-4d9c-8589-27a6877d4478-kube-api-access-scf9h\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.059460 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.059385 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1babb797-d60b-4d9c-8589-27a6877d4478-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.059460 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.059411 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.059923 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.059466 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1babb797-d60b-4d9c-8589-27a6877d4478-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.059923 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.059500 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.161105 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.161070 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scf9h\" (UniqueName: \"kubernetes.io/projected/1babb797-d60b-4d9c-8589-27a6877d4478-kube-api-access-scf9h\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.161105 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.161109 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1babb797-d60b-4d9c-8589-27a6877d4478-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.161366 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.161135 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.161366 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.161189 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1babb797-d60b-4d9c-8589-27a6877d4478-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.161366 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.161224 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.161366 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.161309 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.161596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.161422 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.161596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.161446 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1babb797-d60b-4d9c-8589-27a6877d4478-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.161596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.161492 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.161596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.161582 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-istio-data\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.161891 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.161871 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-credential-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.161985 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.161959 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1babb797-d60b-4d9c-8589-27a6877d4478-istiod-ca-cert\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.162055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.161978 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-workload-socket\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.162115 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.162044 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-workload-certs\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.163689 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.163666 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-istio-envoy\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.163859 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.163842 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1babb797-d60b-4d9c-8589-27a6877d4478-istio-podinfo\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.176696 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.176670 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1babb797-d60b-4d9c-8589-27a6877d4478-istio-token\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.177975 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.177952 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scf9h\" (UniqueName: \"kubernetes.io/projected/1babb797-d60b-4d9c-8589-27a6877d4478-kube-api-access-scf9h\") pod \"e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.182675 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.182642 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" podUID="ca4eff0e-4814-46a5-9822-92d297027447" containerName="payload-processing" containerID="cri-o://d8759005fce25d08f63ac131e1232856e4218dcd420201d11f5a04270f2ba1e7" gracePeriod=25 Oct 30 19:54:18.184130 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.184102 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" podUID="45941330-4133-4795-ae8b-cd3dea49ceb9" containerName="payload-pre-processing" containerID="cri-o://36c7afdc0da2e88e3cddfdf028520f09fcdac90d0d92553b5f98c9920e595d28" gracePeriod=25 Oct 30 19:54:18.317252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.317224 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:18.499241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.499205 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b219df2-601d-4e3f-bf24-d32a34b67ee6" path="/var/lib/kubelet/pods/9b219df2-601d-4e3f-bf24-d32a34b67ee6/volumes" Oct 30 19:54:18.499643 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.499623 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp"] Oct 30 19:54:18.510349 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.510317 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp"] Oct 30 19:54:18.510637 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.510619 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" Oct 30 19:54:18.511831 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.511761 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" Oct 30 19:54:18.517226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.517204 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:54:18.517328 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.517312 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:54:18.563337 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.562542 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" Oct 30 19:54:18.571458 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.571426 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca4eff0e-4814-46a5-9822-92d297027447-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4eff0e-4814-46a5-9822-92d297027447-plugins-config-volume\") pod \"ca4eff0e-4814-46a5-9822-92d297027447\" (UID: \"ca4eff0e-4814-46a5-9822-92d297027447\") " Oct 30 19:54:18.571605 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.571570 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca4eff0e-4814-46a5-9822-92d297027447-kube-api-access-vdd4s\" (UniqueName: \"kubernetes.io/projected/ca4eff0e-4814-46a5-9822-92d297027447-kube-api-access-vdd4s\") pod \"ca4eff0e-4814-46a5-9822-92d297027447\" (UID: \"ca4eff0e-4814-46a5-9822-92d297027447\") " Oct 30 19:54:18.571942 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.571915 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnkq6\" (UniqueName: \"kubernetes.io/projected/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-kube-api-access-tnkq6\") pod \"maas-api-revoke-sub-3d1e132d-8csbp\" (UID: \"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" Oct 30 19:54:18.572138 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.572118 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d1e132d-8csbp\" (UID: \"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" Oct 30 19:54:18.572270 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.572196 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d1e132d-8csbp\" (UID: \"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" Oct 30 19:54:18.572678 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.572653 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca4eff0e-4814-46a5-9822-92d297027447-plugins-config-volume" pod "ca4eff0e-4814-46a5-9822-92d297027447" (UID: "ca4eff0e-4814-46a5-9822-92d297027447"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:18.578165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.578124 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca4eff0e-4814-46a5-9822-92d297027447-kube-api-access-vdd4s" pod "ca4eff0e-4814-46a5-9822-92d297027447" (UID: "ca4eff0e-4814-46a5-9822-92d297027447"). InnerVolumeSpecName "kube-api-access-vdd4s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:18.589471 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.589094 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:18.662948 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:54:18.662911 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1babb797_d60b_4d9c_8589_27a6877d4478.slice/crio-2eab06bbd24c9edd8cd139919ec456c61dbad57d14745753143f2a2dc86745a4 WatchSource:0}: Error finding container 2eab06bbd24c9edd8cd139919ec456c61dbad57d14745753143f2a2dc86745a4: Status 404 returned error can't find the container with id 2eab06bbd24c9edd8cd139919ec456c61dbad57d14745753143f2a2dc86745a4 Oct 30 19:54:18.665388 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.665364 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9"] Oct 30 19:54:18.666435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.666398 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:54:18.666528 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.666476 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:54:18.666528 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.666518 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:54:18.673637 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.673612 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45941330-4133-4795-ae8b-cd3dea49ceb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45941330-4133-4795-ae8b-cd3dea49ceb9-plugins-config-volume\") pod \"45941330-4133-4795-ae8b-cd3dea49ceb9\" (UID: \"45941330-4133-4795-ae8b-cd3dea49ceb9\") " Oct 30 19:54:18.673775 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.673659 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45941330-4133-4795-ae8b-cd3dea49ceb9-kube-api-access-drn6h\" (UniqueName: \"kubernetes.io/projected/45941330-4133-4795-ae8b-cd3dea49ceb9-kube-api-access-drn6h\") pod \"45941330-4133-4795-ae8b-cd3dea49ceb9\" (UID: \"45941330-4133-4795-ae8b-cd3dea49ceb9\") " Oct 30 19:54:18.674068 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.674002 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d1e132d-8csbp\" (UID: \"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" Oct 30 19:54:18.674068 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.674057 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d1e132d-8csbp\" (UID: \"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" Oct 30 19:54:18.674326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.674198 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnkq6\" (UniqueName: \"kubernetes.io/projected/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-kube-api-access-tnkq6\") pod \"maas-api-revoke-sub-3d1e132d-8csbp\" (UID: \"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" Oct 30 19:54:18.674326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.674316 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca4eff0e-4814-46a5-9822-92d297027447-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:18.674530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.674332 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdd4s\" (UniqueName: \"kubernetes.io/projected/ca4eff0e-4814-46a5-9822-92d297027447-kube-api-access-vdd4s\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:18.674899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.674863 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45941330-4133-4795-ae8b-cd3dea49ceb9-plugins-config-volume" pod "45941330-4133-4795-ae8b-cd3dea49ceb9" (UID: "45941330-4133-4795-ae8b-cd3dea49ceb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:18.675469 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.675441 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d1e132d-8csbp\" (UID: \"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" Oct 30 19:54:18.677387 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.677364 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d1e132d-8csbp\" (UID: \"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" Oct 30 19:54:18.677922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.677897 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45941330-4133-4795-ae8b-cd3dea49ceb9-kube-api-access-drn6h" pod "45941330-4133-4795-ae8b-cd3dea49ceb9" (UID: "45941330-4133-4795-ae8b-cd3dea49ceb9"). InnerVolumeSpecName "kube-api-access-drn6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:18.690899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.690872 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnkq6\" (UniqueName: \"kubernetes.io/projected/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-kube-api-access-tnkq6\") pod \"maas-api-revoke-sub-3d1e132d-8csbp\" (UID: \"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" Oct 30 19:54:18.813223 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.812051 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:18.834472 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.834424 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45941330-4133-4795-ae8b-cd3dea49ceb9-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:18.834765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.834723 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drn6h\" (UniqueName: \"kubernetes.io/projected/45941330-4133-4795-ae8b-cd3dea49ceb9-kube-api-access-drn6h\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:18.853195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.848472 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" Oct 30 19:54:18.997875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.997828 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" event={"ID":"1babb797-d60b-4d9c-8589-27a6877d4478","Type":"ContainerStarted","Data":"2eab06bbd24c9edd8cd139919ec456c61dbad57d14745753143f2a2dc86745a4"} Oct 30 19:54:18.999363 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.999337 5091 generic.go:356] "Generic (PLEG): container finished" podID="ca4eff0e-4814-46a5-9822-92d297027447" containerID="d8759005fce25d08f63ac131e1232856e4218dcd420201d11f5a04270f2ba1e7" exitCode=0 Oct 30 19:54:18.999507 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.999485 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" event={"ID":"ca4eff0e-4814-46a5-9822-92d297027447","Type":"ContainerDied","Data":"d8759005fce25d08f63ac131e1232856e4218dcd420201d11f5a04270f2ba1e7"} Oct 30 19:54:18.999639 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.999620 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" event={"ID":"ca4eff0e-4814-46a5-9822-92d297027447","Type":"ContainerDied","Data":"4da96060a319a26da351497af44e0a3aee9e63e13283c854a3efa445a6eba92e"} Oct 30 19:54:18.999718 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.999661 5091 scope.go:122] "RemoveContainer" containerID="d8759005fce25d08f63ac131e1232856e4218dcd420201d11f5a04270f2ba1e7" Oct 30 19:54:18.999718 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:18.999689 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww" Oct 30 19:54:19.002205 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:19.002170 5091 generic.go:356] "Generic (PLEG): container finished" podID="45941330-4133-4795-ae8b-cd3dea49ceb9" containerID="36c7afdc0da2e88e3cddfdf028520f09fcdac90d0d92553b5f98c9920e595d28" exitCode=0 Oct 30 19:54:19.002355 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:19.002226 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" event={"ID":"45941330-4133-4795-ae8b-cd3dea49ceb9","Type":"ContainerDied","Data":"36c7afdc0da2e88e3cddfdf028520f09fcdac90d0d92553b5f98c9920e595d28"} Oct 30 19:54:19.002355 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:19.002266 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" event={"ID":"45941330-4133-4795-ae8b-cd3dea49ceb9","Type":"ContainerDied","Data":"6657bdbc98444c57c4953b6d3ae2ad3923692939c19d6a55af5e45fa4a3cdb31"} Oct 30 19:54:19.002355 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:19.002351 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5" Oct 30 19:54:19.020546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:19.020521 5091 scope.go:122] "RemoveContainer" containerID="d8759005fce25d08f63ac131e1232856e4218dcd420201d11f5a04270f2ba1e7" Oct 30 19:54:19.020844 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:54:19.020823 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8759005fce25d08f63ac131e1232856e4218dcd420201d11f5a04270f2ba1e7\": container with ID starting with d8759005fce25d08f63ac131e1232856e4218dcd420201d11f5a04270f2ba1e7 not found: ID does not exist" containerID="d8759005fce25d08f63ac131e1232856e4218dcd420201d11f5a04270f2ba1e7" Oct 30 19:54:19.020922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:19.020853 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8759005fce25d08f63ac131e1232856e4218dcd420201d11f5a04270f2ba1e7"} err="failed to get container status \"d8759005fce25d08f63ac131e1232856e4218dcd420201d11f5a04270f2ba1e7\": rpc error: code = NotFound desc = could not find container \"d8759005fce25d08f63ac131e1232856e4218dcd420201d11f5a04270f2ba1e7\": container with ID starting with d8759005fce25d08f63ac131e1232856e4218dcd420201d11f5a04270f2ba1e7 not found: ID does not exist" Oct 30 19:54:19.020922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:19.020871 5091 scope.go:122] "RemoveContainer" containerID="36c7afdc0da2e88e3cddfdf028520f09fcdac90d0d92553b5f98c9920e595d28" Oct 30 19:54:19.053963 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:19.053904 5091 scope.go:122] "RemoveContainer" containerID="36c7afdc0da2e88e3cddfdf028520f09fcdac90d0d92553b5f98c9920e595d28" Oct 30 19:54:19.054896 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:19.054874 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww"] Oct 30 19:54:19.063905 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:19.060700 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-55a8e6-5f9b84f74b-b4gww"] Oct 30 19:54:19.063905 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:54:19.060866 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c7afdc0da2e88e3cddfdf028520f09fcdac90d0d92553b5f98c9920e595d28\": container with ID starting with 36c7afdc0da2e88e3cddfdf028520f09fcdac90d0d92553b5f98c9920e595d28 not found: ID does not exist" containerID="36c7afdc0da2e88e3cddfdf028520f09fcdac90d0d92553b5f98c9920e595d28" Oct 30 19:54:19.063905 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:19.060897 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c7afdc0da2e88e3cddfdf028520f09fcdac90d0d92553b5f98c9920e595d28"} err="failed to get container status \"36c7afdc0da2e88e3cddfdf028520f09fcdac90d0d92553b5f98c9920e595d28\": rpc error: code = NotFound desc = could not find container \"36c7afdc0da2e88e3cddfdf028520f09fcdac90d0d92553b5f98c9920e595d28\": container with ID starting with 36c7afdc0da2e88e3cddfdf028520f09fcdac90d0d92553b5f98c9920e595d28 not found: ID does not exist" Oct 30 19:54:19.080905 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:19.080842 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5"] Oct 30 19:54:19.086887 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:19.086852 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-55a8e6-76f98dcfdc-2vzn5"] Oct 30 19:54:19.140253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:19.135466 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:19.296372 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:19.296335 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp"] Oct 30 19:54:19.303141 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:54:19.303108 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd7ff1e6_af60_41f6_9cac_d90fa7ce3ce8.slice/crio-307ccfc8c3991b469babf7494704e0e76d2c72f7d53c8067c18965f8332f2271 WatchSource:0}: Error finding container 307ccfc8c3991b469babf7494704e0e76d2c72f7d53c8067c18965f8332f2271: Status 404 returned error can't find the container with id 307ccfc8c3991b469babf7494704e0e76d2c72f7d53c8067c18965f8332f2271 Oct 30 19:54:20.011401 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:20.011369 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" event={"ID":"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8","Type":"ContainerStarted","Data":"973a8c1d8a749d965e04ada6fc78914ef387e1cf94301f615a47ee9d5d6b44cb"} Oct 30 19:54:20.011858 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:20.011421 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" event={"ID":"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8","Type":"ContainerStarted","Data":"307ccfc8c3991b469babf7494704e0e76d2c72f7d53c8067c18965f8332f2271"} Oct 30 19:54:20.013216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:20.013188 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" event={"ID":"1babb797-d60b-4d9c-8589-27a6877d4478","Type":"ContainerStarted","Data":"51fb4e46ca7fd7c32d738320b767ac0c17e51b6d2f7b4f61a64b84d94b92b6ea"} Oct 30 19:54:20.038475 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:20.038430 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" podStartSLOduration=2.038415369 podStartE2EDuration="2.038415369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:20.034757818 +0000 UTC m=+1379.729965775" watchObservedRunningTime="2026-10-30 19:54:20.038415369 +0000 UTC m=+1379.733623321" Oct 30 19:54:20.102658 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:20.102607 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" podStartSLOduration=3.102592855 podStartE2EDuration="3.102592855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:20.099385578 +0000 UTC m=+1379.794593536" watchObservedRunningTime="2026-10-30 19:54:20.102592855 +0000 UTC m=+1379.797800813" Oct 30 19:54:20.223475 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:20.223438 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:20.330238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:20.329343 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:20.330238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:20.329476 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:54:20.330238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:20.329521 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" podUID="1babb797-d60b-4d9c-8589-27a6877d4478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 19:54:20.442378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:20.442343 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45941330-4133-4795-ae8b-cd3dea49ceb9" path="/var/lib/kubelet/pods/45941330-4133-4795-ae8b-cd3dea49ceb9/volumes" Oct 30 19:54:20.442921 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:20.442899 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca4eff0e-4814-46a5-9822-92d297027447" path="/var/lib/kubelet/pods/ca4eff0e-4814-46a5-9822-92d297027447/volumes" Oct 30 19:54:21.025100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:21.025064 5091 generic.go:356] "Generic (PLEG): container finished" podID="fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8" containerID="973a8c1d8a749d965e04ada6fc78914ef387e1cf94301f615a47ee9d5d6b44cb" exitCode=0 Oct 30 19:54:21.025663 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:21.025178 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" event={"ID":"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8","Type":"ContainerDied","Data":"973a8c1d8a749d965e04ada6fc78914ef387e1cf94301f615a47ee9d5d6b44cb"} Oct 30 19:54:21.299482 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:21.296287 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:21.322597 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:21.322545 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:54:21.322811 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:21.322635 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" podUID="1babb797-d60b-4d9c-8589-27a6877d4478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 19:54:21.609606 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:21.609509 5091 scope.go:122] "RemoveContainer" containerID="c5290dbe8c17452884db4c94fc0d0be5603268d8cb190c74ea8e5a438c5f75b3" Oct 30 19:54:21.627098 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:21.627075 5091 scope.go:122] "RemoveContainer" containerID="25e6f139031abf46a5ae4a6b0059f48cfbab225f02bb0ae47c6239881042f7e6" Oct 30 19:54:22.294545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:22.294476 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" Oct 30 19:54:22.318324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:22.318292 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:54:22.318554 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:22.318348 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" podUID="1babb797-d60b-4d9c-8589-27a6877d4478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 19:54:22.408182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:22.398415 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:22.475180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:22.469200 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-kube-api-access-tnkq6\" (UniqueName: \"kubernetes.io/projected/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-kube-api-access-tnkq6\") pod \"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8\" (UID: \"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8\") " Oct 30 19:54:22.475180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:22.469380 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-maas-api-openshift-ca\") pod \"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8\" (UID: \"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8\") " Oct 30 19:54:22.475180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:22.469458 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-maas-api-xks-ca\") pod \"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8\" (UID: \"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8\") " Oct 30 19:54:22.485850 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:22.485811 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-maas-api-openshift-ca" pod "fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8" (UID: "fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:22.503069 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:22.503040 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-maas-api-xks-ca" pod "fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8" (UID: "fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:22.503979 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:22.503943 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-kube-api-access-tnkq6" pod "fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8" (UID: "fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8"). InnerVolumeSpecName "kube-api-access-tnkq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:22.578950 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:22.578852 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnkq6\" (UniqueName: \"kubernetes.io/projected/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-kube-api-access-tnkq6\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:22.578950 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:22.578903 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:22.578950 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:22.578919 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:23.041163 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:23.041112 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:23.041930 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:23.041908 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" Oct 30 19:54:23.042058 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:23.041939 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp" event={"ID":"fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8","Type":"ContainerDied","Data":"307ccfc8c3991b469babf7494704e0e76d2c72f7d53c8067c18965f8332f2271"} Oct 30 19:54:23.042058 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:23.041987 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="307ccfc8c3991b469babf7494704e0e76d2c72f7d53c8067c18965f8332f2271" Oct 30 19:54:23.278185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:23.277125 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:23.328224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:23.319183 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:54:23.328224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:23.319242 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" podUID="1babb797-d60b-4d9c-8589-27a6877d4478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 19:54:23.866170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:23.864253 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:24.322232 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:24.321843 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:54:24.322232 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:24.321910 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" podUID="1babb797-d60b-4d9c-8589-27a6877d4478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 19:54:24.935459 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:24.935417 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:25.322342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:25.322071 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:54:25.322342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:25.322136 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" podUID="1babb797-d60b-4d9c-8589-27a6877d4478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 19:54:25.898335 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:25.898264 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:26.165204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:26.165103 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:26.319699 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:26.319663 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Oct 30 19:54:26.319982 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:26.319954 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" podUID="1babb797-d60b-4d9c-8589-27a6877d4478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Oct 30 19:54:26.430225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:26.426726 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:27.322453 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:27.322416 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:27.322883 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:27.322819 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:27.323521 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:27.323503 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 19:54:27.646127 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:27.646019 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:28.896698 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:28.896660 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x"] Oct 30 19:54:28.898508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:28.898481 5091 cpu_manager.go:414] "containerMap: removing container" podUID="45941330-4133-4795-ae8b-cd3dea49ceb9" containerName="payload-pre-processing" Oct 30 19:54:28.898746 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:28.898727 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45941330-4133-4795-ae8b-cd3dea49ceb9" containerName="payload-pre-processing" Oct 30 19:54:28.898839 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:28.898830 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8" containerName="revoke-keys" Oct 30 19:54:28.898902 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:28.898893 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8" containerName="revoke-keys" Oct 30 19:54:28.898975 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:28.898967 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ca4eff0e-4814-46a5-9822-92d297027447" containerName="payload-processing" Oct 30 19:54:28.899035 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:28.899027 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca4eff0e-4814-46a5-9822-92d297027447" containerName="payload-processing" Oct 30 19:54:28.901328 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:28.899352 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="45941330-4133-4795-ae8b-cd3dea49ceb9" containerName="payload-pre-processing" Oct 30 19:54:28.902729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:28.902712 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8" containerName="revoke-keys" Oct 30 19:54:28.902857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:28.902846 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca4eff0e-4814-46a5-9822-92d297027447" containerName="payload-processing" Oct 30 19:54:28.916168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:28.912938 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" Oct 30 19:54:28.923512 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:28.923484 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:54:28.923813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:28.923793 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:54:28.944202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:28.935757 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x"] Oct 30 19:54:29.037302 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.037258 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7f6q9\" (UniqueName: \"kubernetes.io/projected/55ee3356-2860-4401-b287-40d98706cc1f-kube-api-access-7f6q9\") pod \"maas-api-revoke-sub-40fed590-xff7x\" (UID: \"55ee3356-2860-4401-b287-40d98706cc1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" Oct 30 19:54:29.037806 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.037779 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ee3356-2860-4401-b287-40d98706cc1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40fed590-xff7x\" (UID: \"55ee3356-2860-4401-b287-40d98706cc1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" Oct 30 19:54:29.038001 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.037986 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ee3356-2860-4401-b287-40d98706cc1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40fed590-xff7x\" (UID: \"55ee3356-2860-4401-b287-40d98706cc1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" Oct 30 19:54:29.206008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.185309 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ee3356-2860-4401-b287-40d98706cc1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40fed590-xff7x\" (UID: \"55ee3356-2860-4401-b287-40d98706cc1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" Oct 30 19:54:29.206008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.185416 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ee3356-2860-4401-b287-40d98706cc1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40fed590-xff7x\" (UID: \"55ee3356-2860-4401-b287-40d98706cc1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" Oct 30 19:54:29.206008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.185496 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7f6q9\" (UniqueName: \"kubernetes.io/projected/55ee3356-2860-4401-b287-40d98706cc1f-kube-api-access-7f6q9\") pod \"maas-api-revoke-sub-40fed590-xff7x\" (UID: \"55ee3356-2860-4401-b287-40d98706cc1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" Oct 30 19:54:29.206008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.186852 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ee3356-2860-4401-b287-40d98706cc1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40fed590-xff7x\" (UID: \"55ee3356-2860-4401-b287-40d98706cc1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" Oct 30 19:54:29.244445 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.244412 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ee3356-2860-4401-b287-40d98706cc1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40fed590-xff7x\" (UID: \"55ee3356-2860-4401-b287-40d98706cc1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" Oct 30 19:54:29.264673 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.263924 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7f6q9\" (UniqueName: \"kubernetes.io/projected/55ee3356-2860-4401-b287-40d98706cc1f-kube-api-access-7f6q9\") pod \"maas-api-revoke-sub-40fed590-xff7x\" (UID: \"55ee3356-2860-4401-b287-40d98706cc1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" Oct 30 19:54:29.290927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.290825 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" Oct 30 19:54:29.374166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.361211 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:29.471248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.468966 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c"] Oct 30 19:54:29.635892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.631393 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" Oct 30 19:54:29.713059 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.712097 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c"] Oct 30 19:54:29.734217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.732715 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nsvs\" (UniqueName: \"kubernetes.io/projected/819637f6-ac70-4a33-a78f-fa3a4b4712bd-kube-api-access-6nsvs\") pod \"maas-api-revoke-sub-ea26fd20-bdh6c\" (UID: \"819637f6-ac70-4a33-a78f-fa3a4b4712bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" Oct 30 19:54:29.734217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.732761 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819637f6-ac70-4a33-a78f-fa3a4b4712bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea26fd20-bdh6c\" (UID: \"819637f6-ac70-4a33-a78f-fa3a4b4712bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" Oct 30 19:54:29.734217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:29.732880 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819637f6-ac70-4a33-a78f-fa3a4b4712bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea26fd20-bdh6c\" (UID: \"819637f6-ac70-4a33-a78f-fa3a4b4712bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" Oct 30 19:54:30.017424 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:30.004190 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nsvs\" (UniqueName: \"kubernetes.io/projected/819637f6-ac70-4a33-a78f-fa3a4b4712bd-kube-api-access-6nsvs\") pod \"maas-api-revoke-sub-ea26fd20-bdh6c\" (UID: \"819637f6-ac70-4a33-a78f-fa3a4b4712bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" Oct 30 19:54:30.017424 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:30.004242 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819637f6-ac70-4a33-a78f-fa3a4b4712bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea26fd20-bdh6c\" (UID: \"819637f6-ac70-4a33-a78f-fa3a4b4712bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" Oct 30 19:54:30.017424 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:30.004370 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819637f6-ac70-4a33-a78f-fa3a4b4712bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea26fd20-bdh6c\" (UID: \"819637f6-ac70-4a33-a78f-fa3a4b4712bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" Oct 30 19:54:30.059306 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:30.046568 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819637f6-ac70-4a33-a78f-fa3a4b4712bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea26fd20-bdh6c\" (UID: \"819637f6-ac70-4a33-a78f-fa3a4b4712bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" Oct 30 19:54:30.192425 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:30.174962 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x"] Oct 30 19:54:30.315816 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:54:30.309420 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ee3356_2860_4401_b287_40d98706cc1f.slice/crio-9b080db4293e39e8724c0c4a44dc7c559cb56fc38792171239aeaddf243b078c WatchSource:0}: Error finding container 9b080db4293e39e8724c0c4a44dc7c559cb56fc38792171239aeaddf243b078c: Status 404 returned error can't find the container with id 9b080db4293e39e8724c0c4a44dc7c559cb56fc38792171239aeaddf243b078c Oct 30 19:54:30.363181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:30.363094 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" event={"ID":"55ee3356-2860-4401-b287-40d98706cc1f","Type":"ContainerStarted","Data":"9b080db4293e39e8724c0c4a44dc7c559cb56fc38792171239aeaddf243b078c"} Oct 30 19:54:30.409184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:30.409118 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nsvs\" (UniqueName: \"kubernetes.io/projected/819637f6-ac70-4a33-a78f-fa3a4b4712bd-kube-api-access-6nsvs\") pod \"maas-api-revoke-sub-ea26fd20-bdh6c\" (UID: \"819637f6-ac70-4a33-a78f-fa3a4b4712bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" Oct 30 19:54:30.410686 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:30.409520 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819637f6-ac70-4a33-a78f-fa3a4b4712bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea26fd20-bdh6c\" (UID: \"819637f6-ac70-4a33-a78f-fa3a4b4712bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" Oct 30 19:54:30.634177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:30.634060 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" Oct 30 19:54:30.755281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:30.755235 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:30.927273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:30.904125 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 19:54:30.940723 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:30.940688 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:30.976217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:30.959934 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:31.179822 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:31.179783 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:31.459231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:31.377843 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:31.615435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:31.612807 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" event={"ID":"55ee3356-2860-4401-b287-40d98706cc1f","Type":"ContainerStarted","Data":"81e65f592713f2edbbc648860ca8835e249e3f376692a8f07534da79262b4132"} Oct 30 19:54:31.627556 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:31.627023 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c"] Oct 30 19:54:31.676165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:31.667170 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" podStartSLOduration=3.66713509 podStartE2EDuration="3.66713509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:31.661073067 +0000 UTC m=+1391.356281025" watchObservedRunningTime="2026-10-30 19:54:31.66713509 +0000 UTC m=+1391.362343049" Oct 30 19:54:31.928944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:31.928908 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:32.622094 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:32.622011 5091 generic.go:356] "Generic (PLEG): container finished" podID="55ee3356-2860-4401-b287-40d98706cc1f" containerID="81e65f592713f2edbbc648860ca8835e249e3f376692a8f07534da79262b4132" exitCode=0 Oct 30 19:54:32.622529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:32.622098 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" event={"ID":"55ee3356-2860-4401-b287-40d98706cc1f","Type":"ContainerDied","Data":"81e65f592713f2edbbc648860ca8835e249e3f376692a8f07534da79262b4132"} Oct 30 19:54:32.623796 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:32.623769 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" event={"ID":"819637f6-ac70-4a33-a78f-fa3a4b4712bd","Type":"ContainerStarted","Data":"39715085beaccacf3ad731be63b51feb88fd46c1ddc139f4a4e9c23b2eead8a2"} Oct 30 19:54:32.623919 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:32.623819 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" event={"ID":"819637f6-ac70-4a33-a78f-fa3a4b4712bd","Type":"ContainerStarted","Data":"0d1964ed2f1502fd02d2262e0da0fb5494835c1b9db0902f2f049083d40d24a8"} Oct 30 19:54:32.744642 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:32.744580 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" podStartSLOduration=3.744560779 podStartE2EDuration="3.744560779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:32.743444566 +0000 UTC m=+1392.438652524" watchObservedRunningTime="2026-10-30 19:54:32.744560779 +0000 UTC m=+1392.439768718" Oct 30 19:54:33.085989 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.085953 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:33.400620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.400585 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq"] Oct 30 19:54:33.405074 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.405049 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" Oct 30 19:54:33.413429 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.413398 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq"] Oct 30 19:54:33.529466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.529428 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq\" (UID: \"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" Oct 30 19:54:33.529662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.529511 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq\" (UID: \"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" Oct 30 19:54:33.529817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.529768 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mklkd\" (UniqueName: \"kubernetes.io/projected/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-kube-api-access-mklkd\") pod \"maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq\" (UID: \"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" Oct 30 19:54:33.631779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.631742 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mklkd\" (UniqueName: \"kubernetes.io/projected/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-kube-api-access-mklkd\") pod \"maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq\" (UID: \"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" Oct 30 19:54:33.632274 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.632247 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq\" (UID: \"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" Oct 30 19:54:33.632353 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.632326 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq\" (UID: \"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" Oct 30 19:54:33.633156 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.633115 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq\" (UID: \"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" Oct 30 19:54:33.634202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.634175 5091 generic.go:356] "Generic (PLEG): container finished" podID="819637f6-ac70-4a33-a78f-fa3a4b4712bd" containerID="39715085beaccacf3ad731be63b51feb88fd46c1ddc139f4a4e9c23b2eead8a2" exitCode=0 Oct 30 19:54:33.634302 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.634241 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" event={"ID":"819637f6-ac70-4a33-a78f-fa3a4b4712bd","Type":"ContainerDied","Data":"39715085beaccacf3ad731be63b51feb88fd46c1ddc139f4a4e9c23b2eead8a2"} Oct 30 19:54:33.635422 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.635403 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq\" (UID: \"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" Oct 30 19:54:33.688809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.684424 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mklkd\" (UniqueName: \"kubernetes.io/projected/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-kube-api-access-mklkd\") pod \"maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq\" (UID: \"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" Oct 30 19:54:33.838879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.838851 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" Oct 30 19:54:33.897529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:33.897502 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" Oct 30 19:54:34.037827 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.037797 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55ee3356-2860-4401-b287-40d98706cc1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ee3356-2860-4401-b287-40d98706cc1f-maas-api-openshift-ca\") pod \"55ee3356-2860-4401-b287-40d98706cc1f\" (UID: \"55ee3356-2860-4401-b287-40d98706cc1f\") " Oct 30 19:54:34.037827 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.037830 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55ee3356-2860-4401-b287-40d98706cc1f-kube-api-access-7f6q9\" (UniqueName: \"kubernetes.io/projected/55ee3356-2860-4401-b287-40d98706cc1f-kube-api-access-7f6q9\") pod \"55ee3356-2860-4401-b287-40d98706cc1f\" (UID: \"55ee3356-2860-4401-b287-40d98706cc1f\") " Oct 30 19:54:34.038097 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.037857 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55ee3356-2860-4401-b287-40d98706cc1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ee3356-2860-4401-b287-40d98706cc1f-maas-api-xks-ca\") pod \"55ee3356-2860-4401-b287-40d98706cc1f\" (UID: \"55ee3356-2860-4401-b287-40d98706cc1f\") " Oct 30 19:54:34.038640 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.038596 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ee3356-2860-4401-b287-40d98706cc1f-maas-api-openshift-ca" pod "55ee3356-2860-4401-b287-40d98706cc1f" (UID: "55ee3356-2860-4401-b287-40d98706cc1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:34.040941 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.040903 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55ee3356-2860-4401-b287-40d98706cc1f-maas-api-xks-ca" pod "55ee3356-2860-4401-b287-40d98706cc1f" (UID: "55ee3356-2860-4401-b287-40d98706cc1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:34.042865 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.042835 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55ee3356-2860-4401-b287-40d98706cc1f-kube-api-access-7f6q9" pod "55ee3356-2860-4401-b287-40d98706cc1f" (UID: "55ee3356-2860-4401-b287-40d98706cc1f"). InnerVolumeSpecName "kube-api-access-7f6q9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:34.139676 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.139637 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55ee3356-2860-4401-b287-40d98706cc1f-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:34.139676 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.139668 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7f6q9\" (UniqueName: \"kubernetes.io/projected/55ee3356-2860-4401-b287-40d98706cc1f-kube-api-access-7f6q9\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:34.139676 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.139682 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55ee3356-2860-4401-b287-40d98706cc1f-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:34.183190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.183017 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:34.207169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.206528 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq"] Oct 30 19:54:34.230358 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:54:34.215293 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf8ca4d_efb0_477a_b21b_b83004cfdcf5.slice/crio-b3d76489fd8b25a4fca93f7b5ade62d69e32f6926b403fa987e6cc157abbbdd9 WatchSource:0}: Error finding container b3d76489fd8b25a4fca93f7b5ade62d69e32f6926b403fa987e6cc157abbbdd9: Status 404 returned error can't find the container with id b3d76489fd8b25a4fca93f7b5ade62d69e32f6926b403fa987e6cc157abbbdd9 Oct 30 19:54:34.643793 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.643765 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" Oct 30 19:54:34.644189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.643792 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x" event={"ID":"55ee3356-2860-4401-b287-40d98706cc1f","Type":"ContainerDied","Data":"9b080db4293e39e8724c0c4a44dc7c559cb56fc38792171239aeaddf243b078c"} Oct 30 19:54:34.644189 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.643869 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b080db4293e39e8724c0c4a44dc7c559cb56fc38792171239aeaddf243b078c" Oct 30 19:54:34.656187 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.645447 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" event={"ID":"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5","Type":"ContainerStarted","Data":"b3d76489fd8b25a4fca93f7b5ade62d69e32f6926b403fa987e6cc157abbbdd9"} Oct 30 19:54:34.923559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:34.923537 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" Oct 30 19:54:35.091275 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:35.090637 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/819637f6-ac70-4a33-a78f-fa3a4b4712bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819637f6-ac70-4a33-a78f-fa3a4b4712bd-maas-api-xks-ca\") pod \"819637f6-ac70-4a33-a78f-fa3a4b4712bd\" (UID: \"819637f6-ac70-4a33-a78f-fa3a4b4712bd\") " Oct 30 19:54:35.091275 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:35.090853 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/819637f6-ac70-4a33-a78f-fa3a4b4712bd-kube-api-access-6nsvs\" (UniqueName: \"kubernetes.io/projected/819637f6-ac70-4a33-a78f-fa3a4b4712bd-kube-api-access-6nsvs\") pod \"819637f6-ac70-4a33-a78f-fa3a4b4712bd\" (UID: \"819637f6-ac70-4a33-a78f-fa3a4b4712bd\") " Oct 30 19:54:35.091275 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:35.090898 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/819637f6-ac70-4a33-a78f-fa3a4b4712bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819637f6-ac70-4a33-a78f-fa3a4b4712bd-maas-api-openshift-ca\") pod \"819637f6-ac70-4a33-a78f-fa3a4b4712bd\" (UID: \"819637f6-ac70-4a33-a78f-fa3a4b4712bd\") " Oct 30 19:54:35.091686 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:35.091657 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/819637f6-ac70-4a33-a78f-fa3a4b4712bd-maas-api-openshift-ca" pod "819637f6-ac70-4a33-a78f-fa3a4b4712bd" (UID: "819637f6-ac70-4a33-a78f-fa3a4b4712bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:35.093800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:35.093767 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/819637f6-ac70-4a33-a78f-fa3a4b4712bd-maas-api-xks-ca" pod "819637f6-ac70-4a33-a78f-fa3a4b4712bd" (UID: "819637f6-ac70-4a33-a78f-fa3a4b4712bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:35.093995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:35.093971 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/819637f6-ac70-4a33-a78f-fa3a4b4712bd-kube-api-access-6nsvs" pod "819637f6-ac70-4a33-a78f-fa3a4b4712bd" (UID: "819637f6-ac70-4a33-a78f-fa3a4b4712bd"). InnerVolumeSpecName "kube-api-access-6nsvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:35.192238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:35.186300 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:35.194833 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:35.194494 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nsvs\" (UniqueName: \"kubernetes.io/projected/819637f6-ac70-4a33-a78f-fa3a4b4712bd-kube-api-access-6nsvs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:35.194833 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:35.194530 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/819637f6-ac70-4a33-a78f-fa3a4b4712bd-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:35.194833 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:35.194558 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/819637f6-ac70-4a33-a78f-fa3a4b4712bd-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:35.656509 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:35.656469 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" event={"ID":"819637f6-ac70-4a33-a78f-fa3a4b4712bd","Type":"ContainerDied","Data":"0d1964ed2f1502fd02d2262e0da0fb5494835c1b9db0902f2f049083d40d24a8"} Oct 30 19:54:35.656972 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:35.656533 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d1964ed2f1502fd02d2262e0da0fb5494835c1b9db0902f2f049083d40d24a8" Oct 30 19:54:35.656972 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:35.656603 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c" Oct 30 19:54:35.658674 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:35.658644 5091 generic.go:356] "Generic (PLEG): container finished" podID="7bf8ca4d-efb0-477a-b21b-b83004cfdcf5" containerID="47360f08022999c3947ac261b80b1fc0968ae0e1668787703aaa5ea23c641fbd" exitCode=0 Oct 30 19:54:35.658812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:35.658719 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" event={"ID":"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5","Type":"ContainerDied","Data":"47360f08022999c3947ac261b80b1fc0968ae0e1668787703aaa5ea23c641fbd"} Oct 30 19:54:36.180231 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:36.180187 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:36.864454 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:36.864426 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" Oct 30 19:54:36.921896 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:36.921788 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-maas-api-xks-ca\") pod \"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5\" (UID: \"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5\") " Oct 30 19:54:36.921896 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:36.921832 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-kube-api-access-mklkd\" (UniqueName: \"kubernetes.io/projected/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-kube-api-access-mklkd\") pod \"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5\" (UID: \"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5\") " Oct 30 19:54:36.921896 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:36.921875 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-maas-api-openshift-ca\") pod \"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5\" (UID: \"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5\") " Oct 30 19:54:36.922651 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:36.922625 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-maas-api-openshift-ca" pod "7bf8ca4d-efb0-477a-b21b-b83004cfdcf5" (UID: "7bf8ca4d-efb0-477a-b21b-b83004cfdcf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:36.924991 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:36.924969 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-maas-api-xks-ca" pod "7bf8ca4d-efb0-477a-b21b-b83004cfdcf5" (UID: "7bf8ca4d-efb0-477a-b21b-b83004cfdcf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:36.927607 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:36.927579 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-kube-api-access-mklkd" pod "7bf8ca4d-efb0-477a-b21b-b83004cfdcf5" (UID: "7bf8ca4d-efb0-477a-b21b-b83004cfdcf5"). InnerVolumeSpecName "kube-api-access-mklkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:37.138235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:37.133391 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:37.138235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:37.133429 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mklkd\" (UniqueName: \"kubernetes.io/projected/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-kube-api-access-mklkd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:37.138235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:37.133445 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:37.403546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:37.403498 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:37.683545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:37.683461 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" Oct 30 19:54:37.683545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:37.683476 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq" event={"ID":"7bf8ca4d-efb0-477a-b21b-b83004cfdcf5","Type":"ContainerDied","Data":"b3d76489fd8b25a4fca93f7b5ade62d69e32f6926b403fa987e6cc157abbbdd9"} Oct 30 19:54:37.683545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:37.683539 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3d76489fd8b25a4fca93f7b5ade62d69e32f6926b403fa987e6cc157abbbdd9" Oct 30 19:54:38.201066 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:38.201024 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:38.996004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:38.995959 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:39.710256 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:39.710156 5091 generic.go:356] "Generic (PLEG): container finished" podID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" containerID="462df2709f0731a0695283434955e2ac9857ad2062b17d2fdd08554e82865560" exitCode=0 Oct 30 19:54:39.710256 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:39.710174 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" event={"ID":"fed9893a-07bd-4fb4-8a22-2c55b0d122bc","Type":"ContainerDied","Data":"462df2709f0731a0695283434955e2ac9857ad2062b17d2fdd08554e82865560"} Oct 30 19:54:39.710852 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:39.710604 5091 scope.go:122] "RemoveContainer" containerID="462df2709f0731a0695283434955e2ac9857ad2062b17d2fdd08554e82865560" Oct 30 19:54:39.712079 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:39.712049 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:54:39.712246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:39.712132 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:54:39.712246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:39.712200 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:54:40.099202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.093541 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:40.480268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.475809 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh"] Oct 30 19:54:40.480268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.478904 5091 cpu_manager.go:414] "containerMap: removing container" podUID="819637f6-ac70-4a33-a78f-fa3a4b4712bd" containerName="revoke-keys" Oct 30 19:54:40.480268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.478934 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="819637f6-ac70-4a33-a78f-fa3a4b4712bd" containerName="revoke-keys" Oct 30 19:54:40.480268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.478960 5091 cpu_manager.go:414] "containerMap: removing container" podUID="55ee3356-2860-4401-b287-40d98706cc1f" containerName="revoke-keys" Oct 30 19:54:40.480268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.478968 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55ee3356-2860-4401-b287-40d98706cc1f" containerName="revoke-keys" Oct 30 19:54:40.480268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.479018 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7bf8ca4d-efb0-477a-b21b-b83004cfdcf5" containerName="revoke-keys" Oct 30 19:54:40.480268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.479025 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf8ca4d-efb0-477a-b21b-b83004cfdcf5" containerName="revoke-keys" Oct 30 19:54:40.480268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.479201 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf8ca4d-efb0-477a-b21b-b83004cfdcf5" containerName="revoke-keys" Oct 30 19:54:40.480268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.479217 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="819637f6-ac70-4a33-a78f-fa3a4b4712bd" containerName="revoke-keys" Oct 30 19:54:40.480268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.479225 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="55ee3356-2860-4401-b287-40d98706cc1f" containerName="revoke-keys" Oct 30 19:54:40.483175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.483092 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" Oct 30 19:54:40.487619 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.487591 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh"] Oct 30 19:54:40.499529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.496296 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:54:40.499529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.496410 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:54:40.518255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.518077 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh\" (UID: \"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" Oct 30 19:54:40.518255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.518218 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh\" (UID: \"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" Oct 30 19:54:40.518416 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.518352 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v9lv\" (UniqueName: \"kubernetes.io/projected/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-kube-api-access-4v9lv\") pod \"maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh\" (UID: \"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" Oct 30 19:54:40.619823 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.619791 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh\" (UID: \"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" Oct 30 19:54:40.620015 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.619912 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v9lv\" (UniqueName: \"kubernetes.io/projected/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-kube-api-access-4v9lv\") pod \"maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh\" (UID: \"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" Oct 30 19:54:40.620083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.620032 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh\" (UID: \"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" Oct 30 19:54:40.620735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.620708 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh\" (UID: \"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" Oct 30 19:54:40.622307 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.622282 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh\" (UID: \"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" Oct 30 19:54:40.638603 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.638572 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v9lv\" (UniqueName: \"kubernetes.io/projected/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-kube-api-access-4v9lv\") pod \"maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh\" (UID: \"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" Oct 30 19:54:40.721333 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.721298 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" event={"ID":"fed9893a-07bd-4fb4-8a22-2c55b0d122bc","Type":"ContainerStarted","Data":"69f51a82bcfc292bacd23d4984a3301a9d21eaa00ff2e18e0917b2ddef968e1d"} Oct 30 19:54:40.803079 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.803049 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" Oct 30 19:54:40.929617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.929457 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:54:40.929617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.929579 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 19:54:40.929827 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:40.929625 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" podUID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 19:54:41.137745 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:54:41.137712 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e6fc02f_fa77_4ae9_9eed_150ee9e06e6d.slice/crio-96766023748a35f6f273b42c3e670a148a21949195c318aaa6a2d89252756a39 WatchSource:0}: Error finding container 96766023748a35f6f273b42c3e670a148a21949195c318aaa6a2d89252756a39: Status 404 returned error can't find the container with id 96766023748a35f6f273b42c3e670a148a21949195c318aaa6a2d89252756a39 Oct 30 19:54:41.140172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.138230 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh"] Oct 30 19:54:41.180257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.180202 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:54:41.340178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.338956 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:41.586965 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.586932 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv"] Oct 30 19:54:41.591740 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.591719 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" Oct 30 19:54:41.608627 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.607248 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv"] Oct 30 19:54:41.637890 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.637859 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11cab416-lx2zv\" (UID: \"611c9ca9-d00a-47e3-9c77-caf4204b3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" Oct 30 19:54:41.638016 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.638000 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtnzd\" (UniqueName: \"kubernetes.io/projected/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-kube-api-access-qtnzd\") pod \"maas-api-revoke-sub-11cab416-lx2zv\" (UID: \"611c9ca9-d00a-47e3-9c77-caf4204b3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" Oct 30 19:54:41.638081 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.638059 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11cab416-lx2zv\" (UID: \"611c9ca9-d00a-47e3-9c77-caf4204b3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" Oct 30 19:54:41.731208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.731174 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" event={"ID":"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d","Type":"ContainerStarted","Data":"522d41964eba19a08c8c8537cd19f8fbe0007793ba90db9c71535647bb4f0c20"} Oct 30 19:54:41.731587 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.731228 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" event={"ID":"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d","Type":"ContainerStarted","Data":"96766023748a35f6f273b42c3e670a148a21949195c318aaa6a2d89252756a39"} Oct 30 19:54:41.739229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.739202 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtnzd\" (UniqueName: \"kubernetes.io/projected/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-kube-api-access-qtnzd\") pod \"maas-api-revoke-sub-11cab416-lx2zv\" (UID: \"611c9ca9-d00a-47e3-9c77-caf4204b3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" Oct 30 19:54:41.739372 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.739244 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11cab416-lx2zv\" (UID: \"611c9ca9-d00a-47e3-9c77-caf4204b3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" Oct 30 19:54:41.739559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.739540 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11cab416-lx2zv\" (UID: \"611c9ca9-d00a-47e3-9c77-caf4204b3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" Oct 30 19:54:41.740105 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.740065 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11cab416-lx2zv\" (UID: \"611c9ca9-d00a-47e3-9c77-caf4204b3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" Oct 30 19:54:41.742264 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.742238 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11cab416-lx2zv\" (UID: \"611c9ca9-d00a-47e3-9c77-caf4204b3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" Oct 30 19:54:41.777903 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.777832 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" podStartSLOduration=1.7778143800000001 podStartE2EDuration="1.77781438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:41.755570794 +0000 UTC m=+1401.450778753" watchObservedRunningTime="2026-10-30 19:54:41.77781438 +0000 UTC m=+1401.473022338" Oct 30 19:54:41.784715 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.784679 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtnzd\" (UniqueName: \"kubernetes.io/projected/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-kube-api-access-qtnzd\") pod \"maas-api-revoke-sub-11cab416-lx2zv\" (UID: \"611c9ca9-d00a-47e3-9c77-caf4204b3b1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" Oct 30 19:54:41.913622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.913543 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" Oct 30 19:54:41.927797 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.927767 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 19:54:41.928007 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:41.927980 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" podUID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 19:54:42.343864 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:42.343043 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:42.361722 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:54:42.361690 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod611c9ca9_d00a_47e3_9c77_caf4204b3b1c.slice/crio-27337cde41f6bc9b5abf81a0be0d9d8943d0820ae0ea43625e7aa454b6521357 WatchSource:0}: Error finding container 27337cde41f6bc9b5abf81a0be0d9d8943d0820ae0ea43625e7aa454b6521357: Status 404 returned error can't find the container with id 27337cde41f6bc9b5abf81a0be0d9d8943d0820ae0ea43625e7aa454b6521357 Oct 30 19:54:42.378203 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:42.378173 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv"] Oct 30 19:54:42.744746 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:42.744701 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" event={"ID":"611c9ca9-d00a-47e3-9c77-caf4204b3b1c","Type":"ContainerStarted","Data":"27337cde41f6bc9b5abf81a0be0d9d8943d0820ae0ea43625e7aa454b6521357"} Oct 30 19:54:42.780749 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:42.780707 5091 generic.go:356] "Generic (PLEG): container finished" podID="3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d" containerID="522d41964eba19a08c8c8537cd19f8fbe0007793ba90db9c71535647bb4f0c20" exitCode=0 Oct 30 19:54:42.782116 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:42.782087 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" event={"ID":"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d","Type":"ContainerDied","Data":"522d41964eba19a08c8c8537cd19f8fbe0007793ba90db9c71535647bb4f0c20"} Oct 30 19:54:42.928813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:42.928765 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 19:54:42.928813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:42.928809 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" podUID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 19:54:43.456747 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:43.456709 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:43.696617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:43.696401 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk"] Oct 30 19:54:43.700041 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:43.700017 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc"] Oct 30 19:54:43.801211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:43.800609 5091 generic.go:356] "Generic (PLEG): container finished" podID="611c9ca9-d00a-47e3-9c77-caf4204b3b1c" containerID="ae2e80493e7836f9c9caf4637eebb1462e3106a21f82bd1aeda9ff3c38394ec5" exitCode=0 Oct 30 19:54:43.801211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:43.801017 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" event={"ID":"611c9ca9-d00a-47e3-9c77-caf4204b3b1c","Type":"ContainerDied","Data":"ae2e80493e7836f9c9caf4637eebb1462e3106a21f82bd1aeda9ff3c38394ec5"} Oct 30 19:54:43.927780 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:43.927753 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 19:54:43.927987 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:43.927968 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" podUID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 19:54:43.982208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:43.982165 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:44.026708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.026679 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" Oct 30 19:54:44.172300 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.170311 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-kube-api-access-4v9lv\" (UniqueName: \"kubernetes.io/projected/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-kube-api-access-4v9lv\") pod \"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d\" (UID: \"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d\") " Oct 30 19:54:44.172300 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.170492 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-maas-api-xks-ca\") pod \"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d\" (UID: \"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d\") " Oct 30 19:54:44.172300 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.170560 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-maas-api-openshift-ca\") pod \"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d\" (UID: \"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d\") " Oct 30 19:54:44.172837 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.172789 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-maas-api-openshift-ca" pod "3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d" (UID: "3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:44.286682 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.279390 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:44.310171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.302734 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-maas-api-xks-ca" pod "3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d" (UID: "3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:44.327305 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.327250 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-kube-api-access-4v9lv" pod "3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d" (UID: "3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d"). InnerVolumeSpecName "kube-api-access-4v9lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:44.339036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.338899 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:44.397206 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.384545 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:44.397443 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.397419 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v9lv\" (UniqueName: \"kubernetes.io/projected/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d-kube-api-access-4v9lv\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:44.810928 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.810898 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" Oct 30 19:54:44.811449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.810935 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh" event={"ID":"3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d","Type":"ContainerDied","Data":"96766023748a35f6f273b42c3e670a148a21949195c318aaa6a2d89252756a39"} Oct 30 19:54:44.811449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.810998 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96766023748a35f6f273b42c3e670a148a21949195c318aaa6a2d89252756a39" Oct 30 19:54:44.867058 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.866948 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:44.942167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.930767 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 19:54:44.942167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:44.930852 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" podUID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 19:54:45.132121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.132095 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" Oct 30 19:54:45.229450 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.229415 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-maas-api-openshift-ca\") pod \"611c9ca9-d00a-47e3-9c77-caf4204b3b1c\" (UID: \"611c9ca9-d00a-47e3-9c77-caf4204b3b1c\") " Oct 30 19:54:45.229647 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.229470 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-kube-api-access-qtnzd\" (UniqueName: \"kubernetes.io/projected/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-kube-api-access-qtnzd\") pod \"611c9ca9-d00a-47e3-9c77-caf4204b3b1c\" (UID: \"611c9ca9-d00a-47e3-9c77-caf4204b3b1c\") " Oct 30 19:54:45.229647 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.229534 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-maas-api-xks-ca\") pod \"611c9ca9-d00a-47e3-9c77-caf4204b3b1c\" (UID: \"611c9ca9-d00a-47e3-9c77-caf4204b3b1c\") " Oct 30 19:54:45.229831 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.229810 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-maas-api-openshift-ca" pod "611c9ca9-d00a-47e3-9c77-caf4204b3b1c" (UID: "611c9ca9-d00a-47e3-9c77-caf4204b3b1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:45.230599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.230571 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:45.232021 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.231986 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-kube-api-access-qtnzd" pod "611c9ca9-d00a-47e3-9c77-caf4204b3b1c" (UID: "611c9ca9-d00a-47e3-9c77-caf4204b3b1c"). InnerVolumeSpecName "kube-api-access-qtnzd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:45.232174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.232078 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-maas-api-xks-ca" pod "611c9ca9-d00a-47e3-9c77-caf4204b3b1c" (UID: "611c9ca9-d00a-47e3-9c77-caf4204b3b1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:45.332438 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.332402 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtnzd\" (UniqueName: \"kubernetes.io/projected/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-kube-api-access-qtnzd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:45.332438 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.332432 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/611c9ca9-d00a-47e3-9c77-caf4204b3b1c-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:45.731282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.731243 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8"] Oct 30 19:54:45.732000 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.731979 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d" containerName="revoke-keys" Oct 30 19:54:45.732000 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.732001 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d" containerName="revoke-keys" Oct 30 19:54:45.732193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.732052 5091 cpu_manager.go:414] "containerMap: removing container" podUID="611c9ca9-d00a-47e3-9c77-caf4204b3b1c" containerName="revoke-keys" Oct 30 19:54:45.732193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.732061 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="611c9ca9-d00a-47e3-9c77-caf4204b3b1c" containerName="revoke-keys" Oct 30 19:54:45.732317 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.732267 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="611c9ca9-d00a-47e3-9c77-caf4204b3b1c" containerName="revoke-keys" Oct 30 19:54:45.732317 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.732296 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d" containerName="revoke-keys" Oct 30 19:54:45.736207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.736185 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" Oct 30 19:54:45.790370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.790339 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8"] Oct 30 19:54:45.827781 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.827750 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" Oct 30 19:54:45.828445 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.828413 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv" event={"ID":"611c9ca9-d00a-47e3-9c77-caf4204b3b1c","Type":"ContainerDied","Data":"27337cde41f6bc9b5abf81a0be0d9d8943d0820ae0ea43625e7aa454b6521357"} Oct 30 19:54:45.828627 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.828607 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27337cde41f6bc9b5abf81a0be0d9d8943d0820ae0ea43625e7aa454b6521357" Oct 30 19:54:45.843057 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.843025 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c58301-710b-4000-a116-20e76db1292b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dafe6264-7k7p8\" (UID: \"09c58301-710b-4000-a116-20e76db1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" Oct 30 19:54:45.843295 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.843270 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c58301-710b-4000-a116-20e76db1292b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dafe6264-7k7p8\" (UID: \"09c58301-710b-4000-a116-20e76db1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" Oct 30 19:54:45.843392 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.843374 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmhnd\" (UniqueName: \"kubernetes.io/projected/09c58301-710b-4000-a116-20e76db1292b-kube-api-access-nmhnd\") pod \"maas-api-revoke-sub-dafe6264-7k7p8\" (UID: \"09c58301-710b-4000-a116-20e76db1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" Oct 30 19:54:45.939760 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.938825 5091 patch_prober.go:29] interesting pod/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" start-of-body= Oct 30 19:54:45.939760 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.938945 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" podUID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.170:15021/healthz/ready\": dial tcp 10.132.0.170:15021: connect: connection refused" Oct 30 19:54:45.942896 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.942858 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:45.956221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.954945 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmhnd\" (UniqueName: \"kubernetes.io/projected/09c58301-710b-4000-a116-20e76db1292b-kube-api-access-nmhnd\") pod \"maas-api-revoke-sub-dafe6264-7k7p8\" (UID: \"09c58301-710b-4000-a116-20e76db1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" Oct 30 19:54:45.956221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.955107 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c58301-710b-4000-a116-20e76db1292b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dafe6264-7k7p8\" (UID: \"09c58301-710b-4000-a116-20e76db1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" Oct 30 19:54:45.956221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.955387 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c58301-710b-4000-a116-20e76db1292b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dafe6264-7k7p8\" (UID: \"09c58301-710b-4000-a116-20e76db1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" Oct 30 19:54:45.977770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.969003 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c58301-710b-4000-a116-20e76db1292b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dafe6264-7k7p8\" (UID: \"09c58301-710b-4000-a116-20e76db1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" Oct 30 19:54:45.977770 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.971462 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c58301-710b-4000-a116-20e76db1292b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dafe6264-7k7p8\" (UID: \"09c58301-710b-4000-a116-20e76db1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" Oct 30 19:54:45.997224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:45.995527 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmhnd\" (UniqueName: \"kubernetes.io/projected/09c58301-710b-4000-a116-20e76db1292b-kube-api-access-nmhnd\") pod \"maas-api-revoke-sub-dafe6264-7k7p8\" (UID: \"09c58301-710b-4000-a116-20e76db1292b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" Oct 30 19:54:46.027772 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:54:46.027631 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:54:46.067506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:46.067466 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" Oct 30 19:54:46.366852 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:46.366824 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8"] Oct 30 19:54:46.367398 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:54:46.367369 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c58301_710b_4000_a116_20e76db1292b.slice/crio-b56a374fc1927eda6c28142a138fb2c68614cacb0e27c1855ecf5a0808121c61 WatchSource:0}: Error finding container b56a374fc1927eda6c28142a138fb2c68614cacb0e27c1855ecf5a0808121c61: Status 404 returned error can't find the container with id b56a374fc1927eda6c28142a138fb2c68614cacb0e27c1855ecf5a0808121c61 Oct 30 19:54:46.839961 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:46.839927 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" event={"ID":"09c58301-710b-4000-a116-20e76db1292b","Type":"ContainerStarted","Data":"38379709aa8c13c2a6832ad79935fcf577456cbbf2946d4644f1109d53459b2d"} Oct 30 19:54:46.841257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:46.840356 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" event={"ID":"09c58301-710b-4000-a116-20e76db1292b","Type":"ContainerStarted","Data":"b56a374fc1927eda6c28142a138fb2c68614cacb0e27c1855ecf5a0808121c61"} Oct 30 19:54:46.872654 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:46.872598 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" podStartSLOduration=1.872579725 podStartE2EDuration="1.872579725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:46.869867977 +0000 UTC m=+1406.565075936" watchObservedRunningTime="2026-10-30 19:54:46.872579725 +0000 UTC m=+1406.567787684" Oct 30 19:54:46.932448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:46.932377 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:54:46.933509 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:46.933490 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:54:46.987166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:46.984130 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:47.597226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:47.591887 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:47.856546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:47.856441 5091 generic.go:356] "Generic (PLEG): container finished" podID="09c58301-710b-4000-a116-20e76db1292b" containerID="38379709aa8c13c2a6832ad79935fcf577456cbbf2946d4644f1109d53459b2d" exitCode=0 Oct 30 19:54:47.857009 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:47.856536 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" event={"ID":"09c58301-710b-4000-a116-20e76db1292b","Type":"ContainerDied","Data":"38379709aa8c13c2a6832ad79935fcf577456cbbf2946d4644f1109d53459b2d"} Oct 30 19:54:48.134971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:48.127452 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:48.697927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:48.697885 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" podUID="afbd3cf9-21cd-4b40-94d2-df49a1146de6" containerName="payload-processing" containerID="cri-o://6eb83afccbfa90bb7d61d4072b2963763176e07c05eb2fb897715a80281c8b1a" gracePeriod=25 Oct 30 19:54:48.701393 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:48.701361 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" podUID="3c150c5f-1392-4a50-964d-a5e1593c2a19" containerName="payload-pre-processing" containerID="cri-o://88dd0237e19085a5e64ecc9e1cc25d9923d11fa3cb3258495b84047907f1a658" gracePeriod=25 Oct 30 19:54:48.823342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:48.823303 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:49.011902 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.011876 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc"] Oct 30 19:54:49.016665 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.016644 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" Oct 30 19:54:49.025515 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.025483 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc"] Oct 30 19:54:49.034385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.034357 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" Oct 30 19:54:49.113035 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.112997 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3427d142-5236-4930-b38e-911de853d9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91ea7035-vf8nc\" (UID: \"3427d142-5236-4930-b38e-911de853d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" Oct 30 19:54:49.113218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.113189 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fvpk\" (UniqueName: \"kubernetes.io/projected/3427d142-5236-4930-b38e-911de853d9af-kube-api-access-9fvpk\") pod \"maas-api-revoke-sub-91ea7035-vf8nc\" (UID: \"3427d142-5236-4930-b38e-911de853d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" Oct 30 19:54:49.113474 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.113449 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3427d142-5236-4930-b38e-911de853d9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91ea7035-vf8nc\" (UID: \"3427d142-5236-4930-b38e-911de853d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" Oct 30 19:54:49.214498 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.214463 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09c58301-710b-4000-a116-20e76db1292b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c58301-710b-4000-a116-20e76db1292b-maas-api-xks-ca\") pod \"09c58301-710b-4000-a116-20e76db1292b\" (UID: \"09c58301-710b-4000-a116-20e76db1292b\") " Oct 30 19:54:49.214662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.214643 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c58301-710b-4000-a116-20e76db1292b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c58301-710b-4000-a116-20e76db1292b-maas-api-openshift-ca\") pod \"09c58301-710b-4000-a116-20e76db1292b\" (UID: \"09c58301-710b-4000-a116-20e76db1292b\") " Oct 30 19:54:49.214760 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.214745 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c58301-710b-4000-a116-20e76db1292b-kube-api-access-nmhnd\" (UniqueName: \"kubernetes.io/projected/09c58301-710b-4000-a116-20e76db1292b-kube-api-access-nmhnd\") pod \"09c58301-710b-4000-a116-20e76db1292b\" (UID: \"09c58301-710b-4000-a116-20e76db1292b\") " Oct 30 19:54:49.215047 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.215026 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3427d142-5236-4930-b38e-911de853d9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91ea7035-vf8nc\" (UID: \"3427d142-5236-4930-b38e-911de853d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" Oct 30 19:54:49.215134 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.215046 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c58301-710b-4000-a116-20e76db1292b-maas-api-openshift-ca" pod "09c58301-710b-4000-a116-20e76db1292b" (UID: "09c58301-710b-4000-a116-20e76db1292b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:49.215134 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.215083 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3427d142-5236-4930-b38e-911de853d9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91ea7035-vf8nc\" (UID: \"3427d142-5236-4930-b38e-911de853d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" Oct 30 19:54:49.215252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.215232 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fvpk\" (UniqueName: \"kubernetes.io/projected/3427d142-5236-4930-b38e-911de853d9af-kube-api-access-9fvpk\") pod \"maas-api-revoke-sub-91ea7035-vf8nc\" (UID: \"3427d142-5236-4930-b38e-911de853d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" Oct 30 19:54:49.215352 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.215315 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c58301-710b-4000-a116-20e76db1292b-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:49.216004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.215977 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3427d142-5236-4930-b38e-911de853d9af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-91ea7035-vf8nc\" (UID: \"3427d142-5236-4930-b38e-911de853d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" Oct 30 19:54:49.216796 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.216776 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c58301-710b-4000-a116-20e76db1292b-maas-api-xks-ca" pod "09c58301-710b-4000-a116-20e76db1292b" (UID: "09c58301-710b-4000-a116-20e76db1292b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:49.217197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.217137 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c58301-710b-4000-a116-20e76db1292b-kube-api-access-nmhnd" pod "09c58301-710b-4000-a116-20e76db1292b" (UID: "09c58301-710b-4000-a116-20e76db1292b"). InnerVolumeSpecName "kube-api-access-nmhnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:49.217482 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.217466 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3427d142-5236-4930-b38e-911de853d9af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-91ea7035-vf8nc\" (UID: \"3427d142-5236-4930-b38e-911de853d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" Oct 30 19:54:49.240988 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.240958 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2"] Oct 30 19:54:49.258803 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.258775 5091 cpu_manager.go:414] "containerMap: removing container" podUID="09c58301-710b-4000-a116-20e76db1292b" containerName="revoke-keys" Oct 30 19:54:49.258803 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.258800 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c58301-710b-4000-a116-20e76db1292b" containerName="revoke-keys" Oct 30 19:54:49.259102 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.259081 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c58301-710b-4000-a116-20e76db1292b" containerName="revoke-keys" Oct 30 19:54:49.261793 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.261768 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fvpk\" (UniqueName: \"kubernetes.io/projected/3427d142-5236-4930-b38e-911de853d9af-kube-api-access-9fvpk\") pod \"maas-api-revoke-sub-91ea7035-vf8nc\" (UID: \"3427d142-5236-4930-b38e-911de853d9af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" Oct 30 19:54:49.263871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.263850 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.308625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.308593 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-a29f33-openshift-default-dockercfg-nz72r\"" Oct 30 19:54:49.356057 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.356022 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmhnd\" (UniqueName: \"kubernetes.io/projected/09c58301-710b-4000-a116-20e76db1292b-kube-api-access-nmhnd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:49.356284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.356263 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c58301-710b-4000-a116-20e76db1292b-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:49.356422 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.356404 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" Oct 30 19:54:49.455166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.444263 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2"] Oct 30 19:54:49.517519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.499079 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-workload-socket\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.517519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.499135 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-envoy\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.517519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.499179 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istiod-ca-cert\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.517519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.499348 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-workload-certs\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.517519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.499434 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-token\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.517519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.499502 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-data\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.517519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.499557 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-credential-socket\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.517519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.499584 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-podinfo\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.517519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.499642 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rbbd\" (UniqueName: \"kubernetes.io/projected/57f477a5-ec74-470b-bb7e-afd4d690e8c7-kube-api-access-2rbbd\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.591790 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.591746 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:49.659409 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.659375 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-workload-certs\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.659695 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.659676 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-token\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.659855 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.659840 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-data\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.659990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.659977 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-credential-socket\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.660098 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.660084 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-podinfo\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.660259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.660243 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rbbd\" (UniqueName: \"kubernetes.io/projected/57f477a5-ec74-470b-bb7e-afd4d690e8c7-kube-api-access-2rbbd\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.660386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.660374 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-workload-socket\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.660485 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.660472 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-envoy\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.660591 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.660578 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istiod-ca-cert\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.661567 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.661544 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istiod-ca-cert\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.661987 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.661969 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-workload-certs\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.662638 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.662616 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-data\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.662998 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.662980 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-credential-socket\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.663773 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.663752 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-workload-socket\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.670170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.670134 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-envoy\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.682833 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.682812 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-podinfo\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.706998 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.706590 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rbbd\" (UniqueName: \"kubernetes.io/projected/57f477a5-ec74-470b-bb7e-afd4d690e8c7-kube-api-access-2rbbd\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.723004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.722957 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-token\") pod \"e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.798054 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.797976 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:49.884880 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.884842 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" event={"ID":"09c58301-710b-4000-a116-20e76db1292b","Type":"ContainerDied","Data":"b56a374fc1927eda6c28142a138fb2c68614cacb0e27c1855ecf5a0808121c61"} Oct 30 19:54:49.885050 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.884900 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b56a374fc1927eda6c28142a138fb2c68614cacb0e27c1855ecf5a0808121c61" Oct 30 19:54:49.885050 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.884963 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8" Oct 30 19:54:49.898879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:49.898848 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc"] Oct 30 19:54:49.900085 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:54:49.900056 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3427d142_5236_4930_b38e_911de853d9af.slice/crio-55bc5a578f0597721cf7fc40a989c6b945a3815d8e7f6a4d624cdcdd96d43b20 WatchSource:0}: Error finding container 55bc5a578f0597721cf7fc40a989c6b945a3815d8e7f6a4d624cdcdd96d43b20: Status 404 returned error can't find the container with id 55bc5a578f0597721cf7fc40a989c6b945a3815d8e7f6a4d624cdcdd96d43b20 Oct 30 19:54:50.111005 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:54:50.110974 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f477a5_ec74_470b_bb7e_afd4d690e8c7.slice/crio-2c1b2982a19ceb7e6463939720b2851d3c3d03168fcbdb23d6b14fa9e848030e WatchSource:0}: Error finding container 2c1b2982a19ceb7e6463939720b2851d3c3d03168fcbdb23d6b14fa9e848030e: Status 404 returned error can't find the container with id 2c1b2982a19ceb7e6463939720b2851d3c3d03168fcbdb23d6b14fa9e848030e Oct 30 19:54:50.113410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:50.113381 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2"] Oct 30 19:54:50.113989 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:50.113959 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:54:50.114043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:50.114023 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:54:50.114076 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:50.114051 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:54:50.944931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:50.944858 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" event={"ID":"57f477a5-ec74-470b-bb7e-afd4d690e8c7","Type":"ContainerStarted","Data":"95dddeac43c00e2a488aa748dd39e4ada5cf271b8855707a4bddede54450ba48"} Oct 30 19:54:50.945271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:50.945248 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" event={"ID":"57f477a5-ec74-470b-bb7e-afd4d690e8c7","Type":"ContainerStarted","Data":"2c1b2982a19ceb7e6463939720b2851d3c3d03168fcbdb23d6b14fa9e848030e"} Oct 30 19:54:50.952726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:50.950133 5091 generic.go:356] "Generic (PLEG): container finished" podID="3427d142-5236-4930-b38e-911de853d9af" containerID="92a7a3b781d8ea84a2d94f68638197296f11f4b34a179d07e40f7831c317b91d" exitCode=0 Oct 30 19:54:50.952726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:50.950274 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" event={"ID":"3427d142-5236-4930-b38e-911de853d9af","Type":"ContainerDied","Data":"92a7a3b781d8ea84a2d94f68638197296f11f4b34a179d07e40f7831c317b91d"} Oct 30 19:54:50.952726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:50.950314 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" event={"ID":"3427d142-5236-4930-b38e-911de853d9af","Type":"ContainerStarted","Data":"55bc5a578f0597721cf7fc40a989c6b945a3815d8e7f6a4d624cdcdd96d43b20"} Oct 30 19:54:51.014780 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:51.014725 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" podStartSLOduration=2.014709578 podStartE2EDuration="2.014709578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:50.978429159 +0000 UTC m=+1410.673637138" watchObservedRunningTime="2026-10-30 19:54:51.014709578 +0000 UTC m=+1410.709917564" Oct 30 19:54:51.798538 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:51.798504 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:51.805011 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:51.804987 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:51.959122 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:51.959087 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:51.960923 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:51.960900 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:54:52.001292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:52.001255 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:52.156791 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:52.156746 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:52.165379 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:52.165357 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" Oct 30 19:54:52.305822 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:52.305783 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3427d142-5236-4930-b38e-911de853d9af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3427d142-5236-4930-b38e-911de853d9af-maas-api-xks-ca\") pod \"3427d142-5236-4930-b38e-911de853d9af\" (UID: \"3427d142-5236-4930-b38e-911de853d9af\") " Oct 30 19:54:52.306023 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:52.305843 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3427d142-5236-4930-b38e-911de853d9af-kube-api-access-9fvpk\" (UniqueName: \"kubernetes.io/projected/3427d142-5236-4930-b38e-911de853d9af-kube-api-access-9fvpk\") pod \"3427d142-5236-4930-b38e-911de853d9af\" (UID: \"3427d142-5236-4930-b38e-911de853d9af\") " Oct 30 19:54:52.306116 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:52.306101 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3427d142-5236-4930-b38e-911de853d9af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3427d142-5236-4930-b38e-911de853d9af-maas-api-openshift-ca\") pod \"3427d142-5236-4930-b38e-911de853d9af\" (UID: \"3427d142-5236-4930-b38e-911de853d9af\") " Oct 30 19:54:52.306503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:52.306472 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3427d142-5236-4930-b38e-911de853d9af-maas-api-openshift-ca" pod "3427d142-5236-4930-b38e-911de853d9af" (UID: "3427d142-5236-4930-b38e-911de853d9af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:52.307095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:52.307070 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3427d142-5236-4930-b38e-911de853d9af-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:52.308692 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:52.308669 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3427d142-5236-4930-b38e-911de853d9af-kube-api-access-9fvpk" pod "3427d142-5236-4930-b38e-911de853d9af" (UID: "3427d142-5236-4930-b38e-911de853d9af"). InnerVolumeSpecName "kube-api-access-9fvpk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:52.311131 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:52.311108 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3427d142-5236-4930-b38e-911de853d9af-maas-api-xks-ca" pod "3427d142-5236-4930-b38e-911de853d9af" (UID: "3427d142-5236-4930-b38e-911de853d9af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:54:52.408527 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:52.408447 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3427d142-5236-4930-b38e-911de853d9af-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:52.408527 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:52.408478 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fvpk\" (UniqueName: \"kubernetes.io/projected/3427d142-5236-4930-b38e-911de853d9af-kube-api-access-9fvpk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:52.969909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:52.969872 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" Oct 30 19:54:52.969909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:52.969888 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc" event={"ID":"3427d142-5236-4930-b38e-911de853d9af","Type":"ContainerDied","Data":"55bc5a578f0597721cf7fc40a989c6b945a3815d8e7f6a4d624cdcdd96d43b20"} Oct 30 19:54:52.970382 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:52.969953 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55bc5a578f0597721cf7fc40a989c6b945a3815d8e7f6a4d624cdcdd96d43b20" Oct 30 19:54:53.308280 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:53.308233 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:53.542364 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:53.542326 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" podUID="3c150c5f-1392-4a50-964d-a5e1593c2a19" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:54:53.542527 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:53.542325 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" podUID="afbd3cf9-21cd-4b40-94d2-df49a1146de6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:54:53.626643 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:53.626564 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:53.848528 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:53.848489 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:54.770527 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:54.764258 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:54.905774 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:54.905742 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb"] Oct 30 19:54:54.906421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:54.906406 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3427d142-5236-4930-b38e-911de853d9af" containerName="revoke-keys" Oct 30 19:54:54.906474 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:54.906421 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3427d142-5236-4930-b38e-911de853d9af" containerName="revoke-keys" Oct 30 19:54:54.906575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:54.906565 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3427d142-5236-4930-b38e-911de853d9af" containerName="revoke-keys" Oct 30 19:54:54.910277 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:54.910261 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" Oct 30 19:54:54.923769 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:54.923744 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l"] Oct 30 19:54:54.927703 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:54.927677 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-a29f33\"" Oct 30 19:54:54.928585 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:54.928553 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" Oct 30 19:54:54.933670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:54.933642 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb"] Oct 30 19:54:54.935808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:54.935783 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-a29f33-dockercfg-hmsbc\"" Oct 30 19:54:54.957011 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:54.956981 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l"] Oct 30 19:54:54.988700 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:54.987814 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:55.053072 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.052981 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5qmq\" (UniqueName: \"kubernetes.io/projected/3f73a998-807d-4652-8c3a-28b4f6d42390-kube-api-access-p5qmq\") pod \"payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l\" (UID: \"3f73a998-807d-4652-8c3a-28b4f6d42390\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" Oct 30 19:54:55.053298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.053134 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02feb7af-17b1-472a-81d4-a5265781f30c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb\" (UID: \"02feb7af-17b1-472a-81d4-a5265781f30c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" Oct 30 19:54:55.053298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.053211 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f73a998-807d-4652-8c3a-28b4f6d42390-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l\" (UID: \"3f73a998-807d-4652-8c3a-28b4f6d42390\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" Oct 30 19:54:55.053298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.053244 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfp57\" (UniqueName: \"kubernetes.io/projected/02feb7af-17b1-472a-81d4-a5265781f30c-kube-api-access-rfp57\") pod \"payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb\" (UID: \"02feb7af-17b1-472a-81d4-a5265781f30c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" Oct 30 19:54:55.173181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.171109 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02feb7af-17b1-472a-81d4-a5265781f30c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb\" (UID: \"02feb7af-17b1-472a-81d4-a5265781f30c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" Oct 30 19:54:55.173181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.171247 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f73a998-807d-4652-8c3a-28b4f6d42390-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l\" (UID: \"3f73a998-807d-4652-8c3a-28b4f6d42390\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" Oct 30 19:54:55.173181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.171289 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfp57\" (UniqueName: \"kubernetes.io/projected/02feb7af-17b1-472a-81d4-a5265781f30c-kube-api-access-rfp57\") pod \"payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb\" (UID: \"02feb7af-17b1-472a-81d4-a5265781f30c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" Oct 30 19:54:55.173181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.171446 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5qmq\" (UniqueName: \"kubernetes.io/projected/3f73a998-807d-4652-8c3a-28b4f6d42390-kube-api-access-p5qmq\") pod \"payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l\" (UID: \"3f73a998-807d-4652-8c3a-28b4f6d42390\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" Oct 30 19:54:55.173181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.172747 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02feb7af-17b1-472a-81d4-a5265781f30c-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb\" (UID: \"02feb7af-17b1-472a-81d4-a5265781f30c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" Oct 30 19:54:55.173591 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.173541 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f73a998-807d-4652-8c3a-28b4f6d42390-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l\" (UID: \"3f73a998-807d-4652-8c3a-28b4f6d42390\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" Oct 30 19:54:55.219181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.218861 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5qmq\" (UniqueName: \"kubernetes.io/projected/3f73a998-807d-4652-8c3a-28b4f6d42390-kube-api-access-p5qmq\") pod \"payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l\" (UID: \"3f73a998-807d-4652-8c3a-28b4f6d42390\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" Oct 30 19:54:55.257187 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.252823 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfp57\" (UniqueName: \"kubernetes.io/projected/02feb7af-17b1-472a-81d4-a5265781f30c-kube-api-access-rfp57\") pod \"payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb\" (UID: \"02feb7af-17b1-472a-81d4-a5265781f30c\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" Oct 30 19:54:55.258137 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.258023 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" Oct 30 19:54:55.525656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.525615 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" Oct 30 19:54:55.618388 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.615319 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l"] Oct 30 19:54:55.829866 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:54:55.829832 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02feb7af_17b1_472a_81d4_a5265781f30c.slice/crio-d3f2a44db3d61cd6c8d1ee26ea65cccfb3727d0995bdb86d659b8758770db9fc WatchSource:0}: Error finding container d3f2a44db3d61cd6c8d1ee26ea65cccfb3727d0995bdb86d659b8758770db9fc: Status 404 returned error can't find the container with id d3f2a44db3d61cd6c8d1ee26ea65cccfb3727d0995bdb86d659b8758770db9fc Oct 30 19:54:55.831383 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.831353 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb"] Oct 30 19:54:55.920037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.920003 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72"] Oct 30 19:54:55.920492 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.920258 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" podUID="da240f1c-8dd4-4205-9985-92f3e42cb8e5" containerName="payload-processing" containerID="cri-o://4e2fdd00bbdfbe9c5e0449416f8231c40db1019b1c9476ba07d41c6f44f56c70" gracePeriod=30 Oct 30 19:54:55.924004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.923976 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:55.961217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.958197 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r"] Oct 30 19:54:55.961217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:55.958460 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" podUID="5b77699f-9a88-4dde-94f0-feee3d4f01e8" containerName="payload-pre-processing" containerID="cri-o://3987717a8f9ef800a474b3828c76e4c22bc17df4b76967d077406855ac11a595" gracePeriod=30 Oct 30 19:54:56.007776 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.007742 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" event={"ID":"02feb7af-17b1-472a-81d4-a5265781f30c","Type":"ContainerStarted","Data":"421151e16a377defdd053f0c132d7de7a43d0076d12e00ead83faf3fb8e32d1d"} Oct 30 19:54:56.007952 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.007812 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" Oct 30 19:54:56.007952 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.007838 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" event={"ID":"02feb7af-17b1-472a-81d4-a5265781f30c","Type":"ContainerStarted","Data":"d3f2a44db3d61cd6c8d1ee26ea65cccfb3727d0995bdb86d659b8758770db9fc"} Oct 30 19:54:56.010735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.010570 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" event={"ID":"3f73a998-807d-4652-8c3a-28b4f6d42390","Type":"ContainerStarted","Data":"3d2859d3610d8ff67d756156c6fc58980bcce43ead1fccf8c327e649a8709b99"} Oct 30 19:54:56.010872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.010772 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" Oct 30 19:54:56.010872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.010799 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" event={"ID":"3f73a998-807d-4652-8c3a-28b4f6d42390","Type":"ContainerStarted","Data":"e9ba0da1a7b5fe9c27d2c2bafbffef72462ae5be9dc6aa446abf0910f8da1231"} Oct 30 19:54:56.052245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.052187 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" podStartSLOduration=2.052168995 podStartE2EDuration="2.052168995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:56.044101212 +0000 UTC m=+1415.739309172" watchObservedRunningTime="2026-10-30 19:54:56.052168995 +0000 UTC m=+1415.747376955" Oct 30 19:54:56.083743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.083643 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" podStartSLOduration=2.083624395 podStartE2EDuration="2.083624395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:54:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:54:56.079211717 +0000 UTC m=+1415.774419679" watchObservedRunningTime="2026-10-30 19:54:56.083624395 +0000 UTC m=+1415.778832355" Oct 30 19:54:56.279180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.274772 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:56.551879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.551853 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh"] Oct 30 19:54:56.558201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.558176 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-426bab-r6lhh"] Oct 30 19:54:56.585351 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.581325 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" Oct 30 19:54:56.661692 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.661233 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da240f1c-8dd4-4205-9985-92f3e42cb8e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da240f1c-8dd4-4205-9985-92f3e42cb8e5-plugins-config-volume\") pod \"da240f1c-8dd4-4205-9985-92f3e42cb8e5\" (UID: \"da240f1c-8dd4-4205-9985-92f3e42cb8e5\") " Oct 30 19:54:56.661692 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.661344 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da240f1c-8dd4-4205-9985-92f3e42cb8e5-kube-api-access-tnljw\" (UniqueName: \"kubernetes.io/projected/da240f1c-8dd4-4205-9985-92f3e42cb8e5-kube-api-access-tnljw\") pod \"da240f1c-8dd4-4205-9985-92f3e42cb8e5\" (UID: \"da240f1c-8dd4-4205-9985-92f3e42cb8e5\") " Oct 30 19:54:56.700544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.700412 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:56.706089 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.706042 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da240f1c-8dd4-4205-9985-92f3e42cb8e5-kube-api-access-tnljw" pod "da240f1c-8dd4-4205-9985-92f3e42cb8e5" (UID: "da240f1c-8dd4-4205-9985-92f3e42cb8e5"). InnerVolumeSpecName "kube-api-access-tnljw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:56.796171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.778834 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnljw\" (UniqueName: \"kubernetes.io/projected/da240f1c-8dd4-4205-9985-92f3e42cb8e5-kube-api-access-tnljw\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:56.966845 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.966758 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da240f1c-8dd4-4205-9985-92f3e42cb8e5-plugins-config-volume" pod "da240f1c-8dd4-4205-9985-92f3e42cb8e5" (UID: "da240f1c-8dd4-4205-9985-92f3e42cb8e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:56.971166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:56.971109 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" Oct 30 19:54:57.067180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.037426 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b77699f-9a88-4dde-94f0-feee3d4f01e8-kube-api-access-glh7r\" (UniqueName: \"kubernetes.io/projected/5b77699f-9a88-4dde-94f0-feee3d4f01e8-kube-api-access-glh7r\") pod \"5b77699f-9a88-4dde-94f0-feee3d4f01e8\" (UID: \"5b77699f-9a88-4dde-94f0-feee3d4f01e8\") " Oct 30 19:54:57.067180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.037536 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b77699f-9a88-4dde-94f0-feee3d4f01e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b77699f-9a88-4dde-94f0-feee3d4f01e8-plugins-config-volume\") pod \"5b77699f-9a88-4dde-94f0-feee3d4f01e8\" (UID: \"5b77699f-9a88-4dde-94f0-feee3d4f01e8\") " Oct 30 19:54:57.067180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.038373 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da240f1c-8dd4-4205-9985-92f3e42cb8e5-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:57.067180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.065502 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b77699f-9a88-4dde-94f0-feee3d4f01e8-plugins-config-volume" pod "5b77699f-9a88-4dde-94f0-feee3d4f01e8" (UID: "5b77699f-9a88-4dde-94f0-feee3d4f01e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:54:57.142188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.132953 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:57.180194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.167467 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b77699f-9a88-4dde-94f0-feee3d4f01e8-kube-api-access-glh7r" pod "5b77699f-9a88-4dde-94f0-feee3d4f01e8" (UID: "5b77699f-9a88-4dde-94f0-feee3d4f01e8"). InnerVolumeSpecName "kube-api-access-glh7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:54:57.180194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.171440 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glh7r\" (UniqueName: \"kubernetes.io/projected/5b77699f-9a88-4dde-94f0-feee3d4f01e8-kube-api-access-glh7r\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:57.180194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.171480 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b77699f-9a88-4dde-94f0-feee3d4f01e8-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:54:57.271483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.271445 5091 generic.go:356] "Generic (PLEG): container finished" podID="5b77699f-9a88-4dde-94f0-feee3d4f01e8" containerID="3987717a8f9ef800a474b3828c76e4c22bc17df4b76967d077406855ac11a595" exitCode=0 Oct 30 19:54:57.272025 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.271980 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" event={"ID":"5b77699f-9a88-4dde-94f0-feee3d4f01e8","Type":"ContainerDied","Data":"3987717a8f9ef800a474b3828c76e4c22bc17df4b76967d077406855ac11a595"} Oct 30 19:54:57.272174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.272089 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" event={"ID":"5b77699f-9a88-4dde-94f0-feee3d4f01e8","Type":"ContainerDied","Data":"df7142542cf9195341cbedba985ccd363ef5b85986cd0637faba1b68f76ad882"} Oct 30 19:54:57.272265 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.271997 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" Oct 30 19:54:57.272466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.272445 5091 scope.go:122] "RemoveContainer" containerID="3987717a8f9ef800a474b3828c76e4c22bc17df4b76967d077406855ac11a595" Oct 30 19:54:57.296861 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.296802 5091 generic.go:356] "Generic (PLEG): container finished" podID="da240f1c-8dd4-4205-9985-92f3e42cb8e5" containerID="4e2fdd00bbdfbe9c5e0449416f8231c40db1019b1c9476ba07d41c6f44f56c70" exitCode=0 Oct 30 19:54:57.303553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.303532 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" Oct 30 19:54:57.304670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.304647 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" event={"ID":"da240f1c-8dd4-4205-9985-92f3e42cb8e5","Type":"ContainerDied","Data":"4e2fdd00bbdfbe9c5e0449416f8231c40db1019b1c9476ba07d41c6f44f56c70"} Oct 30 19:54:57.304849 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.304830 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72" event={"ID":"da240f1c-8dd4-4205-9985-92f3e42cb8e5","Type":"ContainerDied","Data":"9b89cb2b086334f06c6700d041c98d88920fe108b3c7d9654ad928a99c5b77a7"} Oct 30 19:54:57.305131 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.304667 5091 scope.go:122] "RemoveContainer" containerID="3987717a8f9ef800a474b3828c76e4c22bc17df4b76967d077406855ac11a595" Oct 30 19:54:57.305743 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:54:57.305697 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3987717a8f9ef800a474b3828c76e4c22bc17df4b76967d077406855ac11a595\": container with ID starting with 3987717a8f9ef800a474b3828c76e4c22bc17df4b76967d077406855ac11a595 not found: ID does not exist" containerID="3987717a8f9ef800a474b3828c76e4c22bc17df4b76967d077406855ac11a595" Oct 30 19:54:57.305837 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.305756 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3987717a8f9ef800a474b3828c76e4c22bc17df4b76967d077406855ac11a595"} err="failed to get container status \"3987717a8f9ef800a474b3828c76e4c22bc17df4b76967d077406855ac11a595\": rpc error: code = NotFound desc = could not find container \"3987717a8f9ef800a474b3828c76e4c22bc17df4b76967d077406855ac11a595\": container with ID starting with 3987717a8f9ef800a474b3828c76e4c22bc17df4b76967d077406855ac11a595 not found: ID does not exist" Oct 30 19:54:57.305837 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.305780 5091 scope.go:122] "RemoveContainer" containerID="4e2fdd00bbdfbe9c5e0449416f8231c40db1019b1c9476ba07d41c6f44f56c70" Oct 30 19:54:57.365503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.365454 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r"] Oct 30 19:54:57.375644 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.367538 5091 scope.go:122] "RemoveContainer" containerID="4e2fdd00bbdfbe9c5e0449416f8231c40db1019b1c9476ba07d41c6f44f56c70" Oct 30 19:54:57.376758 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:54:57.376715 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2fdd00bbdfbe9c5e0449416f8231c40db1019b1c9476ba07d41c6f44f56c70\": container with ID starting with 4e2fdd00bbdfbe9c5e0449416f8231c40db1019b1c9476ba07d41c6f44f56c70 not found: ID does not exist" containerID="4e2fdd00bbdfbe9c5e0449416f8231c40db1019b1c9476ba07d41c6f44f56c70" Oct 30 19:54:57.376980 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.376949 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2fdd00bbdfbe9c5e0449416f8231c40db1019b1c9476ba07d41c6f44f56c70"} err="failed to get container status \"4e2fdd00bbdfbe9c5e0449416f8231c40db1019b1c9476ba07d41c6f44f56c70\": rpc error: code = NotFound desc = could not find container \"4e2fdd00bbdfbe9c5e0449416f8231c40db1019b1c9476ba07d41c6f44f56c70\": container with ID starting with 4e2fdd00bbdfbe9c5e0449416f8231c40db1019b1c9476ba07d41c6f44f56c70 not found: ID does not exist" Oct 30 19:54:57.442293 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.442204 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r"] Oct 30 19:54:57.464278 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.464206 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72"] Oct 30 19:54:57.490903 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.490811 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-426bab-759c7cdb9b-x2s72"] Oct 30 19:54:57.548173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.534203 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:57.854171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:57.843239 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-426bab-c67495f5-4mq9r" podUID="5b77699f-9a88-4dde-94f0-feee3d4f01e8" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.165:9004: i/o timeout" Oct 30 19:54:58.484130 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:58.482351 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d" path="/var/lib/kubelet/pods/3e6fc02f-fa77-4ae9-9eed-150ee9e06e6d/volumes" Oct 30 19:54:58.484130 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:58.483027 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b77699f-9a88-4dde-94f0-feee3d4f01e8" path="/var/lib/kubelet/pods/5b77699f-9a88-4dde-94f0-feee3d4f01e8/volumes" Oct 30 19:54:58.484130 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:58.483658 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da240f1c-8dd4-4205-9985-92f3e42cb8e5" path="/var/lib/kubelet/pods/da240f1c-8dd4-4205-9985-92f3e42cb8e5/volumes" Oct 30 19:54:58.755229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:58.746278 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:58.755229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:58.747420 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:59.129891 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:59.129806 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:54:59.465475 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:59.464238 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:59.727322 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:59.727227 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" podUID="0025d83b-0a0d-455d-9634-76492646d579" containerName="main" probeResult="failure" output="Get \"https://10.132.0.163:8000/ready\": context deadline exceeded" Oct 30 19:54:59.871860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:59.751250 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 19:54:59.872886 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:59.751304 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:54:59.873090 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:59.873061 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:59.873314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:59.873295 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:54:59.883744 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:59.874227 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:59.883744 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:59.752214 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:54:59.883744 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:54:59.882970 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:55:00.265339 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:00.265303 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:55:00.580524 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:00.580438 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:55:01.038576 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:01.036854 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:02.023197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:02.023119 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:03.029737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.029691 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:03.032711 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.032691 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76"] Oct 30 19:55:03.033345 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.033331 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5b77699f-9a88-4dde-94f0-feee3d4f01e8" containerName="payload-pre-processing" Oct 30 19:55:03.033390 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.033346 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b77699f-9a88-4dde-94f0-feee3d4f01e8" containerName="payload-pre-processing" Oct 30 19:55:03.033390 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.033380 5091 cpu_manager.go:414] "containerMap: removing container" podUID="da240f1c-8dd4-4205-9985-92f3e42cb8e5" containerName="payload-processing" Oct 30 19:55:03.033390 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.033387 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da240f1c-8dd4-4205-9985-92f3e42cb8e5" containerName="payload-processing" Oct 30 19:55:03.033499 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.033488 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="da240f1c-8dd4-4205-9985-92f3e42cb8e5" containerName="payload-processing" Oct 30 19:55:03.033530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.033501 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b77699f-9a88-4dde-94f0-feee3d4f01e8" containerName="payload-pre-processing" Oct 30 19:55:03.038855 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.038839 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" Oct 30 19:55:03.085253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.085225 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:55:03.086114 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.085994 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:55:03.091720 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.091694 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76"] Oct 30 19:55:03.129214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.129131 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/359bcdc9-3351-4285-a1a9-9c45c75d542d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd469540-28p76\" (UID: \"359bcdc9-3351-4285-a1a9-9c45c75d542d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" Oct 30 19:55:03.129351 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.129256 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/359bcdc9-3351-4285-a1a9-9c45c75d542d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd469540-28p76\" (UID: \"359bcdc9-3351-4285-a1a9-9c45c75d542d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" Oct 30 19:55:03.129351 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.129287 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn992\" (UniqueName: \"kubernetes.io/projected/359bcdc9-3351-4285-a1a9-9c45c75d542d-kube-api-access-bn992\") pod \"maas-api-revoke-sub-bd469540-28p76\" (UID: \"359bcdc9-3351-4285-a1a9-9c45c75d542d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" Oct 30 19:55:03.243815 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.243748 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/359bcdc9-3351-4285-a1a9-9c45c75d542d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd469540-28p76\" (UID: \"359bcdc9-3351-4285-a1a9-9c45c75d542d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" Oct 30 19:55:03.326225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.322754 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/359bcdc9-3351-4285-a1a9-9c45c75d542d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd469540-28p76\" (UID: \"359bcdc9-3351-4285-a1a9-9c45c75d542d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" Oct 30 19:55:03.341883 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.321968 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/359bcdc9-3351-4285-a1a9-9c45c75d542d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bd469540-28p76\" (UID: \"359bcdc9-3351-4285-a1a9-9c45c75d542d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" Oct 30 19:55:03.342193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.342167 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn992\" (UniqueName: \"kubernetes.io/projected/359bcdc9-3351-4285-a1a9-9c45c75d542d-kube-api-access-bn992\") pod \"maas-api-revoke-sub-bd469540-28p76\" (UID: \"359bcdc9-3351-4285-a1a9-9c45c75d542d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" Oct 30 19:55:03.381168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.366640 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/359bcdc9-3351-4285-a1a9-9c45c75d542d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bd469540-28p76\" (UID: \"359bcdc9-3351-4285-a1a9-9c45c75d542d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" Oct 30 19:55:03.381168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.376954 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:03.423170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.420884 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn992\" (UniqueName: \"kubernetes.io/projected/359bcdc9-3351-4285-a1a9-9c45c75d542d-kube-api-access-bn992\") pod \"maas-api-revoke-sub-bd469540-28p76\" (UID: \"359bcdc9-3351-4285-a1a9-9c45c75d542d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" Oct 30 19:55:03.503693 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.503660 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq"] Oct 30 19:55:03.509668 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.509639 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-b17e94-dr4cq"] Oct 30 19:55:03.542277 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.542241 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" podUID="afbd3cf9-21cd-4b40-94d2-df49a1146de6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:55:03.542440 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.542257 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" podUID="3c150c5f-1392-4a50-964d-a5e1593c2a19" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 19:55:03.696502 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.694364 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" Oct 30 19:55:03.704690 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.703457 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:03.974227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:03.974112 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:04.079242 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:04.076239 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76"] Oct 30 19:55:04.079242 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:55:04.078887 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359bcdc9_3351_4285_a1a9_9c45c75d542d.slice/crio-7f3796ab8e8bdfa453189b8a2686471431c6ac157db9fca250d2d1b86a82bd5c WatchSource:0}: Error finding container 7f3796ab8e8bdfa453189b8a2686471431c6ac157db9fca250d2d1b86a82bd5c: Status 404 returned error can't find the container with id 7f3796ab8e8bdfa453189b8a2686471431c6ac157db9fca250d2d1b86a82bd5c Oct 30 19:55:04.179170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:04.178918 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:04.442658 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:04.442587 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf8ca4d-efb0-477a-b21b-b83004cfdcf5" path="/var/lib/kubelet/pods/7bf8ca4d-efb0-477a-b21b-b83004cfdcf5/volumes" Oct 30 19:55:04.691235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:04.691099 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" event={"ID":"359bcdc9-3351-4285-a1a9-9c45c75d542d","Type":"ContainerStarted","Data":"cbced8b0b125a518e47944f60d9fb1e46c281d0ff4bcc223b81180ce2eed4607"} Oct 30 19:55:04.691235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:04.691202 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" event={"ID":"359bcdc9-3351-4285-a1a9-9c45c75d542d","Type":"ContainerStarted","Data":"7f3796ab8e8bdfa453189b8a2686471431c6ac157db9fca250d2d1b86a82bd5c"} Oct 30 19:55:04.720300 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:04.720250 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" podStartSLOduration=1.720236916 podStartE2EDuration="1.720236916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:04.716971746 +0000 UTC m=+1424.412179705" watchObservedRunningTime="2026-10-30 19:55:04.720236916 +0000 UTC m=+1424.415444874" Oct 30 19:55:05.020868 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:05.018582 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:05.652809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:05.651645 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:05.705372 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:05.705337 5091 generic.go:356] "Generic (PLEG): container finished" podID="359bcdc9-3351-4285-a1a9-9c45c75d542d" containerID="cbced8b0b125a518e47944f60d9fb1e46c281d0ff4bcc223b81180ce2eed4607" exitCode=0 Oct 30 19:55:05.705540 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:05.705446 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" event={"ID":"359bcdc9-3351-4285-a1a9-9c45c75d542d","Type":"ContainerDied","Data":"cbced8b0b125a518e47944f60d9fb1e46c281d0ff4bcc223b81180ce2eed4607"} Oct 30 19:55:05.849354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:05.849314 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:05.969156 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:05.969049 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk"] Oct 30 19:55:05.969376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:05.969347 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" podUID="1ec5bb43-1784-459b-87b0-a7bff9143031" containerName="istio-proxy" containerID="cri-o://7f151bda3ad0db35995ad38cacb1c3216ee1c29c9cb43a7c7f22aa4b612dd27e" gracePeriod=30 Oct 30 19:55:06.224220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:06.222053 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:06.531684 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:06.531651 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg"] Oct 30 19:55:06.536186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:06.536166 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" Oct 30 19:55:06.548842 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:06.548817 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg"] Oct 30 19:55:06.586123 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:06.586082 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:06.713174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:06.678481 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329cbe0d-56ce-4fb5-a1b7-470f6607b375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg\" (UID: \"329cbe0d-56ce-4fb5-a1b7-470f6607b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" Oct 30 19:55:06.713174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:06.678653 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329cbe0d-56ce-4fb5-a1b7-470f6607b375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg\" (UID: \"329cbe0d-56ce-4fb5-a1b7-470f6607b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" Oct 30 19:55:06.713174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:06.682327 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcrzh\" (UniqueName: \"kubernetes.io/projected/329cbe0d-56ce-4fb5-a1b7-470f6607b375-kube-api-access-kcrzh\") pod \"maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg\" (UID: \"329cbe0d-56ce-4fb5-a1b7-470f6607b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" Oct 30 19:55:06.791856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:06.791822 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329cbe0d-56ce-4fb5-a1b7-470f6607b375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg\" (UID: \"329cbe0d-56ce-4fb5-a1b7-470f6607b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" Oct 30 19:55:06.792301 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:06.792283 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329cbe0d-56ce-4fb5-a1b7-470f6607b375-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg\" (UID: \"329cbe0d-56ce-4fb5-a1b7-470f6607b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" Oct 30 19:55:06.792532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:06.792515 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329cbe0d-56ce-4fb5-a1b7-470f6607b375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg\" (UID: \"329cbe0d-56ce-4fb5-a1b7-470f6607b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" Oct 30 19:55:06.793180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:06.793125 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcrzh\" (UniqueName: \"kubernetes.io/projected/329cbe0d-56ce-4fb5-a1b7-470f6607b375-kube-api-access-kcrzh\") pod \"maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg\" (UID: \"329cbe0d-56ce-4fb5-a1b7-470f6607b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" Oct 30 19:55:06.798637 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:06.798577 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329cbe0d-56ce-4fb5-a1b7-470f6607b375-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg\" (UID: \"329cbe0d-56ce-4fb5-a1b7-470f6607b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" Oct 30 19:55:06.810516 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:06.810454 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcrzh\" (UniqueName: \"kubernetes.io/projected/329cbe0d-56ce-4fb5-a1b7-470f6607b375-kube-api-access-kcrzh\") pod \"maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg\" (UID: \"329cbe0d-56ce-4fb5-a1b7-470f6607b375\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" Oct 30 19:55:06.856680 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:06.856515 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" Oct 30 19:55:07.080666 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.080624 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" Oct 30 19:55:07.203901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.203862 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359bcdc9-3351-4285-a1a9-9c45c75d542d-kube-api-access-bn992\" (UniqueName: \"kubernetes.io/projected/359bcdc9-3351-4285-a1a9-9c45c75d542d-kube-api-access-bn992\") pod \"359bcdc9-3351-4285-a1a9-9c45c75d542d\" (UID: \"359bcdc9-3351-4285-a1a9-9c45c75d542d\") " Oct 30 19:55:07.204088 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.203986 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/359bcdc9-3351-4285-a1a9-9c45c75d542d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/359bcdc9-3351-4285-a1a9-9c45c75d542d-maas-api-openshift-ca\") pod \"359bcdc9-3351-4285-a1a9-9c45c75d542d\" (UID: \"359bcdc9-3351-4285-a1a9-9c45c75d542d\") " Oct 30 19:55:07.204192 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.204174 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/359bcdc9-3351-4285-a1a9-9c45c75d542d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/359bcdc9-3351-4285-a1a9-9c45c75d542d-maas-api-xks-ca\") pod \"359bcdc9-3351-4285-a1a9-9c45c75d542d\" (UID: \"359bcdc9-3351-4285-a1a9-9c45c75d542d\") " Oct 30 19:55:07.205775 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.205735 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359bcdc9-3351-4285-a1a9-9c45c75d542d-maas-api-openshift-ca" pod "359bcdc9-3351-4285-a1a9-9c45c75d542d" (UID: "359bcdc9-3351-4285-a1a9-9c45c75d542d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:07.207191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.207162 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/359bcdc9-3351-4285-a1a9-9c45c75d542d-maas-api-xks-ca" pod "359bcdc9-3351-4285-a1a9-9c45c75d542d" (UID: "359bcdc9-3351-4285-a1a9-9c45c75d542d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:07.207441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.207418 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359bcdc9-3351-4285-a1a9-9c45c75d542d-kube-api-access-bn992" pod "359bcdc9-3351-4285-a1a9-9c45c75d542d" (UID: "359bcdc9-3351-4285-a1a9-9c45c75d542d"). InnerVolumeSpecName "kube-api-access-bn992". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:07.227895 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.227869 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg"] Oct 30 19:55:07.228561 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:55:07.228528 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod329cbe0d_56ce_4fb5_a1b7_470f6607b375.slice/crio-f410698f44b16d74bb973cf6616fc44c259db335651f8ad753b3a9ca3425ecdb WatchSource:0}: Error finding container f410698f44b16d74bb973cf6616fc44c259db335651f8ad753b3a9ca3425ecdb: Status 404 returned error can't find the container with id f410698f44b16d74bb973cf6616fc44c259db335651f8ad753b3a9ca3425ecdb Oct 30 19:55:07.306410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.306325 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/359bcdc9-3351-4285-a1a9-9c45c75d542d-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:07.306410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.306364 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn992\" (UniqueName: \"kubernetes.io/projected/359bcdc9-3351-4285-a1a9-9c45c75d542d-kube-api-access-bn992\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:07.306410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.306382 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/359bcdc9-3351-4285-a1a9-9c45c75d542d-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:07.308691 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.308668 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" Oct 30 19:55:07.308819 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.308780 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" Oct 30 19:55:07.337771 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.337727 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv"] Oct 30 19:55:07.338909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.338882 5091 cpu_manager.go:414] "containerMap: removing container" podUID="359bcdc9-3351-4285-a1a9-9c45c75d542d" containerName="revoke-keys" Oct 30 19:55:07.338909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.338909 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="359bcdc9-3351-4285-a1a9-9c45c75d542d" containerName="revoke-keys" Oct 30 19:55:07.339103 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.339091 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="359bcdc9-3351-4285-a1a9-9c45c75d542d" containerName="revoke-keys" Oct 30 19:55:07.344556 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.344511 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" Oct 30 19:55:07.367188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.360615 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv"] Oct 30 19:55:07.408261 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.408222 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whxgk\" (UniqueName: \"kubernetes.io/projected/d9017622-5b22-40d9-9eb3-052cc0dbb14f-kube-api-access-whxgk\") pod \"maas-api-revoke-sub-6257ee75-vgkvv\" (UID: \"d9017622-5b22-40d9-9eb3-052cc0dbb14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" Oct 30 19:55:07.408438 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.408351 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9017622-5b22-40d9-9eb3-052cc0dbb14f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6257ee75-vgkvv\" (UID: \"d9017622-5b22-40d9-9eb3-052cc0dbb14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" Oct 30 19:55:07.408438 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.408394 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9017622-5b22-40d9-9eb3-052cc0dbb14f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6257ee75-vgkvv\" (UID: \"d9017622-5b22-40d9-9eb3-052cc0dbb14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" Oct 30 19:55:07.510782 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.509857 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9017622-5b22-40d9-9eb3-052cc0dbb14f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6257ee75-vgkvv\" (UID: \"d9017622-5b22-40d9-9eb3-052cc0dbb14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" Oct 30 19:55:07.510782 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.509900 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9017622-5b22-40d9-9eb3-052cc0dbb14f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6257ee75-vgkvv\" (UID: \"d9017622-5b22-40d9-9eb3-052cc0dbb14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" Oct 30 19:55:07.510782 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.510347 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whxgk\" (UniqueName: \"kubernetes.io/projected/d9017622-5b22-40d9-9eb3-052cc0dbb14f-kube-api-access-whxgk\") pod \"maas-api-revoke-sub-6257ee75-vgkvv\" (UID: \"d9017622-5b22-40d9-9eb3-052cc0dbb14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" Oct 30 19:55:07.511074 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.510911 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9017622-5b22-40d9-9eb3-052cc0dbb14f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6257ee75-vgkvv\" (UID: \"d9017622-5b22-40d9-9eb3-052cc0dbb14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" Oct 30 19:55:07.513860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.513838 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9017622-5b22-40d9-9eb3-052cc0dbb14f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6257ee75-vgkvv\" (UID: \"d9017622-5b22-40d9-9eb3-052cc0dbb14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" Oct 30 19:55:07.536039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.536008 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whxgk\" (UniqueName: \"kubernetes.io/projected/d9017622-5b22-40d9-9eb3-052cc0dbb14f-kube-api-access-whxgk\") pod \"maas-api-revoke-sub-6257ee75-vgkvv\" (UID: \"d9017622-5b22-40d9-9eb3-052cc0dbb14f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" Oct 30 19:55:07.555235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.555198 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:07.698654 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.698552 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" Oct 30 19:55:07.758617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.758445 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:07.813186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.810999 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" event={"ID":"329cbe0d-56ce-4fb5-a1b7-470f6607b375","Type":"ContainerStarted","Data":"65347e8ce1b71b107934c01bc5ea0dc0fd0313852cb970bc87073e87d87ce10e"} Oct 30 19:55:07.813186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.811061 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" event={"ID":"329cbe0d-56ce-4fb5-a1b7-470f6607b375","Type":"ContainerStarted","Data":"f410698f44b16d74bb973cf6616fc44c259db335651f8ad753b3a9ca3425ecdb"} Oct 30 19:55:07.826271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.813751 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" event={"ID":"359bcdc9-3351-4285-a1a9-9c45c75d542d","Type":"ContainerDied","Data":"7f3796ab8e8bdfa453189b8a2686471431c6ac157db9fca250d2d1b86a82bd5c"} Oct 30 19:55:07.826271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.813797 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f3796ab8e8bdfa453189b8a2686471431c6ac157db9fca250d2d1b86a82bd5c" Oct 30 19:55:07.826271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.813852 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76" Oct 30 19:55:07.844250 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:07.842427 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" podStartSLOduration=1.842376737 podStartE2EDuration="1.842376737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:07.840975781 +0000 UTC m=+1427.536183742" watchObservedRunningTime="2026-10-30 19:55:07.842376737 +0000 UTC m=+1427.537584696" Oct 30 19:55:08.038902 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.038842 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv"] Oct 30 19:55:08.581663 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.581632 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5"] Oct 30 19:55:08.587301 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.587277 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.620065 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.620034 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5"] Oct 30 19:55:08.642311 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.642280 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj"] Oct 30 19:55:08.646883 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.646860 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.754851 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.754921 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.754999 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.755044 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.756444 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.756583 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.756629 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.756658 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx9c9\" (UniqueName: \"kubernetes.io/projected/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-kube-api-access-xx9c9\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.756718 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.756783 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.756820 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.756849 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.756888 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.756913 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j7mk\" (UniqueName: \"kubernetes.io/projected/34968af2-47bb-4c0a-ba8b-974ae9b70f30-kube-api-access-2j7mk\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.756946 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.756989 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.757093 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.757115 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.764378 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.759944 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj"] Oct 30 19:55:08.819253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.806492 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:08.840162 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.833702 5091 generic.go:356] "Generic (PLEG): container finished" podID="329cbe0d-56ce-4fb5-a1b7-470f6607b375" containerID="65347e8ce1b71b107934c01bc5ea0dc0fd0313852cb970bc87073e87d87ce10e" exitCode=0 Oct 30 19:55:08.840162 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.833771 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" event={"ID":"329cbe0d-56ce-4fb5-a1b7-470f6607b375","Type":"ContainerDied","Data":"65347e8ce1b71b107934c01bc5ea0dc0fd0313852cb970bc87073e87d87ce10e"} Oct 30 19:55:08.840162 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.838239 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" event={"ID":"d9017622-5b22-40d9-9eb3-052cc0dbb14f","Type":"ContainerStarted","Data":"de54bd877557292d6cead3ca1e00b99563746167da6124c1e137ba10d9c91fc2"} Oct 30 19:55:08.840162 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.838301 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" event={"ID":"d9017622-5b22-40d9-9eb3-052cc0dbb14f","Type":"ContainerStarted","Data":"895fb5136c5110b9b06cf662cdb8ae98519d0c17797a9f6642fe8f77040bd2af"} Oct 30 19:55:08.848750 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.845196 5091 generic.go:356] "Generic (PLEG): container finished" podID="afbd3cf9-21cd-4b40-94d2-df49a1146de6" containerID="6eb83afccbfa90bb7d61d4072b2963763176e07c05eb2fb897715a80281c8b1a" exitCode=0 Oct 30 19:55:08.848750 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.845423 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" event={"ID":"afbd3cf9-21cd-4b40-94d2-df49a1146de6","Type":"ContainerDied","Data":"6eb83afccbfa90bb7d61d4072b2963763176e07c05eb2fb897715a80281c8b1a"} Oct 30 19:55:08.850854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.850832 5091 generic.go:356] "Generic (PLEG): container finished" podID="3c150c5f-1392-4a50-964d-a5e1593c2a19" containerID="88dd0237e19085a5e64ecc9e1cc25d9923d11fa3cb3258495b84047907f1a658" exitCode=0 Oct 30 19:55:08.850955 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.850877 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" event={"ID":"3c150c5f-1392-4a50-964d-a5e1593c2a19","Type":"ContainerDied","Data":"88dd0237e19085a5e64ecc9e1cc25d9923d11fa3cb3258495b84047907f1a658"} Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.857988 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858025 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858052 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx9c9\" (UniqueName: \"kubernetes.io/projected/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-kube-api-access-xx9c9\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858110 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858168 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858197 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858216 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858241 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858256 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j7mk\" (UniqueName: \"kubernetes.io/projected/34968af2-47bb-4c0a-ba8b-974ae9b70f30-kube-api-access-2j7mk\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858282 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858306 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858365 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858381 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858410 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858448 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858515 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858542 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.858729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858584 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.864215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.858984 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.864215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.859640 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.864215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.859987 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.864215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.860876 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-workload-certs\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.864215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.861749 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.865305 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.865255 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istiod-ca-cert\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.866541 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.866504 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.867464 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.867423 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-workload-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.867671 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.867640 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-credential-socket\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.867789 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.867777 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-data\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.872379 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.872317 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.872504 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.872476 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.893297 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.887877 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-envoy\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.898083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.896070 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j7mk\" (UniqueName: \"kubernetes.io/projected/34968af2-47bb-4c0a-ba8b-974ae9b70f30-kube-api-access-2j7mk\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.898083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.896216 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx9c9\" (UniqueName: \"kubernetes.io/projected/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-kube-api-access-xx9c9\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.898083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.897521 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-podinfo\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.898083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.897642 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:08.898083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.897850 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-token\") pod \"e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:08.907582 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.907558 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:09.012916 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:08.987176 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:09.033781 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.032949 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" podStartSLOduration=2.032928873 podStartE2EDuration="2.032928873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:08.919850939 +0000 UTC m=+1428.615058899" watchObservedRunningTime="2026-10-30 19:55:09.032928873 +0000 UTC m=+1428.728136832" Oct 30 19:55:09.049485 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.049454 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn"] Oct 30 19:55:09.138119 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.138092 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" Oct 30 19:55:09.245412 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.245360 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:09.262853 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.262825 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" Oct 30 19:55:09.304604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.303727 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c150c5f-1392-4a50-964d-a5e1593c2a19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c150c5f-1392-4a50-964d-a5e1593c2a19-plugins-config-volume\") pod \"3c150c5f-1392-4a50-964d-a5e1593c2a19\" (UID: \"3c150c5f-1392-4a50-964d-a5e1593c2a19\") " Oct 30 19:55:09.304604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.303790 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c150c5f-1392-4a50-964d-a5e1593c2a19-kube-api-access-b6k5n\" (UniqueName: \"kubernetes.io/projected/3c150c5f-1392-4a50-964d-a5e1593c2a19-kube-api-access-b6k5n\") pod \"3c150c5f-1392-4a50-964d-a5e1593c2a19\" (UID: \"3c150c5f-1392-4a50-964d-a5e1593c2a19\") " Oct 30 19:55:09.304604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.303909 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afbd3cf9-21cd-4b40-94d2-df49a1146de6-kube-api-access-2sxwc\" (UniqueName: \"kubernetes.io/projected/afbd3cf9-21cd-4b40-94d2-df49a1146de6-kube-api-access-2sxwc\") pod \"afbd3cf9-21cd-4b40-94d2-df49a1146de6\" (UID: \"afbd3cf9-21cd-4b40-94d2-df49a1146de6\") " Oct 30 19:55:09.304604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.303975 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afbd3cf9-21cd-4b40-94d2-df49a1146de6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afbd3cf9-21cd-4b40-94d2-df49a1146de6-plugins-config-volume\") pod \"afbd3cf9-21cd-4b40-94d2-df49a1146de6\" (UID: \"afbd3cf9-21cd-4b40-94d2-df49a1146de6\") " Oct 30 19:55:09.305042 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.305009 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afbd3cf9-21cd-4b40-94d2-df49a1146de6-plugins-config-volume" pod "afbd3cf9-21cd-4b40-94d2-df49a1146de6" (UID: "afbd3cf9-21cd-4b40-94d2-df49a1146de6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:09.305042 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.305017 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c150c5f-1392-4a50-964d-a5e1593c2a19-plugins-config-volume" pod "3c150c5f-1392-4a50-964d-a5e1593c2a19" (UID: "3c150c5f-1392-4a50-964d-a5e1593c2a19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:09.307044 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.307015 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afbd3cf9-21cd-4b40-94d2-df49a1146de6-kube-api-access-2sxwc" pod "afbd3cf9-21cd-4b40-94d2-df49a1146de6" (UID: "afbd3cf9-21cd-4b40-94d2-df49a1146de6"). InnerVolumeSpecName "kube-api-access-2sxwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:09.307610 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.307588 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c150c5f-1392-4a50-964d-a5e1593c2a19-kube-api-access-b6k5n" pod "3c150c5f-1392-4a50-964d-a5e1593c2a19" (UID: "3c150c5f-1392-4a50-964d-a5e1593c2a19"). InnerVolumeSpecName "kube-api-access-b6k5n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:09.405848 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.405801 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sxwc\" (UniqueName: \"kubernetes.io/projected/afbd3cf9-21cd-4b40-94d2-df49a1146de6-kube-api-access-2sxwc\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:09.405906 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.405855 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afbd3cf9-21cd-4b40-94d2-df49a1146de6-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:09.405906 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.405870 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c150c5f-1392-4a50-964d-a5e1593c2a19-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:09.405906 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.405884 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6k5n\" (UniqueName: \"kubernetes.io/projected/3c150c5f-1392-4a50-964d-a5e1593c2a19-kube-api-access-b6k5n\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:09.409220 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:55:09.409186 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34968af2_47bb_4c0a_ba8b_974ae9b70f30.slice/crio-82e43ff5f02bcb5974edde2c35216eb7e1241613d264197fceb3d09b1265d09d WatchSource:0}: Error finding container 82e43ff5f02bcb5974edde2c35216eb7e1241613d264197fceb3d09b1265d09d: Status 404 returned error can't find the container with id 82e43ff5f02bcb5974edde2c35216eb7e1241613d264197fceb3d09b1265d09d Oct 30 19:55:09.410944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.410861 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5"] Oct 30 19:55:09.411809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.411777 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:55:09.411909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.411852 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:55:09.411909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.411880 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:55:09.424072 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.424046 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj"] Oct 30 19:55:09.424225 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:55:09.424110 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28281f9_b9a0_4522_9336_ea91aa6a4ce0.slice/crio-2970ee9bf48a50b715734ea68a0d10b0c0deebf1e979bc3c86697640a80a3a1e WatchSource:0}: Error finding container 2970ee9bf48a50b715734ea68a0d10b0c0deebf1e979bc3c86697640a80a3a1e: Status 404 returned error can't find the container with id 2970ee9bf48a50b715734ea68a0d10b0c0deebf1e979bc3c86697640a80a3a1e Oct 30 19:55:09.426657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.426626 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:55:09.426753 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.426699 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:55:09.426753 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.426731 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:55:09.865917 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.865885 5091 generic.go:356] "Generic (PLEG): container finished" podID="d9017622-5b22-40d9-9eb3-052cc0dbb14f" containerID="de54bd877557292d6cead3ca1e00b99563746167da6124c1e137ba10d9c91fc2" exitCode=0 Oct 30 19:55:09.866506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.865981 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" event={"ID":"d9017622-5b22-40d9-9eb3-052cc0dbb14f","Type":"ContainerDied","Data":"de54bd877557292d6cead3ca1e00b99563746167da6124c1e137ba10d9c91fc2"} Oct 30 19:55:09.868876 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.868830 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" event={"ID":"34968af2-47bb-4c0a-ba8b-974ae9b70f30","Type":"ContainerStarted","Data":"b0fb368a71550075b1bbe6b65a5a95525b37a2aa52a484d49b50fed08ed127e8"} Oct 30 19:55:09.869085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.868880 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" event={"ID":"34968af2-47bb-4c0a-ba8b-974ae9b70f30","Type":"ContainerStarted","Data":"82e43ff5f02bcb5974edde2c35216eb7e1241613d264197fceb3d09b1265d09d"} Oct 30 19:55:09.870825 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.870800 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" Oct 30 19:55:09.877457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.870840 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk" event={"ID":"afbd3cf9-21cd-4b40-94d2-df49a1146de6","Type":"ContainerDied","Data":"c8162b05cad5f69434aed90f75f28f335a13df4052cd77e35594c862a5941999"} Oct 30 19:55:09.877457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.871387 5091 scope.go:122] "RemoveContainer" containerID="6eb83afccbfa90bb7d61d4072b2963763176e07c05eb2fb897715a80281c8b1a" Oct 30 19:55:09.877457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.873668 5091 generic.go:356] "Generic (PLEG): container finished" podID="8d15b25d-07cf-4e6c-8446-60915a92b92e" containerID="1244a685b6d3d0480d68cff2d53cc40d24045899cd770f19021a2b9c170cd42b" exitCode=0 Oct 30 19:55:09.877457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.873736 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" event={"ID":"8d15b25d-07cf-4e6c-8446-60915a92b92e","Type":"ContainerDied","Data":"1244a685b6d3d0480d68cff2d53cc40d24045899cd770f19021a2b9c170cd42b"} Oct 30 19:55:09.877457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.874264 5091 scope.go:122] "RemoveContainer" containerID="1244a685b6d3d0480d68cff2d53cc40d24045899cd770f19021a2b9c170cd42b" Oct 30 19:55:09.877457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.877158 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:55:09.877457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.877235 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:55:09.877457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.877279 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:55:09.891958 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.890181 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" event={"ID":"f28281f9-b9a0-4522-9336-ea91aa6a4ce0","Type":"ContainerStarted","Data":"3bc3fad560e5f440d0e6b88b03e49522ea90e03df21cc41927de3c8ac0880e27"} Oct 30 19:55:09.891958 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.890238 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" event={"ID":"f28281f9-b9a0-4522-9336-ea91aa6a4ce0","Type":"ContainerStarted","Data":"2970ee9bf48a50b715734ea68a0d10b0c0deebf1e979bc3c86697640a80a3a1e"} Oct 30 19:55:09.893345 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.893323 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" Oct 30 19:55:09.897281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.896970 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc" event={"ID":"3c150c5f-1392-4a50-964d-a5e1593c2a19","Type":"ContainerDied","Data":"a8ff125e85deb9654814e7299959d76dd3c22578f6388ee9fea5b01cd5242b74"} Oct 30 19:55:09.897281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.897029 5091 scope.go:122] "RemoveContainer" containerID="88dd0237e19085a5e64ecc9e1cc25d9923d11fa3cb3258495b84047907f1a658" Oct 30 19:55:09.910909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.910817 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:10.009321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:09.999654 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:10.009321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.005929 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:10.020295 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.020219 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" podStartSLOduration=2.020195899 podStartE2EDuration="2.020195899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:09.957446344 +0000 UTC m=+1429.652654301" watchObservedRunningTime="2026-10-30 19:55:10.020195899 +0000 UTC m=+1429.715403864" Oct 30 19:55:10.049173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.044520 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" podStartSLOduration=2.044497561 podStartE2EDuration="2.044497561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:10.041303102 +0000 UTC m=+1429.736511058" watchObservedRunningTime="2026-10-30 19:55:10.044497561 +0000 UTC m=+1429.739705520" Oct 30 19:55:10.192592 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.192543 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk"] Oct 30 19:55:10.229592 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.229535 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-b17e94-b96ddfb96-9dkqk"] Oct 30 19:55:10.250479 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.250122 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs"] Oct 30 19:55:10.252259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.251022 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" podUID="27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" containerName="istio-proxy" containerID="cri-o://4e4fb733f02cefdb7a1371f4e73082082dced208f7e00bbdd37b1f4a07f108d5" gracePeriod=30 Oct 30 19:55:10.276428 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.276353 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc"] Oct 30 19:55:10.622247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.622212 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afbd3cf9-21cd-4b40-94d2-df49a1146de6" path="/var/lib/kubelet/pods/afbd3cf9-21cd-4b40-94d2-df49a1146de6/volumes" Oct 30 19:55:10.622793 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.622767 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-b17e94-64ccdf6bb7-xfbnc"] Oct 30 19:55:10.634268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.634244 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" Oct 30 19:55:10.743177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.743111 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/329cbe0d-56ce-4fb5-a1b7-470f6607b375-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329cbe0d-56ce-4fb5-a1b7-470f6607b375-maas-api-xks-ca\") pod \"329cbe0d-56ce-4fb5-a1b7-470f6607b375\" (UID: \"329cbe0d-56ce-4fb5-a1b7-470f6607b375\") " Oct 30 19:55:10.743370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.743261 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/329cbe0d-56ce-4fb5-a1b7-470f6607b375-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329cbe0d-56ce-4fb5-a1b7-470f6607b375-maas-api-openshift-ca\") pod \"329cbe0d-56ce-4fb5-a1b7-470f6607b375\" (UID: \"329cbe0d-56ce-4fb5-a1b7-470f6607b375\") " Oct 30 19:55:10.743438 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.743365 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/329cbe0d-56ce-4fb5-a1b7-470f6607b375-kube-api-access-kcrzh\" (UniqueName: \"kubernetes.io/projected/329cbe0d-56ce-4fb5-a1b7-470f6607b375-kube-api-access-kcrzh\") pod \"329cbe0d-56ce-4fb5-a1b7-470f6607b375\" (UID: \"329cbe0d-56ce-4fb5-a1b7-470f6607b375\") " Oct 30 19:55:10.744524 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.744497 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/329cbe0d-56ce-4fb5-a1b7-470f6607b375-maas-api-openshift-ca" pod "329cbe0d-56ce-4fb5-a1b7-470f6607b375" (UID: "329cbe0d-56ce-4fb5-a1b7-470f6607b375"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:10.815217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.812830 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/329cbe0d-56ce-4fb5-a1b7-470f6607b375-kube-api-access-kcrzh" pod "329cbe0d-56ce-4fb5-a1b7-470f6607b375" (UID: "329cbe0d-56ce-4fb5-a1b7-470f6607b375"). InnerVolumeSpecName "kube-api-access-kcrzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:10.815217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.813361 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/329cbe0d-56ce-4fb5-a1b7-470f6607b375-maas-api-xks-ca" pod "329cbe0d-56ce-4fb5-a1b7-470f6607b375" (UID: "329cbe0d-56ce-4fb5-a1b7-470f6607b375"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:10.865393 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.865360 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/329cbe0d-56ce-4fb5-a1b7-470f6607b375-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:10.865646 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.865627 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/329cbe0d-56ce-4fb5-a1b7-470f6607b375-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:10.865778 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.865764 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcrzh\" (UniqueName: \"kubernetes.io/projected/329cbe0d-56ce-4fb5-a1b7-470f6607b375-kube-api-access-kcrzh\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:10.929391 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.929328 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 19:55:10.929895 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.929394 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" podUID="34968af2-47bb-4c0a-ba8b-974ae9b70f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 19:55:10.939956 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.939913 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" event={"ID":"329cbe0d-56ce-4fb5-a1b7-470f6607b375","Type":"ContainerDied","Data":"f410698f44b16d74bb973cf6616fc44c259db335651f8ad753b3a9ca3425ecdb"} Oct 30 19:55:10.940175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.939985 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f410698f44b16d74bb973cf6616fc44c259db335651f8ad753b3a9ca3425ecdb" Oct 30 19:55:10.940175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.940053 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg" Oct 30 19:55:10.944010 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:10.943986 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" event={"ID":"8d15b25d-07cf-4e6c-8446-60915a92b92e","Type":"ContainerStarted","Data":"179fdd028d79e8880f229f84ed5828d4c8addc2eb8c6ca4e5a85538d1a83ffec"} Oct 30 19:55:11.022610 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:11.018653 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 19:55:11.022610 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:11.018712 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" podUID="f28281f9-b9a0-4522-9336-ea91aa6a4ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 19:55:11.037195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:11.032195 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:11.605166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:11.579929 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:55:11.718186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:11.681887 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 19:55:11.718186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:11.681949 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" podUID="8d15b25d-07cf-4e6c-8446-60915a92b92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 19:55:12.133272 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:11.933778 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 19:55:12.133272 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:11.933837 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" podUID="34968af2-47bb-4c0a-ba8b-974ae9b70f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 19:55:12.449220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:12.196054 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:12.557188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:12.485272 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:12.557188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:12.504764 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 19:55:12.557188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:12.504837 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" podUID="f28281f9-b9a0-4522-9336-ea91aa6a4ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 19:55:12.557188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:12.522308 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:12.791391 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:12.791354 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" podUID="0025d83b-0a0d-455d-9634-76492646d579" containerName="main" probeResult="failure" output="Get \"https://10.132.0.163:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:12.792314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:12.791690 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" Oct 30 19:55:12.828328 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:12.828262 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:12.868252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:12.868217 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 19:55:13.015170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.008375 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" podUID="8d15b25d-07cf-4e6c-8446-60915a92b92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 19:55:13.015170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:12.948769 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:13.015170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:12.901079 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:13.015170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:12.924302 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 19:55:13.015170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.008555 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" podUID="34968af2-47bb-4c0a-ba8b-974ae9b70f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 19:55:13.015170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.008309 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 19:55:13.015170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.008601 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" podUID="f28281f9-b9a0-4522-9336-ea91aa6a4ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 19:55:13.123367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.056475 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c150c5f-1392-4a50-964d-a5e1593c2a19" path="/var/lib/kubelet/pods/3c150c5f-1392-4a50-964d-a5e1593c2a19/volumes" Oct 30 19:55:13.123367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.084376 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" event={"ID":"d9017622-5b22-40d9-9eb3-052cc0dbb14f","Type":"ContainerDied","Data":"895fb5136c5110b9b06cf662cdb8ae98519d0c17797a9f6642fe8f77040bd2af"} Oct 30 19:55:13.123367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.084436 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="895fb5136c5110b9b06cf662cdb8ae98519d0c17797a9f6642fe8f77040bd2af" Oct 30 19:55:13.123367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.085534 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9017622-5b22-40d9-9eb3-052cc0dbb14f-kube-api-access-whxgk\" (UniqueName: \"kubernetes.io/projected/d9017622-5b22-40d9-9eb3-052cc0dbb14f-kube-api-access-whxgk\") pod \"d9017622-5b22-40d9-9eb3-052cc0dbb14f\" (UID: \"d9017622-5b22-40d9-9eb3-052cc0dbb14f\") " Oct 30 19:55:13.123367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.085592 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d9017622-5b22-40d9-9eb3-052cc0dbb14f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9017622-5b22-40d9-9eb3-052cc0dbb14f-maas-api-xks-ca\") pod \"d9017622-5b22-40d9-9eb3-052cc0dbb14f\" (UID: \"d9017622-5b22-40d9-9eb3-052cc0dbb14f\") " Oct 30 19:55:13.123367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.085639 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9017622-5b22-40d9-9eb3-052cc0dbb14f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9017622-5b22-40d9-9eb3-052cc0dbb14f-maas-api-openshift-ca\") pod \"d9017622-5b22-40d9-9eb3-052cc0dbb14f\" (UID: \"d9017622-5b22-40d9-9eb3-052cc0dbb14f\") " Oct 30 19:55:13.123367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.111527 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9017622-5b22-40d9-9eb3-052cc0dbb14f-maas-api-openshift-ca" pod "d9017622-5b22-40d9-9eb3-052cc0dbb14f" (UID: "d9017622-5b22-40d9-9eb3-052cc0dbb14f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:13.141505 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.141472 5091 generic.go:356] "Generic (PLEG): container finished" podID="1ec5bb43-1784-459b-87b0-a7bff9143031" containerID="7f151bda3ad0db35995ad38cacb1c3216ee1c29c9cb43a7c7f22aa4b612dd27e" exitCode=0 Oct 30 19:55:13.143190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.143163 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" event={"ID":"1ec5bb43-1784-459b-87b0-a7bff9143031","Type":"ContainerDied","Data":"7f151bda3ad0db35995ad38cacb1c3216ee1c29c9cb43a7c7f22aa4b612dd27e"} Oct 30 19:55:13.154355 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.154329 5091 scope.go:122] "RemoveContainer" containerID="a7002857e0db2a31df496bacaa0b755a2b7029a85cc7edc7108d4cf5469309e2" Oct 30 19:55:13.184171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.157620 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:55:13.254233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.201879 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d9017622-5b22-40d9-9eb3-052cc0dbb14f-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.314090 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.304132 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-workload-certs\") pod \"1ec5bb43-1784-459b-87b0-a7bff9143031\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " Oct 30 19:55:13.335544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.335508 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-workload-certs" pod "1ec5bb43-1784-459b-87b0-a7bff9143031" (UID: "1ec5bb43-1784-459b-87b0-a7bff9143031"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:13.336080 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.336052 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-data\") pod \"1ec5bb43-1784-459b-87b0-a7bff9143031\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " Oct 30 19:55:13.342068 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.342045 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-credential-socket\") pod \"1ec5bb43-1784-459b-87b0-a7bff9143031\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " Oct 30 19:55:13.370619 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.370587 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-envoy\") pod \"1ec5bb43-1784-459b-87b0-a7bff9143031\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " Oct 30 19:55:13.370924 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.370902 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec5bb43-1784-459b-87b0-a7bff9143031-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec5bb43-1784-459b-87b0-a7bff9143031-istiod-ca-cert\") pod \"1ec5bb43-1784-459b-87b0-a7bff9143031\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " Oct 30 19:55:13.371097 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.371079 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-podinfo\") pod \"1ec5bb43-1784-459b-87b0-a7bff9143031\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " Oct 30 19:55:13.371255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.371237 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec5bb43-1784-459b-87b0-a7bff9143031-kube-api-access-ql298\" (UniqueName: \"kubernetes.io/projected/1ec5bb43-1784-459b-87b0-a7bff9143031-kube-api-access-ql298\") pod \"1ec5bb43-1784-459b-87b0-a7bff9143031\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " Oct 30 19:55:13.371400 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.371385 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-workload-socket\") pod \"1ec5bb43-1784-459b-87b0-a7bff9143031\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " Oct 30 19:55:13.371626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.371605 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-token\" (UniqueName: \"kubernetes.io/projected/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-token\") pod \"1ec5bb43-1784-459b-87b0-a7bff9143031\" (UID: \"1ec5bb43-1784-459b-87b0-a7bff9143031\") " Oct 30 19:55:13.372752 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.372726 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.373267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.341422 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-data" pod "1ec5bb43-1784-459b-87b0-a7bff9143031" (UID: "1ec5bb43-1784-459b-87b0-a7bff9143031"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:13.374546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.374162 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec5bb43-1784-459b-87b0-a7bff9143031-istiod-ca-cert" pod "1ec5bb43-1784-459b-87b0-a7bff9143031" (UID: "1ec5bb43-1784-459b-87b0-a7bff9143031"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:13.379244 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.379217 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-workload-socket" pod "1ec5bb43-1784-459b-87b0-a7bff9143031" (UID: "1ec5bb43-1784-459b-87b0-a7bff9143031"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:13.379698 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.379674 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-credential-socket" pod "1ec5bb43-1784-459b-87b0-a7bff9143031" (UID: "1ec5bb43-1784-459b-87b0-a7bff9143031"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:13.445806 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.445740 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:13.487343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.487311 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ec5bb43-1784-459b-87b0-a7bff9143031-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.506244 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.487638 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.507340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.507306 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.507490 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.507461 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.626248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.625316 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9017622-5b22-40d9-9eb3-052cc0dbb14f-maas-api-xks-ca" pod "d9017622-5b22-40d9-9eb3-052cc0dbb14f" (UID: "d9017622-5b22-40d9-9eb3-052cc0dbb14f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:13.626248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.625656 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 19:55:13.626248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.625710 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" podUID="8d15b25d-07cf-4e6c-8446-60915a92b92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 19:55:13.626542 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.626316 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec5bb43-1784-459b-87b0-a7bff9143031-kube-api-access-ql298" pod "1ec5bb43-1784-459b-87b0-a7bff9143031" (UID: "1ec5bb43-1784-459b-87b0-a7bff9143031"). InnerVolumeSpecName "kube-api-access-ql298". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:13.644489 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.643810 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql298\" (UniqueName: \"kubernetes.io/projected/1ec5bb43-1784-459b-87b0-a7bff9143031-kube-api-access-ql298\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.644489 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.643846 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d9017622-5b22-40d9-9eb3-052cc0dbb14f-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.644489 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.644298 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-podinfo" pod "1ec5bb43-1784-459b-87b0-a7bff9143031" (UID: "1ec5bb43-1784-459b-87b0-a7bff9143031"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:55:13.644489 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.644311 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-token" pod "1ec5bb43-1784-459b-87b0-a7bff9143031" (UID: "1ec5bb43-1784-459b-87b0-a7bff9143031"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:13.644489 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.644304 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-envoy" pod "1ec5bb43-1784-459b-87b0-a7bff9143031" (UID: "1ec5bb43-1784-459b-87b0-a7bff9143031"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:13.654243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.647268 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9017622-5b22-40d9-9eb3-052cc0dbb14f-kube-api-access-whxgk" pod "d9017622-5b22-40d9-9eb3-052cc0dbb14f" (UID: "d9017622-5b22-40d9-9eb3-052cc0dbb14f"). InnerVolumeSpecName "kube-api-access-whxgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:13.737215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.736808 5091 patch_prober.go:29] interesting pod/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:55:13.737215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.736869 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" podUID="1ec5bb43-1784-459b-87b0-a7bff9143031" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:13.872260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.850750 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:13.872260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.852328 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:55:13.872260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.871259 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" podUID="0025d83b-0a0d-455d-9634-76492646d579" containerName="main" probeResult="failure" output="Get \"https://10.132.0.163:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:13.888872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.872863 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:55:13.931851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.931779 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 19:55:13.932251 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.932192 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" podUID="34968af2-47bb-4c0a-ba8b-974ae9b70f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 19:55:13.967203 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.947088 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.967203 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.947131 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.988207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.986269 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whxgk\" (UniqueName: \"kubernetes.io/projected/d9017622-5b22-40d9-9eb3-052cc0dbb14f-kube-api-access-whxgk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:13.988207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:13.986306 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ec5bb43-1784-459b-87b0-a7bff9143031-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:14.036905 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:14.036862 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 19:55:14.037429 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:14.037392 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" podUID="f28281f9-b9a0-4522-9336-ea91aa6a4ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 19:55:14.277576 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:14.277539 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:14.585535 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:14.585450 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 19:55:14.585780 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:14.585750 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" podUID="8d15b25d-07cf-4e6c-8446-60915a92b92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 19:55:14.587895 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:14.587873 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv" Oct 30 19:55:14.601187 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:14.588226 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" Oct 30 19:55:14.638228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:14.638198 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:55:14.668253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:14.668215 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk" event={"ID":"1ec5bb43-1784-459b-87b0-a7bff9143031","Type":"ContainerDied","Data":"929120d4170d63379b74b8afc6d72801c5c763518bffd93ee0009f40c7e24d4d"} Oct 30 19:55:14.668509 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:14.668491 5091 scope.go:122] "RemoveContainer" containerID="7f151bda3ad0db35995ad38cacb1c3216ee1c29c9cb43a7c7f22aa4b612dd27e" Oct 30 19:55:14.882594 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:14.874597 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" podUID="0025d83b-0a0d-455d-9634-76492646d579" containerName="main" probeResult="failure" output="Get \"https://10.132.0.163:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:14.924369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:14.924334 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 19:55:14.924691 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:14.924642 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" podUID="34968af2-47bb-4c0a-ba8b-974ae9b70f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 19:55:15.011410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:15.011133 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 19:55:15.011650 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:15.011618 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" podUID="f28281f9-b9a0-4522-9336-ea91aa6a4ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 19:55:15.130239 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:15.130203 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk"] Oct 30 19:55:15.136840 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:15.136751 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-426bab-openshift-default-55bc58fb6f-7kkfk"] Oct 30 19:55:15.140346 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:15.140321 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:55:15.572545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:15.546822 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 19:55:15.576631 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:15.575222 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:15.828791 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:15.792307 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:55:15.982029 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:15.968751 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 19:55:15.982029 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:15.968812 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" podUID="8d15b25d-07cf-4e6c-8446-60915a92b92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 19:55:15.998765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:15.995444 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 19:55:15.998765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:15.995508 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" podUID="34968af2-47bb-4c0a-ba8b-974ae9b70f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 19:55:16.010236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.009179 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 19:55:16.010236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.009235 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" podUID="f28281f9-b9a0-4522-9336-ea91aa6a4ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 19:55:16.186493 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:55:16.186359 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:55:16.336212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.325773 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:16.532072 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.531590 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec5bb43-1784-459b-87b0-a7bff9143031" path="/var/lib/kubelet/pods/1ec5bb43-1784-459b-87b0-a7bff9143031/volumes" Oct 30 19:55:16.532589 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.532558 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:55:16.586131 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.585374 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" start-of-body= Oct 30 19:55:16.586131 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.585642 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" podUID="8d15b25d-07cf-4e6c-8446-60915a92b92e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.177:15021/healthz/ready\": dial tcp 10.132.0.177:15021: connect: connection refused" Oct 30 19:55:16.639971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.639938 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-envoy\") pod \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " Oct 30 19:55:16.640179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.639978 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-token\" (UniqueName: \"kubernetes.io/projected/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-token\") pod \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " Oct 30 19:55:16.640179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.640034 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-kube-api-access-rs29v\" (UniqueName: \"kubernetes.io/projected/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-kube-api-access-rs29v\") pod \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " Oct 30 19:55:16.640179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.640074 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istiod-ca-cert\") pod \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " Oct 30 19:55:16.640179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.640125 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-credential-socket\") pod \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " Oct 30 19:55:16.640407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.640193 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-data\") pod \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " Oct 30 19:55:16.640407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.640259 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-podinfo\") pod \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " Oct 30 19:55:16.640407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.640336 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-workload-socket\") pod \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " Oct 30 19:55:16.640574 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.640422 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-workload-certs\") pod \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\" (UID: \"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a\") " Oct 30 19:55:16.641188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.641097 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-workload-socket" pod "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" (UID: "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:16.641538 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.641376 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-workload-certs" pod "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" (UID: "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:16.641538 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.641506 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istiod-ca-cert" pod "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" (UID: "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:16.641773 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.641625 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-credential-socket" pod "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" (UID: "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:16.642043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.642011 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-data" pod "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" (UID: "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:16.644452 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.644425 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-podinfo" pod "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" (UID: "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:55:16.644452 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.644437 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-envoy" pod "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" (UID: "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:16.644882 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.644850 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-kube-api-access-rs29v" pod "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" (UID: "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a"). InnerVolumeSpecName "kube-api-access-rs29v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:16.645701 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.645668 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-token" pod "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" (UID: "27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:16.742262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.742174 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:16.742262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.742211 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:16.742262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.742226 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs29v\" (UniqueName: \"kubernetes.io/projected/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-kube-api-access-rs29v\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:16.742262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.742236 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:16.742262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.742245 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:16.742262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.742254 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:16.742262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.742262 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:16.742598 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.742270 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:16.742598 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.742279 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:16.909479 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.909445 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" start-of-body= Oct 30 19:55:16.909732 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.909704 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" podUID="34968af2-47bb-4c0a-ba8b-974ae9b70f30" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.193:15021/healthz/ready\": dial tcp 10.132.0.193:15021: connect: connection refused" Oct 30 19:55:16.939040 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.939003 5091 generic.go:356] "Generic (PLEG): container finished" podID="27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" containerID="4e4fb733f02cefdb7a1371f4e73082082dced208f7e00bbdd37b1f4a07f108d5" exitCode=0 Oct 30 19:55:16.939234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.939058 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" Oct 30 19:55:16.939234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.939081 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" event={"ID":"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a","Type":"ContainerDied","Data":"4e4fb733f02cefdb7a1371f4e73082082dced208f7e00bbdd37b1f4a07f108d5"} Oct 30 19:55:16.939234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.939140 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs" event={"ID":"27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a","Type":"ContainerDied","Data":"8da519ea956fcfe1fe910da1dd4104a9f6c1b5130449f0a9f9780037ca8dea9d"} Oct 30 19:55:16.939234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.939204 5091 scope.go:122] "RemoveContainer" containerID="4e4fb733f02cefdb7a1371f4e73082082dced208f7e00bbdd37b1f4a07f108d5" Oct 30 19:55:16.988507 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.988456 5091 scope.go:122] "RemoveContainer" containerID="4e4fb733f02cefdb7a1371f4e73082082dced208f7e00bbdd37b1f4a07f108d5" Oct 30 19:55:16.990311 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.990286 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" start-of-body= Oct 30 19:55:16.990464 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.990439 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" podUID="f28281f9-b9a0-4522-9336-ea91aa6a4ce0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.194:15021/healthz/ready\": dial tcp 10.132.0.194:15021: connect: connection refused" Oct 30 19:55:16.990732 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:55:16.990297 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4fb733f02cefdb7a1371f4e73082082dced208f7e00bbdd37b1f4a07f108d5\": container with ID starting with 4e4fb733f02cefdb7a1371f4e73082082dced208f7e00bbdd37b1f4a07f108d5 not found: ID does not exist" containerID="4e4fb733f02cefdb7a1371f4e73082082dced208f7e00bbdd37b1f4a07f108d5" Oct 30 19:55:16.990816 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:16.990748 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4fb733f02cefdb7a1371f4e73082082dced208f7e00bbdd37b1f4a07f108d5"} err="failed to get container status \"4e4fb733f02cefdb7a1371f4e73082082dced208f7e00bbdd37b1f4a07f108d5\": rpc error: code = NotFound desc = could not find container \"4e4fb733f02cefdb7a1371f4e73082082dced208f7e00bbdd37b1f4a07f108d5\": container with ID starting with 4e4fb733f02cefdb7a1371f4e73082082dced208f7e00bbdd37b1f4a07f108d5 not found: ID does not exist" Oct 30 19:55:17.010975 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:17.010945 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs"] Oct 30 19:55:17.029354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:17.029290 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-8rtvs"] Oct 30 19:55:17.216805 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:17.216770 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:17.688903 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:17.688870 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:55:17.693273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:17.693234 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:55:18.033275 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:18.033237 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:18.034540 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:18.034514 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:18.035240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:18.035219 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 19:55:18.137768 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:18.134453 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:18.137768 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:18.135728 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:18.137768 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:18.136622 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 19:55:18.417862 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:18.417769 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:18.481005 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:18.479774 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" path="/var/lib/kubelet/pods/27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a/volumes" Oct 30 19:55:18.811761 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:18.811725 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:19.181448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.181348 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc"] Oct 30 19:55:19.181639 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.181613 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerName="istio-proxy" containerID="cri-o://057048b586b7d00f31eb1fab4ac107bd7c936d13502ad3955fea4424fa94805e" gracePeriod=30 Oct 30 19:55:19.488215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.486245 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc"] Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510388 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3c150c5f-1392-4a50-964d-a5e1593c2a19" containerName="payload-pre-processing" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510421 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c150c5f-1392-4a50-964d-a5e1593c2a19" containerName="payload-pre-processing" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510456 5091 cpu_manager.go:414] "containerMap: removing container" podUID="329cbe0d-56ce-4fb5-a1b7-470f6607b375" containerName="revoke-keys" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510465 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="329cbe0d-56ce-4fb5-a1b7-470f6607b375" containerName="revoke-keys" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510477 5091 cpu_manager.go:414] "containerMap: removing container" podUID="27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" containerName="istio-proxy" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510485 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" containerName="istio-proxy" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510496 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1ec5bb43-1784-459b-87b0-a7bff9143031" containerName="istio-proxy" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510504 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec5bb43-1784-459b-87b0-a7bff9143031" containerName="istio-proxy" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510537 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1ec5bb43-1784-459b-87b0-a7bff9143031" containerName="istio-proxy" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510544 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec5bb43-1784-459b-87b0-a7bff9143031" containerName="istio-proxy" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510571 5091 cpu_manager.go:414] "containerMap: removing container" podUID="afbd3cf9-21cd-4b40-94d2-df49a1146de6" containerName="payload-processing" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510578 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afbd3cf9-21cd-4b40-94d2-df49a1146de6" containerName="payload-processing" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510592 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d9017622-5b22-40d9-9eb3-052cc0dbb14f" containerName="revoke-keys" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510599 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9017622-5b22-40d9-9eb3-052cc0dbb14f" containerName="revoke-keys" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510800 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec5bb43-1784-459b-87b0-a7bff9143031" containerName="istio-proxy" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510821 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="afbd3cf9-21cd-4b40-94d2-df49a1146de6" containerName="payload-processing" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510832 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9017622-5b22-40d9-9eb3-052cc0dbb14f" containerName="revoke-keys" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510847 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c150c5f-1392-4a50-964d-a5e1593c2a19" containerName="payload-pre-processing" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510861 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="329cbe0d-56ce-4fb5-a1b7-470f6607b375" containerName="revoke-keys" Oct 30 19:55:19.521413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.510870 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ffb4f4-ae0f-445d-8ee6-c4276dcb9b2a" containerName="istio-proxy" Oct 30 19:55:19.537530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.536134 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc"] Oct 30 19:55:19.537530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.536290 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" Oct 30 19:55:19.551306 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.551274 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:55:19.551573 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.551556 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:55:19.627229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.627190 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a678b-bb04-4755-b0c1-7284663f0455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-401c9b63-44qzc\" (UID: \"aa6a678b-bb04-4755-b0c1-7284663f0455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" Oct 30 19:55:19.627401 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.627383 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6a678b-bb04-4755-b0c1-7284663f0455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-401c9b63-44qzc\" (UID: \"aa6a678b-bb04-4755-b0c1-7284663f0455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" Oct 30 19:55:19.627510 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.627489 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdjjd\" (UniqueName: \"kubernetes.io/projected/aa6a678b-bb04-4755-b0c1-7284663f0455-kube-api-access-kdjjd\") pod \"maas-api-revoke-sub-401c9b63-44qzc\" (UID: \"aa6a678b-bb04-4755-b0c1-7284663f0455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" Oct 30 19:55:19.757164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.749327 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6a678b-bb04-4755-b0c1-7284663f0455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-401c9b63-44qzc\" (UID: \"aa6a678b-bb04-4755-b0c1-7284663f0455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" Oct 30 19:55:19.757164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.749434 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdjjd\" (UniqueName: \"kubernetes.io/projected/aa6a678b-bb04-4755-b0c1-7284663f0455-kube-api-access-kdjjd\") pod \"maas-api-revoke-sub-401c9b63-44qzc\" (UID: \"aa6a678b-bb04-4755-b0c1-7284663f0455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" Oct 30 19:55:19.757164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.749598 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a678b-bb04-4755-b0c1-7284663f0455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-401c9b63-44qzc\" (UID: \"aa6a678b-bb04-4755-b0c1-7284663f0455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" Oct 30 19:55:19.757164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.750569 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a678b-bb04-4755-b0c1-7284663f0455-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-401c9b63-44qzc\" (UID: \"aa6a678b-bb04-4755-b0c1-7284663f0455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" Oct 30 19:55:19.760268 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.760239 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6a678b-bb04-4755-b0c1-7284663f0455-maas-api-xks-ca\") pod \"maas-api-revoke-sub-401c9b63-44qzc\" (UID: \"aa6a678b-bb04-4755-b0c1-7284663f0455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" Oct 30 19:55:19.771167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.771107 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdjjd\" (UniqueName: \"kubernetes.io/projected/aa6a678b-bb04-4755-b0c1-7284663f0455-kube-api-access-kdjjd\") pod \"maas-api-revoke-sub-401c9b63-44qzc\" (UID: \"aa6a678b-bb04-4755-b0c1-7284663f0455\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" Oct 30 19:55:19.789526 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.789490 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:19.871289 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:19.871256 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" Oct 30 19:55:20.170253 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:55:20.170213 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6a678b_bb04_4755_b0c1_7284663f0455.slice/crio-1409a3204be3e22cf402c637c35eb537adaff42bd96f8a99930baec5a240f207 WatchSource:0}: Error finding container 1409a3204be3e22cf402c637c35eb537adaff42bd96f8a99930baec5a240f207: Status 404 returned error can't find the container with id 1409a3204be3e22cf402c637c35eb537adaff42bd96f8a99930baec5a240f207 Oct 30 19:55:20.172202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:20.172138 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc"] Oct 30 19:55:20.806588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:20.806495 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:20.994238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:20.994204 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" event={"ID":"aa6a678b-bb04-4755-b0c1-7284663f0455","Type":"ContainerStarted","Data":"f3e5b8531ea2df0b8f22f148d17cc1027d3d1e7802bed3ea0058cceb99caaf03"} Oct 30 19:55:20.994656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:20.994262 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" event={"ID":"aa6a678b-bb04-4755-b0c1-7284663f0455","Type":"ContainerStarted","Data":"1409a3204be3e22cf402c637c35eb537adaff42bd96f8a99930baec5a240f207"} Oct 30 19:55:21.027677 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:21.027625 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" podStartSLOduration=2.027611569 podStartE2EDuration="2.027611569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:21.024266501 +0000 UTC m=+1440.719474460" watchObservedRunningTime="2026-10-30 19:55:21.027611569 +0000 UTC m=+1440.722819530" Oct 30 19:55:21.747700 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:21.747664 5091 scope.go:122] "RemoveContainer" containerID="c7eadfefefdd941544d14c55acbade5ebf8185893f8a6531d49a66a2103f7dde" Oct 30 19:55:21.758823 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:21.758799 5091 scope.go:122] "RemoveContainer" containerID="535c3b687d4c08c0b3e284ddec83c2ad358ded2ade7903c142b152af1701fe3c" Oct 30 19:55:21.790384 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:21.790229 5091 scope.go:122] "RemoveContainer" containerID="707a911c1a24031c9b4b293dd84b75485a24c71a81879e125eee459b1394dc12" Oct 30 19:55:21.849193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:21.837412 5091 scope.go:122] "RemoveContainer" containerID="c02f45b949dca6064862cb795f8ee59bf5800fe7526e2009f9014af0338a43df" Oct 30 19:55:21.865287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:21.865263 5091 scope.go:122] "RemoveContainer" containerID="691589cf34eae4d3bef60a1e983d8bb8d0aa2a9021e1e6b87d5bc485318cf98a" Oct 30 19:55:21.888337 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:21.888300 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:55:21.888471 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:21.888376 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:55:22.003815 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:22.003734 5091 generic.go:356] "Generic (PLEG): container finished" podID="aa6a678b-bb04-4755-b0c1-7284663f0455" containerID="f3e5b8531ea2df0b8f22f148d17cc1027d3d1e7802bed3ea0058cceb99caaf03" exitCode=0 Oct 30 19:55:22.004196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:22.003817 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" event={"ID":"aa6a678b-bb04-4755-b0c1-7284663f0455","Type":"ContainerDied","Data":"f3e5b8531ea2df0b8f22f148d17cc1027d3d1e7802bed3ea0058cceb99caaf03"} Oct 30 19:55:22.364010 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:22.363923 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:22.699946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:22.698830 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:23.191849 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:23.191825 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" Oct 30 19:55:23.212063 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:23.212034 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa6a678b-bb04-4755-b0c1-7284663f0455-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a678b-bb04-4755-b0c1-7284663f0455-maas-api-openshift-ca\") pod \"aa6a678b-bb04-4755-b0c1-7284663f0455\" (UID: \"aa6a678b-bb04-4755-b0c1-7284663f0455\") " Oct 30 19:55:23.212211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:23.212084 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6a678b-bb04-4755-b0c1-7284663f0455-kube-api-access-kdjjd\" (UniqueName: \"kubernetes.io/projected/aa6a678b-bb04-4755-b0c1-7284663f0455-kube-api-access-kdjjd\") pod \"aa6a678b-bb04-4755-b0c1-7284663f0455\" (UID: \"aa6a678b-bb04-4755-b0c1-7284663f0455\") " Oct 30 19:55:23.212359 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:23.212337 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa6a678b-bb04-4755-b0c1-7284663f0455-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6a678b-bb04-4755-b0c1-7284663f0455-maas-api-xks-ca\") pod \"aa6a678b-bb04-4755-b0c1-7284663f0455\" (UID: \"aa6a678b-bb04-4755-b0c1-7284663f0455\") " Oct 30 19:55:23.212476 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:23.212457 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6a678b-bb04-4755-b0c1-7284663f0455-maas-api-openshift-ca" pod "aa6a678b-bb04-4755-b0c1-7284663f0455" (UID: "aa6a678b-bb04-4755-b0c1-7284663f0455"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:23.212908 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:23.212884 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa6a678b-bb04-4755-b0c1-7284663f0455-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:23.214470 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:23.214446 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa6a678b-bb04-4755-b0c1-7284663f0455-maas-api-xks-ca" pod "aa6a678b-bb04-4755-b0c1-7284663f0455" (UID: "aa6a678b-bb04-4755-b0c1-7284663f0455"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:23.214542 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:23.214513 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6a678b-bb04-4755-b0c1-7284663f0455-kube-api-access-kdjjd" pod "aa6a678b-bb04-4755-b0c1-7284663f0455" (UID: "aa6a678b-bb04-4755-b0c1-7284663f0455"). InnerVolumeSpecName "kube-api-access-kdjjd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:23.314095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:23.314012 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa6a678b-bb04-4755-b0c1-7284663f0455-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:23.314095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:23.314046 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdjjd\" (UniqueName: \"kubernetes.io/projected/aa6a678b-bb04-4755-b0c1-7284663f0455-kube-api-access-kdjjd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:23.686158 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:23.676284 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:23.835994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:23.835958 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:24.021139 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.021111 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" Oct 30 19:55:24.021319 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.021160 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc" event={"ID":"aa6a678b-bb04-4755-b0c1-7284663f0455","Type":"ContainerDied","Data":"1409a3204be3e22cf402c637c35eb537adaff42bd96f8a99930baec5a240f207"} Oct 30 19:55:24.021319 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.021215 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1409a3204be3e22cf402c637c35eb537adaff42bd96f8a99930baec5a240f207" Oct 30 19:55:24.144345 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.144300 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" podUID="0025d83b-0a0d-455d-9634-76492646d579" containerName="main" containerID="cri-o://a53abbc1383d9480daecf2c2ec0c95e5f24df3f27b103ff6b78e94cb91cc7375" gracePeriod=45 Oct 30 19:55:24.642758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.642664 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:24.705769 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.705733 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:55:24.708502 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.708478 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:55:24.775917 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.775872 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-dshm\") pod \"0025d83b-0a0d-455d-9634-76492646d579\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " Oct 30 19:55:24.776090 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.775932 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-model-cache\") pod \"0025d83b-0a0d-455d-9634-76492646d579\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " Oct 30 19:55:24.776090 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.776002 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0025d83b-0a0d-455d-9634-76492646d579-tls-certs\" (UniqueName: \"kubernetes.io/secret/0025d83b-0a0d-455d-9634-76492646d579-tls-certs\") pod \"0025d83b-0a0d-455d-9634-76492646d579\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " Oct 30 19:55:24.776262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.776110 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-home\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-home\") pod \"0025d83b-0a0d-455d-9634-76492646d579\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " Oct 30 19:55:24.776262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.776187 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-envoy\") pod \"253ed971-2d90-4358-86d1-314bdb0fad0f\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " Oct 30 19:55:24.776262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.776202 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-model-cache" pod "0025d83b-0a0d-455d-9634-76492646d579" (UID: "0025d83b-0a0d-455d-9634-76492646d579"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:24.776262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.776252 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-credential-socket\") pod \"253ed971-2d90-4358-86d1-314bdb0fad0f\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " Oct 30 19:55:24.776480 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.776310 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-token\" (UniqueName: \"kubernetes.io/projected/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-token\") pod \"253ed971-2d90-4358-86d1-314bdb0fad0f\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " Oct 30 19:55:24.776480 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.776356 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/253ed971-2d90-4358-86d1-314bdb0fad0f-kube-api-access-xx8ww\" (UniqueName: \"kubernetes.io/projected/253ed971-2d90-4358-86d1-314bdb0fad0f-kube-api-access-xx8ww\") pod \"253ed971-2d90-4358-86d1-314bdb0fad0f\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " Oct 30 19:55:24.776480 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.776401 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0025d83b-0a0d-455d-9634-76492646d579-kube-api-access-gdcct\" (UniqueName: \"kubernetes.io/projected/0025d83b-0a0d-455d-9634-76492646d579-kube-api-access-gdcct\") pod \"0025d83b-0a0d-455d-9634-76492646d579\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " Oct 30 19:55:24.776480 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.776444 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-tmp-dir\") pod \"0025d83b-0a0d-455d-9634-76492646d579\" (UID: \"0025d83b-0a0d-455d-9634-76492646d579\") " Oct 30 19:55:24.776480 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.776451 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-home" pod "0025d83b-0a0d-455d-9634-76492646d579" (UID: "0025d83b-0a0d-455d-9634-76492646d579"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:24.776759 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.776488 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-podinfo\") pod \"253ed971-2d90-4358-86d1-314bdb0fad0f\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " Oct 30 19:55:24.776759 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.776533 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-data\") pod \"253ed971-2d90-4358-86d1-314bdb0fad0f\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " Oct 30 19:55:24.777060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.777028 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-credential-socket" pod "253ed971-2d90-4358-86d1-314bdb0fad0f" (UID: "253ed971-2d90-4358-86d1-314bdb0fad0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:24.777235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.777209 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-tmp-dir" pod "0025d83b-0a0d-455d-9634-76492646d579" (UID: "0025d83b-0a0d-455d-9634-76492646d579"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:24.777336 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.777315 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:24.777415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.777363 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-model-cache\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:24.777415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.777379 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-home\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:24.777639 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.777615 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-data" pod "253ed971-2d90-4358-86d1-314bdb0fad0f" (UID: "253ed971-2d90-4358-86d1-314bdb0fad0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:24.788276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.788248 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0025d83b-0a0d-455d-9634-76492646d579-tls-certs" pod "0025d83b-0a0d-455d-9634-76492646d579" (UID: "0025d83b-0a0d-455d-9634-76492646d579"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:24.788420 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.788399 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-dshm" pod "0025d83b-0a0d-455d-9634-76492646d579" (UID: "0025d83b-0a0d-455d-9634-76492646d579"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:24.791456 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.791320 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-podinfo" pod "253ed971-2d90-4358-86d1-314bdb0fad0f" (UID: "253ed971-2d90-4358-86d1-314bdb0fad0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:55:24.791456 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.791339 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-token" pod "253ed971-2d90-4358-86d1-314bdb0fad0f" (UID: "253ed971-2d90-4358-86d1-314bdb0fad0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:24.791456 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.791375 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0025d83b-0a0d-455d-9634-76492646d579-kube-api-access-gdcct" pod "0025d83b-0a0d-455d-9634-76492646d579" (UID: "0025d83b-0a0d-455d-9634-76492646d579"). InnerVolumeSpecName "kube-api-access-gdcct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:24.791456 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.791375 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/253ed971-2d90-4358-86d1-314bdb0fad0f-kube-api-access-xx8ww" pod "253ed971-2d90-4358-86d1-314bdb0fad0f" (UID: "253ed971-2d90-4358-86d1-314bdb0fad0f"). InnerVolumeSpecName "kube-api-access-xx8ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:24.791708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.791552 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-envoy" pod "253ed971-2d90-4358-86d1-314bdb0fad0f" (UID: "253ed971-2d90-4358-86d1-314bdb0fad0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:24.877817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.877782 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-workload-certs\") pod \"253ed971-2d90-4358-86d1-314bdb0fad0f\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " Oct 30 19:55:24.878062 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878042 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-workload-certs" pod "253ed971-2d90-4358-86d1-314bdb0fad0f" (UID: "253ed971-2d90-4358-86d1-314bdb0fad0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:24.878122 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878090 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/253ed971-2d90-4358-86d1-314bdb0fad0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253ed971-2d90-4358-86d1-314bdb0fad0f-istiod-ca-cert\") pod \"253ed971-2d90-4358-86d1-314bdb0fad0f\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " Oct 30 19:55:24.878209 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878122 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-workload-socket\") pod \"253ed971-2d90-4358-86d1-314bdb0fad0f\" (UID: \"253ed971-2d90-4358-86d1-314bdb0fad0f\") " Oct 30 19:55:24.878466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878376 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/253ed971-2d90-4358-86d1-314bdb0fad0f-istiod-ca-cert" pod "253ed971-2d90-4358-86d1-314bdb0fad0f" (UID: "253ed971-2d90-4358-86d1-314bdb0fad0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:24.878584 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878536 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-workload-socket" pod "253ed971-2d90-4358-86d1-314bdb0fad0f" (UID: "253ed971-2d90-4358-86d1-314bdb0fad0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:24.878846 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878821 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:24.879036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878850 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-dshm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:24.879036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878864 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0025d83b-0a0d-455d-9634-76492646d579-tls-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:24.879036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878873 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:24.879036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878881 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:24.879036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878890 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx8ww\" (UniqueName: \"kubernetes.io/projected/253ed971-2d90-4358-86d1-314bdb0fad0f-kube-api-access-xx8ww\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:24.879036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878900 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gdcct\" (UniqueName: \"kubernetes.io/projected/0025d83b-0a0d-455d-9634-76492646d579-kube-api-access-gdcct\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:24.879036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878911 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0025d83b-0a0d-455d-9634-76492646d579-tmp-dir\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:24.879036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878924 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:24.879036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878936 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:24.879036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878949 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/253ed971-2d90-4358-86d1-314bdb0fad0f-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:24.879036 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.878962 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/253ed971-2d90-4358-86d1-314bdb0fad0f-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:24.919496 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:24.919414 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:25.277171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.272850 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:25.302171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.291660 5091 generic.go:356] "Generic (PLEG): container finished" podID="0025d83b-0a0d-455d-9634-76492646d579" containerID="a53abbc1383d9480daecf2c2ec0c95e5f24df3f27b103ff6b78e94cb91cc7375" exitCode=0 Oct 30 19:55:25.302171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.291920 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" event={"ID":"0025d83b-0a0d-455d-9634-76492646d579","Type":"ContainerDied","Data":"a53abbc1383d9480daecf2c2ec0c95e5f24df3f27b103ff6b78e94cb91cc7375"} Oct 30 19:55:25.302171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.291966 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" event={"ID":"0025d83b-0a0d-455d-9634-76492646d579","Type":"ContainerDied","Data":"84f6c9f8fd171de81d2047ce4c9411dd9d8ec4d9cddc3b572a0d527ad4be353a"} Oct 30 19:55:25.302171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.291990 5091 scope.go:122] "RemoveContainer" containerID="a53abbc1383d9480daecf2c2ec0c95e5f24df3f27b103ff6b78e94cb91cc7375" Oct 30 19:55:25.302171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.292201 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn" Oct 30 19:55:25.338740 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.336989 5091 generic.go:356] "Generic (PLEG): container finished" podID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerID="057048b586b7d00f31eb1fab4ac107bd7c936d13502ad3955fea4424fa94805e" exitCode=0 Oct 30 19:55:25.338740 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.337125 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" event={"ID":"253ed971-2d90-4358-86d1-314bdb0fad0f","Type":"ContainerDied","Data":"057048b586b7d00f31eb1fab4ac107bd7c936d13502ad3955fea4424fa94805e"} Oct 30 19:55:25.338740 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.337242 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" event={"ID":"253ed971-2d90-4358-86d1-314bdb0fad0f","Type":"ContainerDied","Data":"4aedf90faebe628350b2a9d89adaf009d45845b788e8824ee2ac24b7b1b71a79"} Oct 30 19:55:25.338740 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.337348 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc" Oct 30 19:55:25.413887 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.412873 5091 scope.go:122] "RemoveContainer" containerID="a53abbc1383d9480daecf2c2ec0c95e5f24df3f27b103ff6b78e94cb91cc7375" Oct 30 19:55:25.414545 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:55:25.414162 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a53abbc1383d9480daecf2c2ec0c95e5f24df3f27b103ff6b78e94cb91cc7375\": container with ID starting with a53abbc1383d9480daecf2c2ec0c95e5f24df3f27b103ff6b78e94cb91cc7375 not found: ID does not exist" containerID="a53abbc1383d9480daecf2c2ec0c95e5f24df3f27b103ff6b78e94cb91cc7375" Oct 30 19:55:25.414545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.414208 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a53abbc1383d9480daecf2c2ec0c95e5f24df3f27b103ff6b78e94cb91cc7375"} err="failed to get container status \"a53abbc1383d9480daecf2c2ec0c95e5f24df3f27b103ff6b78e94cb91cc7375\": rpc error: code = NotFound desc = could not find container \"a53abbc1383d9480daecf2c2ec0c95e5f24df3f27b103ff6b78e94cb91cc7375\": container with ID starting with a53abbc1383d9480daecf2c2ec0c95e5f24df3f27b103ff6b78e94cb91cc7375 not found: ID does not exist" Oct 30 19:55:25.414545 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.414236 5091 scope.go:122] "RemoveContainer" containerID="057048b586b7d00f31eb1fab4ac107bd7c936d13502ad3955fea4424fa94805e" Oct 30 19:55:25.416966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.416942 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg"] Oct 30 19:55:25.433043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.432672 5091 scope.go:122] "RemoveContainer" containerID="355d7d7b08a7b8dca0bcfa3c481dc37e62a6271f7f73a60fe12e415f7db89f9e" Oct 30 19:55:25.440681 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.440649 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-55a8e6-mvvwg"] Oct 30 19:55:25.462254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.461906 5091 scope.go:122] "RemoveContainer" containerID="057048b586b7d00f31eb1fab4ac107bd7c936d13502ad3955fea4424fa94805e" Oct 30 19:55:25.462794 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:55:25.462499 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"057048b586b7d00f31eb1fab4ac107bd7c936d13502ad3955fea4424fa94805e\": container with ID starting with 057048b586b7d00f31eb1fab4ac107bd7c936d13502ad3955fea4424fa94805e not found: ID does not exist" containerID="057048b586b7d00f31eb1fab4ac107bd7c936d13502ad3955fea4424fa94805e" Oct 30 19:55:25.462794 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.462541 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"057048b586b7d00f31eb1fab4ac107bd7c936d13502ad3955fea4424fa94805e"} err="failed to get container status \"057048b586b7d00f31eb1fab4ac107bd7c936d13502ad3955fea4424fa94805e\": rpc error: code = NotFound desc = could not find container \"057048b586b7d00f31eb1fab4ac107bd7c936d13502ad3955fea4424fa94805e\": container with ID starting with 057048b586b7d00f31eb1fab4ac107bd7c936d13502ad3955fea4424fa94805e not found: ID does not exist" Oct 30 19:55:25.462794 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.462567 5091 scope.go:122] "RemoveContainer" containerID="355d7d7b08a7b8dca0bcfa3c481dc37e62a6271f7f73a60fe12e415f7db89f9e" Oct 30 19:55:25.463234 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:55:25.463136 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355d7d7b08a7b8dca0bcfa3c481dc37e62a6271f7f73a60fe12e415f7db89f9e\": container with ID starting with 355d7d7b08a7b8dca0bcfa3c481dc37e62a6271f7f73a60fe12e415f7db89f9e not found: ID does not exist" containerID="355d7d7b08a7b8dca0bcfa3c481dc37e62a6271f7f73a60fe12e415f7db89f9e" Oct 30 19:55:25.463234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.463182 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355d7d7b08a7b8dca0bcfa3c481dc37e62a6271f7f73a60fe12e415f7db89f9e"} err="failed to get container status \"355d7d7b08a7b8dca0bcfa3c481dc37e62a6271f7f73a60fe12e415f7db89f9e\": rpc error: code = NotFound desc = could not find container \"355d7d7b08a7b8dca0bcfa3c481dc37e62a6271f7f73a60fe12e415f7db89f9e\": container with ID starting with 355d7d7b08a7b8dca0bcfa3c481dc37e62a6271f7f73a60fe12e415f7db89f9e not found: ID does not exist" Oct 30 19:55:25.511022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.510987 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb"] Oct 30 19:55:25.512260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.512234 5091 cpu_manager.go:414] "containerMap: removing container" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerName="istio-proxy" Oct 30 19:55:25.512457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.512442 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerName="istio-proxy" Oct 30 19:55:25.512588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.512575 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0025d83b-0a0d-455d-9634-76492646d579" containerName="main" Oct 30 19:55:25.512679 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.512668 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0025d83b-0a0d-455d-9634-76492646d579" containerName="main" Oct 30 19:55:25.512813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.512803 5091 cpu_manager.go:414] "containerMap: removing container" podUID="aa6a678b-bb04-4755-b0c1-7284663f0455" containerName="revoke-keys" Oct 30 19:55:25.512937 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.512924 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6a678b-bb04-4755-b0c1-7284663f0455" containerName="revoke-keys" Oct 30 19:55:25.513583 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.513551 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6a678b-bb04-4755-b0c1-7284663f0455" containerName="revoke-keys" Oct 30 19:55:25.517845 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.517821 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerName="istio-proxy" Oct 30 19:55:25.518007 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.517993 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec5bb43-1784-459b-87b0-a7bff9143031" containerName="istio-proxy" Oct 30 19:55:25.518106 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.518094 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0025d83b-0a0d-455d-9634-76492646d579" containerName="main" Oct 30 19:55:25.552207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.546011 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" Oct 30 19:55:25.554847 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.554785 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:55:25.555252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.555205 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:55:25.568249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.568197 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb"] Oct 30 19:55:25.647313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.647265 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc"] Oct 30 19:55:25.710456 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.660084 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3229e72-73a2-4251-a0d5-dc0632635b30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb\" (UID: \"e3229e72-73a2-4251-a0d5-dc0632635b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" Oct 30 19:55:25.710456 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.692367 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3229e72-73a2-4251-a0d5-dc0632635b30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb\" (UID: \"e3229e72-73a2-4251-a0d5-dc0632635b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" Oct 30 19:55:25.710456 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.692571 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmkbq\" (UniqueName: \"kubernetes.io/projected/e3229e72-73a2-4251-a0d5-dc0632635b30-kube-api-access-lmkbq\") pod \"maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb\" (UID: \"e3229e72-73a2-4251-a0d5-dc0632635b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" Oct 30 19:55:25.740379 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.740294 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-9lwvc"] Oct 30 19:55:25.828304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.822354 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmkbq\" (UniqueName: \"kubernetes.io/projected/e3229e72-73a2-4251-a0d5-dc0632635b30-kube-api-access-lmkbq\") pod \"maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb\" (UID: \"e3229e72-73a2-4251-a0d5-dc0632635b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" Oct 30 19:55:25.828304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.822502 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3229e72-73a2-4251-a0d5-dc0632635b30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb\" (UID: \"e3229e72-73a2-4251-a0d5-dc0632635b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" Oct 30 19:55:25.828304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.822583 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3229e72-73a2-4251-a0d5-dc0632635b30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb\" (UID: \"e3229e72-73a2-4251-a0d5-dc0632635b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" Oct 30 19:55:25.828304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.823479 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3229e72-73a2-4251-a0d5-dc0632635b30-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb\" (UID: \"e3229e72-73a2-4251-a0d5-dc0632635b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" Oct 30 19:55:25.857320 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.857287 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn"] Oct 30 19:55:25.862585 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.862041 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-b17e94/praxis-route-b17e94-kserve-6c5f5d757-4pszn"] Oct 30 19:55:25.901890 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.901830 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmkbq\" (UniqueName: \"kubernetes.io/projected/e3229e72-73a2-4251-a0d5-dc0632635b30-kube-api-access-lmkbq\") pod \"maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb\" (UID: \"e3229e72-73a2-4251-a0d5-dc0632635b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" Oct 30 19:55:25.946625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.946583 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:25.946983 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:25.946955 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3229e72-73a2-4251-a0d5-dc0632635b30-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb\" (UID: \"e3229e72-73a2-4251-a0d5-dc0632635b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" Oct 30 19:55:26.219404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:26.201569 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" Oct 30 19:55:26.677755 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:26.677712 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0025d83b-0a0d-455d-9634-76492646d579" path="/var/lib/kubelet/pods/0025d83b-0a0d-455d-9634-76492646d579/volumes" Oct 30 19:55:26.701170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:26.695334 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" path="/var/lib/kubelet/pods/253ed971-2d90-4358-86d1-314bdb0fad0f/volumes" Oct 30 19:55:26.702380 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:26.702348 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="329cbe0d-56ce-4fb5-a1b7-470f6607b375" path="/var/lib/kubelet/pods/329cbe0d-56ce-4fb5-a1b7-470f6607b375/volumes" Oct 30 19:55:27.301535 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:27.301488 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 19:55:27.411935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:27.411894 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:27.477362 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:27.477252 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 19:55:27.565264 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:27.565195 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:27.921212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:27.921169 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:28.018854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:28.018744 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb"] Oct 30 19:55:28.064182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:28.060798 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" event={"ID":"e3229e72-73a2-4251-a0d5-dc0632635b30","Type":"ContainerStarted","Data":"ac767934f9cdf01b784406f55d8c6b5697f5571680a4788c8d307885f71c9c63"} Oct 30 19:55:28.307293 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:28.307237 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:28.309516 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:28.309471 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:55:28.545228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:28.525337 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:28.545228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:28.527770 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:28.605181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:28.572793 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 19:55:28.605181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:28.588340 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:55:28.605181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:28.588398 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" podUID="1babb797-d60b-4d9c-8589-27a6877d4478" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:28.802639 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:28.781094 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:55:29.091585 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.091555 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:29.224634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.224603 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:55:29.277217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.277186 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:55:29.528633 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.528601 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz"] Oct 30 19:55:29.529336 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.529314 5091 cpu_manager.go:414] "containerMap: removing container" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerName="istio-proxy" Oct 30 19:55:29.529443 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.529335 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerName="istio-proxy" Oct 30 19:55:29.529553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.529539 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="253ed971-2d90-4358-86d1-314bdb0fad0f" containerName="istio-proxy" Oct 30 19:55:29.533314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.533294 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" Oct 30 19:55:29.597170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.594057 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz"] Oct 30 19:55:29.619363 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.619335 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be3d162b-06c8-4418-a135-0bd64038f695-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d799a878-hm5vz\" (UID: \"be3d162b-06c8-4418-a135-0bd64038f695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" Oct 30 19:55:29.619537 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.619486 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be3d162b-06c8-4418-a135-0bd64038f695-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d799a878-hm5vz\" (UID: \"be3d162b-06c8-4418-a135-0bd64038f695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" Oct 30 19:55:29.619599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.619586 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69zvq\" (UniqueName: \"kubernetes.io/projected/be3d162b-06c8-4418-a135-0bd64038f695-kube-api-access-69zvq\") pod \"maas-api-revoke-sub-d799a878-hm5vz\" (UID: \"be3d162b-06c8-4418-a135-0bd64038f695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" Oct 30 19:55:29.724679 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.724638 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69zvq\" (UniqueName: \"kubernetes.io/projected/be3d162b-06c8-4418-a135-0bd64038f695-kube-api-access-69zvq\") pod \"maas-api-revoke-sub-d799a878-hm5vz\" (UID: \"be3d162b-06c8-4418-a135-0bd64038f695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" Oct 30 19:55:29.728797 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.728455 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be3d162b-06c8-4418-a135-0bd64038f695-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d799a878-hm5vz\" (UID: \"be3d162b-06c8-4418-a135-0bd64038f695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" Oct 30 19:55:29.728797 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.728680 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be3d162b-06c8-4418-a135-0bd64038f695-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d799a878-hm5vz\" (UID: \"be3d162b-06c8-4418-a135-0bd64038f695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" Oct 30 19:55:29.734064 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.730009 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be3d162b-06c8-4418-a135-0bd64038f695-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d799a878-hm5vz\" (UID: \"be3d162b-06c8-4418-a135-0bd64038f695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" Oct 30 19:55:29.734064 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.733969 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be3d162b-06c8-4418-a135-0bd64038f695-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d799a878-hm5vz\" (UID: \"be3d162b-06c8-4418-a135-0bd64038f695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" Oct 30 19:55:29.782724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.782196 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69zvq\" (UniqueName: \"kubernetes.io/projected/be3d162b-06c8-4418-a135-0bd64038f695-kube-api-access-69zvq\") pod \"maas-api-revoke-sub-d799a878-hm5vz\" (UID: \"be3d162b-06c8-4418-a135-0bd64038f695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" Oct 30 19:55:29.784987 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.783521 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:29.893526 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:29.893495 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" Oct 30 19:55:30.130825 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:30.130741 5091 generic.go:356] "Generic (PLEG): container finished" podID="e3229e72-73a2-4251-a0d5-dc0632635b30" containerID="17f1494ef3449916735b5e3dc13b9c209bbc7741866d0d493357bb9bd39d4a9e" exitCode=0 Oct 30 19:55:30.131228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:30.130828 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" event={"ID":"e3229e72-73a2-4251-a0d5-dc0632635b30","Type":"ContainerDied","Data":"17f1494ef3449916735b5e3dc13b9c209bbc7741866d0d493357bb9bd39d4a9e"} Oct 30 19:55:30.224881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:30.224852 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz"] Oct 30 19:55:30.228009 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:55:30.227981 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3d162b_06c8_4418_a135_0bd64038f695.slice/crio-acabca71403e2c00ff3d818797de438815691b656fe4943c7f16b32a90600aea WatchSource:0}: Error finding container acabca71403e2c00ff3d818797de438815691b656fe4943c7f16b32a90600aea: Status 404 returned error can't find the container with id acabca71403e2c00ff3d818797de438815691b656fe4943c7f16b32a90600aea Oct 30 19:55:30.967583 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:30.967454 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:31.166172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:31.155860 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" event={"ID":"be3d162b-06c8-4418-a135-0bd64038f695","Type":"ContainerStarted","Data":"7dd0c666a99b8e2d36669e541fab5639b80a8710d1e3f81621ac988ad65ab123"} Oct 30 19:55:31.166172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:31.155960 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" event={"ID":"be3d162b-06c8-4418-a135-0bd64038f695","Type":"ContainerStarted","Data":"acabca71403e2c00ff3d818797de438815691b656fe4943c7f16b32a90600aea"} Oct 30 19:55:31.200106 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:31.200031 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" podStartSLOduration=2.200011346 podStartE2EDuration="2.200011346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:31.198747887 +0000 UTC m=+1450.893955845" watchObservedRunningTime="2026-10-30 19:55:31.200011346 +0000 UTC m=+1450.895219305" Oct 30 19:55:31.499321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:31.499295 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" Oct 30 19:55:31.596421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:31.595469 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3229e72-73a2-4251-a0d5-dc0632635b30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3229e72-73a2-4251-a0d5-dc0632635b30-maas-api-openshift-ca\") pod \"e3229e72-73a2-4251-a0d5-dc0632635b30\" (UID: \"e3229e72-73a2-4251-a0d5-dc0632635b30\") " Oct 30 19:55:31.596421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:31.595925 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3229e72-73a2-4251-a0d5-dc0632635b30-maas-api-openshift-ca" pod "e3229e72-73a2-4251-a0d5-dc0632635b30" (UID: "e3229e72-73a2-4251-a0d5-dc0632635b30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:31.596421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:31.596065 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3229e72-73a2-4251-a0d5-dc0632635b30-kube-api-access-lmkbq\" (UniqueName: \"kubernetes.io/projected/e3229e72-73a2-4251-a0d5-dc0632635b30-kube-api-access-lmkbq\") pod \"e3229e72-73a2-4251-a0d5-dc0632635b30\" (UID: \"e3229e72-73a2-4251-a0d5-dc0632635b30\") " Oct 30 19:55:31.596421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:31.596333 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e3229e72-73a2-4251-a0d5-dc0632635b30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3229e72-73a2-4251-a0d5-dc0632635b30-maas-api-xks-ca\") pod \"e3229e72-73a2-4251-a0d5-dc0632635b30\" (UID: \"e3229e72-73a2-4251-a0d5-dc0632635b30\") " Oct 30 19:55:31.597162 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:31.597126 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e3229e72-73a2-4251-a0d5-dc0632635b30-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:31.598575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:31.598544 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3229e72-73a2-4251-a0d5-dc0632635b30-kube-api-access-lmkbq" pod "e3229e72-73a2-4251-a0d5-dc0632635b30" (UID: "e3229e72-73a2-4251-a0d5-dc0632635b30"). InnerVolumeSpecName "kube-api-access-lmkbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:31.599801 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:31.599770 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3229e72-73a2-4251-a0d5-dc0632635b30-maas-api-xks-ca" pod "e3229e72-73a2-4251-a0d5-dc0632635b30" (UID: "e3229e72-73a2-4251-a0d5-dc0632635b30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:31.698320 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:31.698288 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e3229e72-73a2-4251-a0d5-dc0632635b30-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:31.698320 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:31.698316 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmkbq\" (UniqueName: \"kubernetes.io/projected/e3229e72-73a2-4251-a0d5-dc0632635b30-kube-api-access-lmkbq\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:32.220165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:32.220106 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:32.220769 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:32.220742 5091 generic.go:356] "Generic (PLEG): container finished" podID="be3d162b-06c8-4418-a135-0bd64038f695" containerID="7dd0c666a99b8e2d36669e541fab5639b80a8710d1e3f81621ac988ad65ab123" exitCode=0 Oct 30 19:55:32.220951 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:32.220929 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" event={"ID":"be3d162b-06c8-4418-a135-0bd64038f695","Type":"ContainerDied","Data":"7dd0c666a99b8e2d36669e541fab5639b80a8710d1e3f81621ac988ad65ab123"} Oct 30 19:55:32.223120 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:32.223091 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" Oct 30 19:55:32.223287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:32.223171 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb" event={"ID":"e3229e72-73a2-4251-a0d5-dc0632635b30","Type":"ContainerDied","Data":"ac767934f9cdf01b784406f55d8c6b5697f5571680a4788c8d307885f71c9c63"} Oct 30 19:55:32.223287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:32.223222 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac767934f9cdf01b784406f55d8c6b5697f5571680a4788c8d307885f71c9c63" Oct 30 19:55:33.516227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:33.509398 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" Oct 30 19:55:33.640860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:33.640831 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3d162b-06c8-4418-a135-0bd64038f695-kube-api-access-69zvq\" (UniqueName: \"kubernetes.io/projected/be3d162b-06c8-4418-a135-0bd64038f695-kube-api-access-69zvq\") pod \"be3d162b-06c8-4418-a135-0bd64038f695\" (UID: \"be3d162b-06c8-4418-a135-0bd64038f695\") " Oct 30 19:55:33.640986 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:33.640944 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be3d162b-06c8-4418-a135-0bd64038f695-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be3d162b-06c8-4418-a135-0bd64038f695-maas-api-openshift-ca\") pod \"be3d162b-06c8-4418-a135-0bd64038f695\" (UID: \"be3d162b-06c8-4418-a135-0bd64038f695\") " Oct 30 19:55:33.641039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:33.640986 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be3d162b-06c8-4418-a135-0bd64038f695-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be3d162b-06c8-4418-a135-0bd64038f695-maas-api-xks-ca\") pod \"be3d162b-06c8-4418-a135-0bd64038f695\" (UID: \"be3d162b-06c8-4418-a135-0bd64038f695\") " Oct 30 19:55:33.641384 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:33.641357 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3d162b-06c8-4418-a135-0bd64038f695-maas-api-openshift-ca" pod "be3d162b-06c8-4418-a135-0bd64038f695" (UID: "be3d162b-06c8-4418-a135-0bd64038f695"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:33.641648 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:33.641620 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be3d162b-06c8-4418-a135-0bd64038f695-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:33.643067 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:33.643041 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3d162b-06c8-4418-a135-0bd64038f695-kube-api-access-69zvq" pod "be3d162b-06c8-4418-a135-0bd64038f695" (UID: "be3d162b-06c8-4418-a135-0bd64038f695"). InnerVolumeSpecName "kube-api-access-69zvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:33.643282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:33.643254 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be3d162b-06c8-4418-a135-0bd64038f695-maas-api-xks-ca" pod "be3d162b-06c8-4418-a135-0bd64038f695" (UID: "be3d162b-06c8-4418-a135-0bd64038f695"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:33.692139 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:33.692106 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:33.743340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:33.743304 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69zvq\" (UniqueName: \"kubernetes.io/projected/be3d162b-06c8-4418-a135-0bd64038f695-kube-api-access-69zvq\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:33.743340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:33.743341 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be3d162b-06c8-4418-a135-0bd64038f695-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:34.243399 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:34.243367 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" Oct 30 19:55:34.243598 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:34.243396 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz" event={"ID":"be3d162b-06c8-4418-a135-0bd64038f695","Type":"ContainerDied","Data":"acabca71403e2c00ff3d818797de438815691b656fe4943c7f16b32a90600aea"} Oct 30 19:55:34.243598 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:34.243441 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acabca71403e2c00ff3d818797de438815691b656fe4943c7f16b32a90600aea" Oct 30 19:55:34.620168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:34.609931 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:35.399171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:35.396430 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:35.564626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:35.564579 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:35.704219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:35.703357 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth"] Oct 30 19:55:35.704219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:35.703627 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" podUID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" containerName="istio-proxy" containerID="cri-o://69f51a82bcfc292bacd23d4984a3301a9d21eaa00ff2e18e0917b2ddef968e1d" gracePeriod=30 Oct 30 19:55:35.769816 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:35.769750 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:36.000768 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:35.992074 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:36.615913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.615873 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg"] Oct 30 19:55:36.616935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.616912 5091 cpu_manager.go:414] "containerMap: removing container" podUID="be3d162b-06c8-4418-a135-0bd64038f695" containerName="revoke-keys" Oct 30 19:55:36.616996 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.616940 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3d162b-06c8-4418-a135-0bd64038f695" containerName="revoke-keys" Oct 30 19:55:36.617030 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.617014 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e3229e72-73a2-4251-a0d5-dc0632635b30" containerName="revoke-keys" Oct 30 19:55:36.617030 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.617023 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3229e72-73a2-4251-a0d5-dc0632635b30" containerName="revoke-keys" Oct 30 19:55:36.617257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.617233 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3d162b-06c8-4418-a135-0bd64038f695" containerName="revoke-keys" Oct 30 19:55:36.617361 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.617267 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3229e72-73a2-4251-a0d5-dc0632635b30" containerName="revoke-keys" Oct 30 19:55:36.622693 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.622672 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" Oct 30 19:55:36.635339 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.633640 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:55:36.635514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.635371 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:55:36.642848 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.642336 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg"] Oct 30 19:55:36.732685 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.732651 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da0b084c-s4jrg\" (UID: \"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" Oct 30 19:55:36.733051 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.732822 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da0b084c-s4jrg\" (UID: \"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" Oct 30 19:55:36.733051 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.732926 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ss97\" (UniqueName: \"kubernetes.io/projected/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-kube-api-access-8ss97\") pod \"maas-api-revoke-sub-da0b084c-s4jrg\" (UID: \"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" Oct 30 19:55:36.841368 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.841319 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da0b084c-s4jrg\" (UID: \"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" Oct 30 19:55:36.843093 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.843063 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da0b084c-s4jrg\" (UID: \"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" Oct 30 19:55:36.867225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.843370 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ss97\" (UniqueName: \"kubernetes.io/projected/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-kube-api-access-8ss97\") pod \"maas-api-revoke-sub-da0b084c-s4jrg\" (UID: \"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" Oct 30 19:55:36.881437 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.849686 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da0b084c-s4jrg\" (UID: \"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" Oct 30 19:55:36.887214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.868016 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da0b084c-s4jrg\" (UID: \"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" Oct 30 19:55:36.887214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.873332 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:36.892513 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.892474 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ss97\" (UniqueName: \"kubernetes.io/projected/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-kube-api-access-8ss97\") pod \"maas-api-revoke-sub-da0b084c-s4jrg\" (UID: \"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" Oct 30 19:55:36.940631 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:36.940601 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" Oct 30 19:55:37.247668 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:37.247639 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg"] Oct 30 19:55:37.252535 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:55:37.252500 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd93c0a_27fa_4b3f_a54d_9e3ef0712f2a.slice/crio-b98c24ef28aba328a5288c2bc3b645fea47259e884cacaa0654ea328b2ee5fbb WatchSource:0}: Error finding container b98c24ef28aba328a5288c2bc3b645fea47259e884cacaa0654ea328b2ee5fbb: Status 404 returned error can't find the container with id b98c24ef28aba328a5288c2bc3b645fea47259e884cacaa0654ea328b2ee5fbb Oct 30 19:55:37.299512 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:37.299480 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" event={"ID":"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a","Type":"ContainerStarted","Data":"b98c24ef28aba328a5288c2bc3b645fea47259e884cacaa0654ea328b2ee5fbb"} Oct 30 19:55:37.421135 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:37.421088 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:37.939169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:37.935716 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:38.009167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:38.008302 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:55:38.009167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:38.008365 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" podUID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:38.163880 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:38.163744 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:38.312066 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:38.312035 5091 generic.go:356] "Generic (PLEG): container finished" podID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" containerID="95dddeac43c00e2a488aa748dd39e4ada5cf271b8855707a4bddede54450ba48" exitCode=0 Oct 30 19:55:38.312239 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:38.312105 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" event={"ID":"57f477a5-ec74-470b-bb7e-afd4d690e8c7","Type":"ContainerDied","Data":"95dddeac43c00e2a488aa748dd39e4ada5cf271b8855707a4bddede54450ba48"} Oct 30 19:55:38.312616 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:38.312591 5091 scope.go:122] "RemoveContainer" containerID="95dddeac43c00e2a488aa748dd39e4ada5cf271b8855707a4bddede54450ba48" Oct 30 19:55:38.314295 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:38.314269 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:55:38.314395 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:38.314344 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:55:38.314453 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:38.314395 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:55:38.314510 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:38.314450 5091 generic.go:356] "Generic (PLEG): container finished" podID="8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a" containerID="4983fdc4b1de371a23c3c6442be191c3c8e9915b02f353c1aae890ba52acff53" exitCode=0 Oct 30 19:55:38.314571 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:38.314542 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" event={"ID":"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a","Type":"ContainerDied","Data":"4983fdc4b1de371a23c3c6442be191c3c8e9915b02f353c1aae890ba52acff53"} Oct 30 19:55:38.534897 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:38.534807 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:39.089536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.089498 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:39.326599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.326566 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" event={"ID":"57f477a5-ec74-470b-bb7e-afd4d690e8c7","Type":"ContainerStarted","Data":"7aa3069b2f5607b1aadfb82a6851d55ed03154cef57075c24f42705c8d855dd1"} Oct 30 19:55:39.568169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.564445 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" Oct 30 19:55:39.738243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.738156 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-maas-api-openshift-ca\") pod \"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a\" (UID: \"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a\") " Oct 30 19:55:39.738406 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.738244 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-kube-api-access-8ss97\" (UniqueName: \"kubernetes.io/projected/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-kube-api-access-8ss97\") pod \"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a\" (UID: \"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a\") " Oct 30 19:55:39.738406 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.738286 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-maas-api-xks-ca\") pod \"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a\" (UID: \"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a\") " Oct 30 19:55:39.738539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.738481 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-maas-api-openshift-ca" pod "8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a" (UID: "8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:39.738802 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.738779 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:39.740437 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.740405 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-maas-api-xks-ca" pod "8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a" (UID: "8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:39.740539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.740484 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-kube-api-access-8ss97" pod "8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a" (UID: "8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a"). InnerVolumeSpecName "kube-api-access-8ss97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:39.799080 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.799041 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:55:39.799267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.799246 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 19:55:39.799329 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.799293 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" podUID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 19:55:39.840228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.840196 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ss97\" (UniqueName: \"kubernetes.io/projected/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-kube-api-access-8ss97\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:39.840228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.840229 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:39.985778 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:39.985738 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:40.336696 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:40.336665 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" Oct 30 19:55:40.337171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:40.336701 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg" event={"ID":"8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a","Type":"ContainerDied","Data":"b98c24ef28aba328a5288c2bc3b645fea47259e884cacaa0654ea328b2ee5fbb"} Oct 30 19:55:40.337171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:40.336747 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b98c24ef28aba328a5288c2bc3b645fea47259e884cacaa0654ea328b2ee5fbb" Oct 30 19:55:40.799285 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:40.799248 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 19:55:40.799446 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:40.799310 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" podUID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 19:55:41.087578 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.087549 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:55:41.280765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.280678 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-envoy\") pod \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " Oct 30 19:55:41.281031 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.280800 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-kube-api-access-2cnsk\" (UniqueName: \"kubernetes.io/projected/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-kube-api-access-2cnsk\") pod \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " Oct 30 19:55:41.281031 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.280917 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-credential-socket\") pod \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " Oct 30 19:55:41.281031 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.280962 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-data\") pod \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " Oct 30 19:55:41.281271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.281049 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istiod-ca-cert\") pod \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " Oct 30 19:55:41.281271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.281189 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-podinfo\") pod \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " Oct 30 19:55:41.281271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.281244 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-workload-certs\") pod \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " Oct 30 19:55:41.281451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.281271 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-credential-socket" pod "fed9893a-07bd-4fb4-8a22-2c55b0d122bc" (UID: "fed9893a-07bd-4fb4-8a22-2c55b0d122bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:41.281451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.281289 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-data" pod "fed9893a-07bd-4fb4-8a22-2c55b0d122bc" (UID: "fed9893a-07bd-4fb4-8a22-2c55b0d122bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:41.281451 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.281312 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-token\" (UniqueName: \"kubernetes.io/projected/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-token\") pod \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " Oct 30 19:55:41.281623 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.281484 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istiod-ca-cert" pod "fed9893a-07bd-4fb4-8a22-2c55b0d122bc" (UID: "fed9893a-07bd-4fb4-8a22-2c55b0d122bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:41.281623 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.281504 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-workload-certs" pod "fed9893a-07bd-4fb4-8a22-2c55b0d122bc" (UID: "fed9893a-07bd-4fb4-8a22-2c55b0d122bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:41.281623 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.281535 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-workload-socket\") pod \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\" (UID: \"fed9893a-07bd-4fb4-8a22-2c55b0d122bc\") " Oct 30 19:55:41.281927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.281904 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-workload-socket" pod "fed9893a-07bd-4fb4-8a22-2c55b0d122bc" (UID: "fed9893a-07bd-4fb4-8a22-2c55b0d122bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:41.282263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.282241 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:41.282358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.282267 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:41.282358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.282284 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:41.282358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.282298 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:41.282358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.282311 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:41.283538 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.283494 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-envoy" pod "fed9893a-07bd-4fb4-8a22-2c55b0d122bc" (UID: "fed9893a-07bd-4fb4-8a22-2c55b0d122bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:41.283665 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.283643 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-token" pod "fed9893a-07bd-4fb4-8a22-2c55b0d122bc" (UID: "fed9893a-07bd-4fb4-8a22-2c55b0d122bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:41.283769 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.283717 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-podinfo" pod "fed9893a-07bd-4fb4-8a22-2c55b0d122bc" (UID: "fed9893a-07bd-4fb4-8a22-2c55b0d122bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:55:41.284045 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.284023 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-kube-api-access-2cnsk" pod "fed9893a-07bd-4fb4-8a22-2c55b0d122bc" (UID: "fed9893a-07bd-4fb4-8a22-2c55b0d122bc"). InnerVolumeSpecName "kube-api-access-2cnsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:41.364171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.353786 5091 generic.go:356] "Generic (PLEG): container finished" podID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" containerID="69f51a82bcfc292bacd23d4984a3301a9d21eaa00ff2e18e0917b2ddef968e1d" exitCode=0 Oct 30 19:55:41.364171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.354029 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" event={"ID":"fed9893a-07bd-4fb4-8a22-2c55b0d122bc","Type":"ContainerDied","Data":"69f51a82bcfc292bacd23d4984a3301a9d21eaa00ff2e18e0917b2ddef968e1d"} Oct 30 19:55:41.364171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.354085 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" event={"ID":"fed9893a-07bd-4fb4-8a22-2c55b0d122bc","Type":"ContainerDied","Data":"af3e415e14c7227f2543c6aaec44fd92661891685fe319669219548d1b29c0e1"} Oct 30 19:55:41.364171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.354116 5091 scope.go:122] "RemoveContainer" containerID="69f51a82bcfc292bacd23d4984a3301a9d21eaa00ff2e18e0917b2ddef968e1d" Oct 30 19:55:41.364171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.354311 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth" Oct 30 19:55:41.403172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.397896 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cnsk\" (UniqueName: \"kubernetes.io/projected/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-kube-api-access-2cnsk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:41.403172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.397942 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:41.403172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.397967 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:41.403172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.397979 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed9893a-07bd-4fb4-8a22-2c55b0d122bc-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:41.470853 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.470824 5091 scope.go:122] "RemoveContainer" containerID="462df2709f0731a0695283434955e2ac9857ad2062b17d2fdd08554e82865560" Oct 30 19:55:41.499453 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.499077 5091 scope.go:122] "RemoveContainer" containerID="69f51a82bcfc292bacd23d4984a3301a9d21eaa00ff2e18e0917b2ddef968e1d" Oct 30 19:55:41.499697 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:55:41.499624 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f51a82bcfc292bacd23d4984a3301a9d21eaa00ff2e18e0917b2ddef968e1d\": container with ID starting with 69f51a82bcfc292bacd23d4984a3301a9d21eaa00ff2e18e0917b2ddef968e1d not found: ID does not exist" containerID="69f51a82bcfc292bacd23d4984a3301a9d21eaa00ff2e18e0917b2ddef968e1d" Oct 30 19:55:41.499697 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.499664 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f51a82bcfc292bacd23d4984a3301a9d21eaa00ff2e18e0917b2ddef968e1d"} err="failed to get container status \"69f51a82bcfc292bacd23d4984a3301a9d21eaa00ff2e18e0917b2ddef968e1d\": rpc error: code = NotFound desc = could not find container \"69f51a82bcfc292bacd23d4984a3301a9d21eaa00ff2e18e0917b2ddef968e1d\": container with ID starting with 69f51a82bcfc292bacd23d4984a3301a9d21eaa00ff2e18e0917b2ddef968e1d not found: ID does not exist" Oct 30 19:55:41.499697 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.499690 5091 scope.go:122] "RemoveContainer" containerID="462df2709f0731a0695283434955e2ac9857ad2062b17d2fdd08554e82865560" Oct 30 19:55:41.502094 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:55:41.502066 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"462df2709f0731a0695283434955e2ac9857ad2062b17d2fdd08554e82865560\": container with ID starting with 462df2709f0731a0695283434955e2ac9857ad2062b17d2fdd08554e82865560 not found: ID does not exist" containerID="462df2709f0731a0695283434955e2ac9857ad2062b17d2fdd08554e82865560" Oct 30 19:55:41.502226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.502104 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"462df2709f0731a0695283434955e2ac9857ad2062b17d2fdd08554e82865560"} err="failed to get container status \"462df2709f0731a0695283434955e2ac9857ad2062b17d2fdd08554e82865560\": rpc error: code = NotFound desc = could not find container \"462df2709f0731a0695283434955e2ac9857ad2062b17d2fdd08554e82865560\": container with ID starting with 462df2709f0731a0695283434955e2ac9857ad2062b17d2fdd08554e82865560 not found: ID does not exist" Oct 30 19:55:41.564222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.558711 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:41.578620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.578571 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth"] Oct 30 19:55:41.583214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.583187 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-55a8e6-openshift-default-dccc759cc-hswth"] Oct 30 19:55:41.798448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.798409 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 19:55:41.798660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.798483 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" podUID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 19:55:41.840730 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:41.840645 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:42.144105 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:42.144011 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:42.443278 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:42.443201 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" path="/var/lib/kubelet/pods/fed9893a-07bd-4fb4-8a22-2c55b0d122bc/volumes" Oct 30 19:55:42.805042 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:42.802498 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 19:55:42.805042 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:42.802562 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" podUID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 19:55:42.838172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:42.835714 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:42.993170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:42.992817 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj"] Oct 30 19:55:42.993568 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:42.993550 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a" containerName="revoke-keys" Oct 30 19:55:42.993614 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:42.993569 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a" containerName="revoke-keys" Oct 30 19:55:42.993614 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:42.993586 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" containerName="istio-proxy" Oct 30 19:55:42.993614 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:42.993591 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" containerName="istio-proxy" Oct 30 19:55:42.993742 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:42.993732 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a" containerName="revoke-keys" Oct 30 19:55:42.993779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:42.993745 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" containerName="istio-proxy" Oct 30 19:55:42.993779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:42.993755 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" containerName="istio-proxy" Oct 30 19:55:42.998553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:42.998531 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" Oct 30 19:55:43.011377 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.011343 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-927bf8\"" Oct 30 19:55:43.012038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.012018 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-927bf8-dockercfg-jd75v\"" Oct 30 19:55:43.013645 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.013610 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj"] Oct 30 19:55:43.178969 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.164765 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72z9q\" (UniqueName: \"kubernetes.io/projected/b4277311-2a23-4736-99a8-29b0690cbeb4-kube-api-access-72z9q\") pod \"payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj\" (UID: \"b4277311-2a23-4736-99a8-29b0690cbeb4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" Oct 30 19:55:43.178969 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.164947 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4277311-2a23-4736-99a8-29b0690cbeb4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj\" (UID: \"b4277311-2a23-4736-99a8-29b0690cbeb4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" Oct 30 19:55:43.225417 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.223785 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:43.270178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.268892 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4277311-2a23-4736-99a8-29b0690cbeb4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj\" (UID: \"b4277311-2a23-4736-99a8-29b0690cbeb4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" Oct 30 19:55:43.270178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.269107 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72z9q\" (UniqueName: \"kubernetes.io/projected/b4277311-2a23-4736-99a8-29b0690cbeb4-kube-api-access-72z9q\") pod \"payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj\" (UID: \"b4277311-2a23-4736-99a8-29b0690cbeb4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" Oct 30 19:55:43.274170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.270842 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4277311-2a23-4736-99a8-29b0690cbeb4-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj\" (UID: \"b4277311-2a23-4736-99a8-29b0690cbeb4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" Oct 30 19:55:43.402008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.401908 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72z9q\" (UniqueName: \"kubernetes.io/projected/b4277311-2a23-4736-99a8-29b0690cbeb4-kube-api-access-72z9q\") pod \"payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj\" (UID: \"b4277311-2a23-4736-99a8-29b0690cbeb4\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" Oct 30 19:55:43.623338 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.623295 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" Oct 30 19:55:43.626542 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.626513 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz"] Oct 30 19:55:43.627272 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.627255 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" containerName="istio-proxy" Oct 30 19:55:43.627352 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.627272 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed9893a-07bd-4fb4-8a22-2c55b0d122bc" containerName="istio-proxy" Oct 30 19:55:43.631372 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.631352 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" Oct 30 19:55:43.639225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.639164 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:55:43.639315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.639259 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:55:43.643573 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.643544 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz"] Oct 30 19:55:43.701694 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.701663 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8acc1040-cd5f-4233-8372-b74b4309229e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09beadb9-zlkqz\" (UID: \"8acc1040-cd5f-4233-8372-b74b4309229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" Oct 30 19:55:43.701862 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.701751 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vttkg\" (UniqueName: \"kubernetes.io/projected/8acc1040-cd5f-4233-8372-b74b4309229e-kube-api-access-vttkg\") pod \"maas-api-revoke-sub-09beadb9-zlkqz\" (UID: \"8acc1040-cd5f-4233-8372-b74b4309229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" Oct 30 19:55:43.701862 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.701844 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8acc1040-cd5f-4233-8372-b74b4309229e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09beadb9-zlkqz\" (UID: \"8acc1040-cd5f-4233-8372-b74b4309229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" Oct 30 19:55:43.808340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.798705 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Oct 30 19:55:43.808340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.798757 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" podUID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Oct 30 19:55:43.808340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.802906 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vttkg\" (UniqueName: \"kubernetes.io/projected/8acc1040-cd5f-4233-8372-b74b4309229e-kube-api-access-vttkg\") pod \"maas-api-revoke-sub-09beadb9-zlkqz\" (UID: \"8acc1040-cd5f-4233-8372-b74b4309229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" Oct 30 19:55:43.808340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.803326 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8acc1040-cd5f-4233-8372-b74b4309229e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09beadb9-zlkqz\" (UID: \"8acc1040-cd5f-4233-8372-b74b4309229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" Oct 30 19:55:43.808340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.803411 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8acc1040-cd5f-4233-8372-b74b4309229e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09beadb9-zlkqz\" (UID: \"8acc1040-cd5f-4233-8372-b74b4309229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" Oct 30 19:55:43.808340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.806191 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8acc1040-cd5f-4233-8372-b74b4309229e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-09beadb9-zlkqz\" (UID: \"8acc1040-cd5f-4233-8372-b74b4309229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" Oct 30 19:55:43.808340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.806857 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8acc1040-cd5f-4233-8372-b74b4309229e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-09beadb9-zlkqz\" (UID: \"8acc1040-cd5f-4233-8372-b74b4309229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" Oct 30 19:55:43.828047 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.827994 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vttkg\" (UniqueName: \"kubernetes.io/projected/8acc1040-cd5f-4233-8372-b74b4309229e-kube-api-access-vttkg\") pod \"maas-api-revoke-sub-09beadb9-zlkqz\" (UID: \"8acc1040-cd5f-4233-8372-b74b4309229e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" Oct 30 19:55:43.910103 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:43.910028 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:44.002100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.002069 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" Oct 30 19:55:44.018519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.018188 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj"] Oct 30 19:55:44.128232 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.128105 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:44.399200 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.390190 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb"] Oct 30 19:55:44.399200 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.393773 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d2dd78-wk5mb"] Oct 30 19:55:44.400892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.400730 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:44.532663 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.532635 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3229e72-73a2-4251-a0d5-dc0632635b30" path="/var/lib/kubelet/pods/e3229e72-73a2-4251-a0d5-dc0632635b30/volumes" Oct 30 19:55:44.533221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.533198 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" event={"ID":"b4277311-2a23-4736-99a8-29b0690cbeb4","Type":"ContainerStarted","Data":"81cc912fabb0c5ced91ae414fd9f5ba788b619d6e8f9360f9cf785e603abab65"} Oct 30 19:55:44.533382 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.533369 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" event={"ID":"b4277311-2a23-4736-99a8-29b0690cbeb4","Type":"ContainerStarted","Data":"3f46e8edb8e9c63055c2c8d88eb9e769afa748601d10e3e22cab4693f82abee7"} Oct 30 19:55:44.533468 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.533459 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" Oct 30 19:55:44.539619 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.539548 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" podStartSLOduration=2.539531554 podStartE2EDuration="2.539531554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:44.537494137 +0000 UTC m=+1464.232702098" watchObservedRunningTime="2026-10-30 19:55:44.539531554 +0000 UTC m=+1464.234739513" Oct 30 19:55:44.550115 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.550050 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:44.574224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.573645 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz"] Oct 30 19:55:44.582960 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:55:44.582897 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8acc1040_cd5f_4233_8372_b74b4309229e.slice/crio-3226584985df0b77215340b80da7cff23de231474745e970bff67e50e9190a66 WatchSource:0}: Error finding container 3226584985df0b77215340b80da7cff23de231474745e970bff67e50e9190a66: Status 404 returned error can't find the container with id 3226584985df0b77215340b80da7cff23de231474745e970bff67e50e9190a66 Oct 30 19:55:44.843777 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.843681 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:55:44.844931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.844687 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:55:44.845478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.845456 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:55:44.896200 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:44.896133 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:45.423992 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:45.423957 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:45.553994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:45.553367 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" event={"ID":"8acc1040-cd5f-4233-8372-b74b4309229e","Type":"ContainerStarted","Data":"081107cf61caf6fd577131076a96bb92b0d3c87adf1f36bcb9f8aa0b8b556cf3"} Oct 30 19:55:45.553994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:45.553433 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" event={"ID":"8acc1040-cd5f-4233-8372-b74b4309229e","Type":"ContainerStarted","Data":"3226584985df0b77215340b80da7cff23de231474745e970bff67e50e9190a66"} Oct 30 19:55:45.649183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:45.648429 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:45.982425 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:55:45.982387 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:55:46.607169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:46.598821 5091 generic.go:356] "Generic (PLEG): container finished" podID="8acc1040-cd5f-4233-8372-b74b4309229e" containerID="081107cf61caf6fd577131076a96bb92b0d3c87adf1f36bcb9f8aa0b8b556cf3" exitCode=0 Oct 30 19:55:46.611170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:46.610118 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" event={"ID":"8acc1040-cd5f-4233-8372-b74b4309229e","Type":"ContainerDied","Data":"081107cf61caf6fd577131076a96bb92b0d3c87adf1f36bcb9f8aa0b8b556cf3"} Oct 30 19:55:46.635898 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:46.635857 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:47.793691 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:47.793657 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" Oct 30 19:55:47.831476 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:47.831444 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8acc1040-cd5f-4233-8372-b74b4309229e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8acc1040-cd5f-4233-8372-b74b4309229e-maas-api-xks-ca\") pod \"8acc1040-cd5f-4233-8372-b74b4309229e\" (UID: \"8acc1040-cd5f-4233-8372-b74b4309229e\") " Oct 30 19:55:47.831651 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:47.831532 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8acc1040-cd5f-4233-8372-b74b4309229e-kube-api-access-vttkg\" (UniqueName: \"kubernetes.io/projected/8acc1040-cd5f-4233-8372-b74b4309229e-kube-api-access-vttkg\") pod \"8acc1040-cd5f-4233-8372-b74b4309229e\" (UID: \"8acc1040-cd5f-4233-8372-b74b4309229e\") " Oct 30 19:55:47.831651 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:47.831594 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8acc1040-cd5f-4233-8372-b74b4309229e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8acc1040-cd5f-4233-8372-b74b4309229e-maas-api-openshift-ca\") pod \"8acc1040-cd5f-4233-8372-b74b4309229e\" (UID: \"8acc1040-cd5f-4233-8372-b74b4309229e\") " Oct 30 19:55:47.846091 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:47.832011 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8acc1040-cd5f-4233-8372-b74b4309229e-maas-api-openshift-ca" pod "8acc1040-cd5f-4233-8372-b74b4309229e" (UID: "8acc1040-cd5f-4233-8372-b74b4309229e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:47.846091 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:47.832243 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8acc1040-cd5f-4233-8372-b74b4309229e-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:47.846091 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:47.833784 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8acc1040-cd5f-4233-8372-b74b4309229e-maas-api-xks-ca" pod "8acc1040-cd5f-4233-8372-b74b4309229e" (UID: "8acc1040-cd5f-4233-8372-b74b4309229e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:47.846091 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:47.834650 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acc1040-cd5f-4233-8372-b74b4309229e-kube-api-access-vttkg" pod "8acc1040-cd5f-4233-8372-b74b4309229e" (UID: "8acc1040-cd5f-4233-8372-b74b4309229e"). InnerVolumeSpecName "kube-api-access-vttkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:47.933763 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:47.933688 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8acc1040-cd5f-4233-8372-b74b4309229e-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:47.933763 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:47.933715 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vttkg\" (UniqueName: \"kubernetes.io/projected/8acc1040-cd5f-4233-8372-b74b4309229e-kube-api-access-vttkg\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:48.092169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:48.083828 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:48.616062 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:48.616028 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" event={"ID":"8acc1040-cd5f-4233-8372-b74b4309229e","Type":"ContainerDied","Data":"3226584985df0b77215340b80da7cff23de231474745e970bff67e50e9190a66"} Oct 30 19:55:48.616062 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:48.616056 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz" Oct 30 19:55:48.616329 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:48.616078 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3226584985df0b77215340b80da7cff23de231474745e970bff67e50e9190a66" Oct 30 19:55:48.962360 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:48.962319 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:49.089235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.089202 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl"] Oct 30 19:55:49.089851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.089834 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8acc1040-cd5f-4233-8372-b74b4309229e" containerName="revoke-keys" Oct 30 19:55:49.089851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.089849 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8acc1040-cd5f-4233-8372-b74b4309229e" containerName="revoke-keys" Oct 30 19:55:49.090038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.089966 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8acc1040-cd5f-4233-8372-b74b4309229e" containerName="revoke-keys" Oct 30 19:55:49.093741 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.093718 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" Oct 30 19:55:49.099270 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.099242 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:55:49.103183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.101296 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:55:49.103183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.102791 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl"] Oct 30 19:55:49.148530 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.148501 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57mbk\" (UniqueName: \"kubernetes.io/projected/397b980b-d738-4eee-b685-3404c364fb1e-kube-api-access-57mbk\") pod \"maas-api-revoke-sub-98fda6a7-n8wdl\" (UID: \"397b980b-d738-4eee-b685-3404c364fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" Oct 30 19:55:49.148676 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.148577 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/397b980b-d738-4eee-b685-3404c364fb1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98fda6a7-n8wdl\" (UID: \"397b980b-d738-4eee-b685-3404c364fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" Oct 30 19:55:49.148759 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.148685 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/397b980b-d738-4eee-b685-3404c364fb1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98fda6a7-n8wdl\" (UID: \"397b980b-d738-4eee-b685-3404c364fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" Oct 30 19:55:49.249866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.249832 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57mbk\" (UniqueName: \"kubernetes.io/projected/397b980b-d738-4eee-b685-3404c364fb1e-kube-api-access-57mbk\") pod \"maas-api-revoke-sub-98fda6a7-n8wdl\" (UID: \"397b980b-d738-4eee-b685-3404c364fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" Oct 30 19:55:49.250066 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.249939 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/397b980b-d738-4eee-b685-3404c364fb1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98fda6a7-n8wdl\" (UID: \"397b980b-d738-4eee-b685-3404c364fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" Oct 30 19:55:49.250066 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.249987 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/397b980b-d738-4eee-b685-3404c364fb1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98fda6a7-n8wdl\" (UID: \"397b980b-d738-4eee-b685-3404c364fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" Oct 30 19:55:49.250912 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.250887 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/397b980b-d738-4eee-b685-3404c364fb1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98fda6a7-n8wdl\" (UID: \"397b980b-d738-4eee-b685-3404c364fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" Oct 30 19:55:49.252524 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.252501 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/397b980b-d738-4eee-b685-3404c364fb1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98fda6a7-n8wdl\" (UID: \"397b980b-d738-4eee-b685-3404c364fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" Oct 30 19:55:49.262306 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.262277 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:49.273326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.273308 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57mbk\" (UniqueName: \"kubernetes.io/projected/397b980b-d738-4eee-b685-3404c364fb1e-kube-api-access-57mbk\") pod \"maas-api-revoke-sub-98fda6a7-n8wdl\" (UID: \"397b980b-d738-4eee-b685-3404c364fb1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" Oct 30 19:55:49.413346 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.413315 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" Oct 30 19:55:49.703692 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:55:49.703659 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397b980b_d738_4eee_b685_3404c364fb1e.slice/crio-9078eedede275b5a9464734f2cf6cfac0123f306675122451142985e6a114468 WatchSource:0}: Error finding container 9078eedede275b5a9464734f2cf6cfac0123f306675122451142985e6a114468: Status 404 returned error can't find the container with id 9078eedede275b5a9464734f2cf6cfac0123f306675122451142985e6a114468 Oct 30 19:55:49.705963 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:49.705942 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl"] Oct 30 19:55:50.110230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:50.105014 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:50.726083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:50.726046 5091 generic.go:356] "Generic (PLEG): container finished" podID="397b980b-d738-4eee-b685-3404c364fb1e" containerID="00ca306477bdb853ca39453ad41cf69a27b85162f3c4159aa0bd9326f3fb6f78" exitCode=0 Oct 30 19:55:50.726476 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:50.726453 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" event={"ID":"397b980b-d738-4eee-b685-3404c364fb1e","Type":"ContainerDied","Data":"00ca306477bdb853ca39453ad41cf69a27b85162f3c4159aa0bd9326f3fb6f78"} Oct 30 19:55:50.726623 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:50.726611 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" event={"ID":"397b980b-d738-4eee-b685-3404c364fb1e","Type":"ContainerStarted","Data":"9078eedede275b5a9464734f2cf6cfac0123f306675122451142985e6a114468"} Oct 30 19:55:52.119122 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:52.119027 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:52.418564 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:52.418531 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:52.457282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:52.457213 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:52.512275 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:52.512225 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:52.626170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:52.622132 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:52.870303 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:52.870259 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:53.532353 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:53.532306 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 19:55:53.582655 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:53.552965 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:55:53.622170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:53.596452 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:55:53.622170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:53.597082 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:55:53.664168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:53.640501 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:53.678191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:53.677934 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:54.171886 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.171847 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:54.313470 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.313372 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:54.321308 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.321279 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" Oct 30 19:55:54.364316 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.364264 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:54.364642 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.364619 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf"] Oct 30 19:55:54.371952 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.364960 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" podUID="8d15b25d-07cf-4e6c-8446-60915a92b92e" containerName="istio-proxy" containerID="cri-o://179fdd028d79e8880f229f84ed5828d4c8addc2eb8c6ca4e5a85538d1a83ffec" gracePeriod=30 Oct 30 19:55:54.549172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.536260 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/397b980b-d738-4eee-b685-3404c364fb1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/397b980b-d738-4eee-b685-3404c364fb1e-maas-api-xks-ca\") pod \"397b980b-d738-4eee-b685-3404c364fb1e\" (UID: \"397b980b-d738-4eee-b685-3404c364fb1e\") " Oct 30 19:55:54.602465 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.602376 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/397b980b-d738-4eee-b685-3404c364fb1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/397b980b-d738-4eee-b685-3404c364fb1e-maas-api-openshift-ca\") pod \"397b980b-d738-4eee-b685-3404c364fb1e\" (UID: \"397b980b-d738-4eee-b685-3404c364fb1e\") " Oct 30 19:55:54.602873 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.602853 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/397b980b-d738-4eee-b685-3404c364fb1e-kube-api-access-57mbk\" (UniqueName: \"kubernetes.io/projected/397b980b-d738-4eee-b685-3404c364fb1e-kube-api-access-57mbk\") pod \"397b980b-d738-4eee-b685-3404c364fb1e\" (UID: \"397b980b-d738-4eee-b685-3404c364fb1e\") " Oct 30 19:55:54.630009 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.552728 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:55:54.631296 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.602264 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:55:54.661214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.652391 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/397b980b-d738-4eee-b685-3404c364fb1e-maas-api-openshift-ca" pod "397b980b-d738-4eee-b685-3404c364fb1e" (UID: "397b980b-d738-4eee-b685-3404c364fb1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:54.661214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.652485 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 19:55:54.777180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.776462 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/397b980b-d738-4eee-b685-3404c364fb1e-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:54.777180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.776688 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397b980b-d738-4eee-b685-3404c364fb1e-kube-api-access-57mbk" pod "397b980b-d738-4eee-b685-3404c364fb1e" (UID: "397b980b-d738-4eee-b685-3404c364fb1e"). InnerVolumeSpecName "kube-api-access-57mbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:54.816282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.814335 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:55:54.827389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.827342 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz"] Oct 30 19:55:54.949349 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.852633 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/397b980b-d738-4eee-b685-3404c364fb1e-maas-api-xks-ca" pod "397b980b-d738-4eee-b685-3404c364fb1e" (UID: "397b980b-d738-4eee-b685-3404c364fb1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:54.966369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.894053 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57mbk\" (UniqueName: \"kubernetes.io/projected/397b980b-d738-4eee-b685-3404c364fb1e-kube-api-access-57mbk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:54.966619 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.956959 5091 cpu_manager.go:414] "containerMap: removing container" podUID="397b980b-d738-4eee-b685-3404c364fb1e" containerName="revoke-keys" Oct 30 19:55:54.966778 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:54.966759 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397b980b-d738-4eee-b685-3404c364fb1e" containerName="revoke-keys" Oct 30 19:55:55.081020 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.080983 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="397b980b-d738-4eee-b685-3404c364fb1e" containerName="revoke-keys" Oct 30 19:55:55.140838 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.140804 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:55:55.141169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.141132 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:55:55.141337 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.141321 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz"] Oct 30 19:55:55.141615 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.141598 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" Oct 30 19:55:55.214700 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.214553 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:55:55.225871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.225845 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" Oct 30 19:55:55.226929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.226892 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl" event={"ID":"397b980b-d738-4eee-b685-3404c364fb1e","Type":"ContainerDied","Data":"9078eedede275b5a9464734f2cf6cfac0123f306675122451142985e6a114468"} Oct 30 19:55:55.227073 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.226957 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9078eedede275b5a9464734f2cf6cfac0123f306675122451142985e6a114468" Oct 30 19:55:55.247246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.246724 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/292828eb-f86c-47d1-8113-9213119a3afb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz\" (UID: \"292828eb-f86c-47d1-8113-9213119a3afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" Oct 30 19:55:55.251109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.250114 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrjgn\" (UniqueName: \"kubernetes.io/projected/292828eb-f86c-47d1-8113-9213119a3afb-kube-api-access-hrjgn\") pod \"maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz\" (UID: \"292828eb-f86c-47d1-8113-9213119a3afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" Oct 30 19:55:55.251109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.250417 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/292828eb-f86c-47d1-8113-9213119a3afb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz\" (UID: \"292828eb-f86c-47d1-8113-9213119a3afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" Oct 30 19:55:55.251109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.250840 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/397b980b-d738-4eee-b685-3404c364fb1e-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:55.345085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.345045 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:55.353088 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.351834 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/292828eb-f86c-47d1-8113-9213119a3afb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz\" (UID: \"292828eb-f86c-47d1-8113-9213119a3afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" Oct 30 19:55:55.353088 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.351928 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrjgn\" (UniqueName: \"kubernetes.io/projected/292828eb-f86c-47d1-8113-9213119a3afb-kube-api-access-hrjgn\") pod \"maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz\" (UID: \"292828eb-f86c-47d1-8113-9213119a3afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" Oct 30 19:55:55.353088 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.352042 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/292828eb-f86c-47d1-8113-9213119a3afb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz\" (UID: \"292828eb-f86c-47d1-8113-9213119a3afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" Oct 30 19:55:55.360210 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.355067 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/292828eb-f86c-47d1-8113-9213119a3afb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz\" (UID: \"292828eb-f86c-47d1-8113-9213119a3afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" Oct 30 19:55:55.360210 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.355696 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/292828eb-f86c-47d1-8113-9213119a3afb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz\" (UID: \"292828eb-f86c-47d1-8113-9213119a3afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" Oct 30 19:55:55.372909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.372878 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrjgn\" (UniqueName: \"kubernetes.io/projected/292828eb-f86c-47d1-8113-9213119a3afb-kube-api-access-hrjgn\") pod \"maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz\" (UID: \"292828eb-f86c-47d1-8113-9213119a3afb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" Oct 30 19:55:55.382596 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:55:55.382089 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod397b980b_d738_4eee_b685_3404c364fb1e.slice\": RecentStats: unable to find data in memory cache]" Oct 30 19:55:55.549883 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.549849 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" Oct 30 19:55:55.844483 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:55:55.844453 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod292828eb_f86c_47d1_8113_9213119a3afb.slice/crio-de893b2afb6e8ec214d5aadbe1863b34c7ba12adfa0cc8662a4774106540d6d4 WatchSource:0}: Error finding container de893b2afb6e8ec214d5aadbe1863b34c7ba12adfa0cc8662a4774106540d6d4: Status 404 returned error can't find the container with id de893b2afb6e8ec214d5aadbe1863b34c7ba12adfa0cc8662a4774106540d6d4 Oct 30 19:55:55.846680 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:55.846654 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz"] Oct 30 19:55:56.236333 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:56.236297 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" event={"ID":"292828eb-f86c-47d1-8113-9213119a3afb","Type":"ContainerStarted","Data":"9af42e62d6297b68c7754ae35f3e8d12d84b2b25cf8cadbc48b1b5ccec1e9142"} Oct 30 19:55:56.236481 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:56.236360 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" event={"ID":"292828eb-f86c-47d1-8113-9213119a3afb","Type":"ContainerStarted","Data":"de893b2afb6e8ec214d5aadbe1863b34c7ba12adfa0cc8662a4774106540d6d4"} Oct 30 19:55:56.265186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:56.265116 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" podStartSLOduration=2.265102606 podStartE2EDuration="2.265102606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:55:56.262267121 +0000 UTC m=+1475.957475080" watchObservedRunningTime="2026-10-30 19:55:56.265102606 +0000 UTC m=+1475.960310563" Oct 30 19:55:56.598168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:56.595228 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:56.624420 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:56.621431 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" Oct 30 19:55:57.246743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:57.246707 5091 generic.go:356] "Generic (PLEG): container finished" podID="292828eb-f86c-47d1-8113-9213119a3afb" containerID="9af42e62d6297b68c7754ae35f3e8d12d84b2b25cf8cadbc48b1b5ccec1e9142" exitCode=0 Oct 30 19:55:57.246921 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:57.246808 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" event={"ID":"292828eb-f86c-47d1-8113-9213119a3afb","Type":"ContainerDied","Data":"9af42e62d6297b68c7754ae35f3e8d12d84b2b25cf8cadbc48b1b5ccec1e9142"} Oct 30 19:55:57.716547 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:57.714091 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:57.888818 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:57.888784 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:58.118390 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.118358 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l"] Oct 30 19:55:58.118626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.118595 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerName="istio-proxy" containerID="cri-o://7b35a104a2c0206cddd13cdd5bad53831309527bfedf7c0908b8201c04b763a6" gracePeriod=30 Oct 30 19:55:58.316171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.315897 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:58.387966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.387886 5091 patch_prober.go:29] interesting pod/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:55:58.388375 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.388345 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:55:58.596657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.596630 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" Oct 30 19:55:58.633946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.633908 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/292828eb-f86c-47d1-8113-9213119a3afb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/292828eb-f86c-47d1-8113-9213119a3afb-maas-api-openshift-ca\") pod \"292828eb-f86c-47d1-8113-9213119a3afb\" (UID: \"292828eb-f86c-47d1-8113-9213119a3afb\") " Oct 30 19:55:58.634135 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.633968 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/292828eb-f86c-47d1-8113-9213119a3afb-kube-api-access-hrjgn\" (UniqueName: \"kubernetes.io/projected/292828eb-f86c-47d1-8113-9213119a3afb-kube-api-access-hrjgn\") pod \"292828eb-f86c-47d1-8113-9213119a3afb\" (UID: \"292828eb-f86c-47d1-8113-9213119a3afb\") " Oct 30 19:55:58.634135 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.634057 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/292828eb-f86c-47d1-8113-9213119a3afb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/292828eb-f86c-47d1-8113-9213119a3afb-maas-api-xks-ca\") pod \"292828eb-f86c-47d1-8113-9213119a3afb\" (UID: \"292828eb-f86c-47d1-8113-9213119a3afb\") " Oct 30 19:55:58.634350 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.634318 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/292828eb-f86c-47d1-8113-9213119a3afb-maas-api-openshift-ca" pod "292828eb-f86c-47d1-8113-9213119a3afb" (UID: "292828eb-f86c-47d1-8113-9213119a3afb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:58.644234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.634774 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/292828eb-f86c-47d1-8113-9213119a3afb-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:58.644234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.636782 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/292828eb-f86c-47d1-8113-9213119a3afb-kube-api-access-hrjgn" pod "292828eb-f86c-47d1-8113-9213119a3afb" (UID: "292828eb-f86c-47d1-8113-9213119a3afb"). InnerVolumeSpecName "kube-api-access-hrjgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:58.644234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.637395 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/292828eb-f86c-47d1-8113-9213119a3afb-maas-api-xks-ca" pod "292828eb-f86c-47d1-8113-9213119a3afb" (UID: "292828eb-f86c-47d1-8113-9213119a3afb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:55:58.735964 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.735931 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrjgn\" (UniqueName: \"kubernetes.io/projected/292828eb-f86c-47d1-8113-9213119a3afb-kube-api-access-hrjgn\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:58.735964 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.735967 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/292828eb-f86c-47d1-8113-9213119a3afb-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:58.800091 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:58.800056 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:55:59.317860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.317827 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" Oct 30 19:55:59.318043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.317852 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz" event={"ID":"292828eb-f86c-47d1-8113-9213119a3afb","Type":"ContainerDied","Data":"de893b2afb6e8ec214d5aadbe1863b34c7ba12adfa0cc8662a4774106540d6d4"} Oct 30 19:55:59.318043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.317912 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de893b2afb6e8ec214d5aadbe1863b34c7ba12adfa0cc8662a4774106540d6d4" Oct 30 19:55:59.668909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.668875 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:55:59.860013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.859975 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-credential-socket\") pod \"8d15b25d-07cf-4e6c-8446-60915a92b92e\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " Oct 30 19:55:59.860618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.860067 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-token\" (UniqueName: \"kubernetes.io/projected/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-token\") pod \"8d15b25d-07cf-4e6c-8446-60915a92b92e\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " Oct 30 19:55:59.860618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.860199 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d15b25d-07cf-4e6c-8446-60915a92b92e-kube-api-access-ldltb\" (UniqueName: \"kubernetes.io/projected/8d15b25d-07cf-4e6c-8446-60915a92b92e-kube-api-access-ldltb\") pod \"8d15b25d-07cf-4e6c-8446-60915a92b92e\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " Oct 30 19:55:59.860618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.860263 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-workload-socket\") pod \"8d15b25d-07cf-4e6c-8446-60915a92b92e\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " Oct 30 19:55:59.860618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.860296 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-envoy\") pod \"8d15b25d-07cf-4e6c-8446-60915a92b92e\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " Oct 30 19:55:59.860618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.860374 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d15b25d-07cf-4e6c-8446-60915a92b92e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d15b25d-07cf-4e6c-8446-60915a92b92e-istiod-ca-cert\") pod \"8d15b25d-07cf-4e6c-8446-60915a92b92e\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " Oct 30 19:55:59.860618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.860380 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-credential-socket" pod "8d15b25d-07cf-4e6c-8446-60915a92b92e" (UID: "8d15b25d-07cf-4e6c-8446-60915a92b92e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:59.860618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.860419 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-workload-certs\") pod \"8d15b25d-07cf-4e6c-8446-60915a92b92e\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " Oct 30 19:55:59.860618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.860459 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-data\") pod \"8d15b25d-07cf-4e6c-8446-60915a92b92e\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " Oct 30 19:55:59.860618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.860529 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-podinfo\") pod \"8d15b25d-07cf-4e6c-8446-60915a92b92e\" (UID: \"8d15b25d-07cf-4e6c-8446-60915a92b92e\") " Oct 30 19:55:59.860618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.860534 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-workload-socket" pod "8d15b25d-07cf-4e6c-8446-60915a92b92e" (UID: "8d15b25d-07cf-4e6c-8446-60915a92b92e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:59.861112 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.860747 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d15b25d-07cf-4e6c-8446-60915a92b92e-istiod-ca-cert" pod "8d15b25d-07cf-4e6c-8446-60915a92b92e" (UID: "8d15b25d-07cf-4e6c-8446-60915a92b92e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:55:59.861112 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.860763 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-workload-certs" pod "8d15b25d-07cf-4e6c-8446-60915a92b92e" (UID: "8d15b25d-07cf-4e6c-8446-60915a92b92e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:59.861112 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.860781 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-data" pod "8d15b25d-07cf-4e6c-8446-60915a92b92e" (UID: "8d15b25d-07cf-4e6c-8446-60915a92b92e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:59.861258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.861197 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:59.861258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.861224 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8d15b25d-07cf-4e6c-8446-60915a92b92e-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:59.861258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.861238 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:59.861258 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.861251 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:59.861408 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.861264 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:59.862714 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.862686 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d15b25d-07cf-4e6c-8446-60915a92b92e-kube-api-access-ldltb" pod "8d15b25d-07cf-4e6c-8446-60915a92b92e" (UID: "8d15b25d-07cf-4e6c-8446-60915a92b92e"). InnerVolumeSpecName "kube-api-access-ldltb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:59.862800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.862767 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-podinfo" pod "8d15b25d-07cf-4e6c-8446-60915a92b92e" (UID: "8d15b25d-07cf-4e6c-8446-60915a92b92e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:55:59.862937 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.862914 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-envoy" pod "8d15b25d-07cf-4e6c-8446-60915a92b92e" (UID: "8d15b25d-07cf-4e6c-8446-60915a92b92e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:55:59.863048 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.862966 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-token" pod "8d15b25d-07cf-4e6c-8446-60915a92b92e" (UID: "8d15b25d-07cf-4e6c-8446-60915a92b92e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:55:59.962802 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.962767 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:59.962802 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.962796 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:59.962802 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.962806 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldltb\" (UniqueName: \"kubernetes.io/projected/8d15b25d-07cf-4e6c-8446-60915a92b92e-kube-api-access-ldltb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:59.963040 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.962817 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8d15b25d-07cf-4e6c-8446-60915a92b92e-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:55:59.983684 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:55:59.983643 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:00.330459 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:00.330419 5091 generic.go:356] "Generic (PLEG): container finished" podID="8d15b25d-07cf-4e6c-8446-60915a92b92e" containerID="179fdd028d79e8880f229f84ed5828d4c8addc2eb8c6ca4e5a85538d1a83ffec" exitCode=0 Oct 30 19:56:00.330631 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:00.330478 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" Oct 30 19:56:00.330631 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:00.330499 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" event={"ID":"8d15b25d-07cf-4e6c-8446-60915a92b92e","Type":"ContainerDied","Data":"179fdd028d79e8880f229f84ed5828d4c8addc2eb8c6ca4e5a85538d1a83ffec"} Oct 30 19:56:00.330631 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:00.330553 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf" event={"ID":"8d15b25d-07cf-4e6c-8446-60915a92b92e","Type":"ContainerDied","Data":"1ecb1e5429d128cccc25dad50c681abb13e75c867c955ed16fec18d6e5641c5f"} Oct 30 19:56:00.330631 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:00.330576 5091 scope.go:122] "RemoveContainer" containerID="179fdd028d79e8880f229f84ed5828d4c8addc2eb8c6ca4e5a85538d1a83ffec" Oct 30 19:56:00.342553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:00.342533 5091 scope.go:122] "RemoveContainer" containerID="1244a685b6d3d0480d68cff2d53cc40d24045899cd770f19021a2b9c170cd42b" Oct 30 19:56:00.353053 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:00.353020 5091 scope.go:122] "RemoveContainer" containerID="179fdd028d79e8880f229f84ed5828d4c8addc2eb8c6ca4e5a85538d1a83ffec" Oct 30 19:56:00.353339 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:56:00.353321 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"179fdd028d79e8880f229f84ed5828d4c8addc2eb8c6ca4e5a85538d1a83ffec\": container with ID starting with 179fdd028d79e8880f229f84ed5828d4c8addc2eb8c6ca4e5a85538d1a83ffec not found: ID does not exist" containerID="179fdd028d79e8880f229f84ed5828d4c8addc2eb8c6ca4e5a85538d1a83ffec" Oct 30 19:56:00.353397 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:00.353348 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"179fdd028d79e8880f229f84ed5828d4c8addc2eb8c6ca4e5a85538d1a83ffec"} err="failed to get container status \"179fdd028d79e8880f229f84ed5828d4c8addc2eb8c6ca4e5a85538d1a83ffec\": rpc error: code = NotFound desc = could not find container \"179fdd028d79e8880f229f84ed5828d4c8addc2eb8c6ca4e5a85538d1a83ffec\": container with ID starting with 179fdd028d79e8880f229f84ed5828d4c8addc2eb8c6ca4e5a85538d1a83ffec not found: ID does not exist" Oct 30 19:56:00.353397 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:00.353365 5091 scope.go:122] "RemoveContainer" containerID="1244a685b6d3d0480d68cff2d53cc40d24045899cd770f19021a2b9c170cd42b" Oct 30 19:56:00.353597 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:56:00.353581 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1244a685b6d3d0480d68cff2d53cc40d24045899cd770f19021a2b9c170cd42b\": container with ID starting with 1244a685b6d3d0480d68cff2d53cc40d24045899cd770f19021a2b9c170cd42b not found: ID does not exist" containerID="1244a685b6d3d0480d68cff2d53cc40d24045899cd770f19021a2b9c170cd42b" Oct 30 19:56:00.353641 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:00.353601 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1244a685b6d3d0480d68cff2d53cc40d24045899cd770f19021a2b9c170cd42b"} err="failed to get container status \"1244a685b6d3d0480d68cff2d53cc40d24045899cd770f19021a2b9c170cd42b\": rpc error: code = NotFound desc = could not find container \"1244a685b6d3d0480d68cff2d53cc40d24045899cd770f19021a2b9c170cd42b\": container with ID starting with 1244a685b6d3d0480d68cff2d53cc40d24045899cd770f19021a2b9c170cd42b not found: ID does not exist" Oct 30 19:56:00.368436 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:00.368412 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf"] Oct 30 19:56:00.372489 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:00.372466 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-d2dd78-openshift-default-7548dd44b9-sxjvf"] Oct 30 19:56:00.442292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:00.442262 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d15b25d-07cf-4e6c-8446-60915a92b92e" path="/var/lib/kubelet/pods/8d15b25d-07cf-4e6c-8446-60915a92b92e/volumes" Oct 30 19:56:01.217955 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.215594 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:01.336134 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.336101 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx"] Oct 30 19:56:01.336871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.336847 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8d15b25d-07cf-4e6c-8446-60915a92b92e" containerName="istio-proxy" Oct 30 19:56:01.336871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.336869 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d15b25d-07cf-4e6c-8446-60915a92b92e" containerName="istio-proxy" Oct 30 19:56:01.337053 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.336902 5091 cpu_manager.go:414] "containerMap: removing container" podUID="292828eb-f86c-47d1-8113-9213119a3afb" containerName="revoke-keys" Oct 30 19:56:01.337053 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.336911 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="292828eb-f86c-47d1-8113-9213119a3afb" containerName="revoke-keys" Oct 30 19:56:01.337121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.337101 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d15b25d-07cf-4e6c-8446-60915a92b92e" containerName="istio-proxy" Oct 30 19:56:01.337192 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.337132 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="292828eb-f86c-47d1-8113-9213119a3afb" containerName="revoke-keys" Oct 30 19:56:01.342591 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.342572 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" Oct 30 19:56:01.351573 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.351543 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx"] Oct 30 19:56:01.352205 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.352181 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:56:01.352526 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.352508 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:56:01.382272 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.382242 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km48s\" (UniqueName: \"kubernetes.io/projected/96550e8d-b40a-43f1-9942-9eb094b9293d-kube-api-access-km48s\") pod \"maas-api-revoke-sub-e183ab3d-gvklx\" (UID: \"96550e8d-b40a-43f1-9942-9eb094b9293d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" Oct 30 19:56:01.382272 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.382276 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96550e8d-b40a-43f1-9942-9eb094b9293d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e183ab3d-gvklx\" (UID: \"96550e8d-b40a-43f1-9942-9eb094b9293d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" Oct 30 19:56:01.382559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.382529 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96550e8d-b40a-43f1-9942-9eb094b9293d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e183ab3d-gvklx\" (UID: \"96550e8d-b40a-43f1-9942-9eb094b9293d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" Oct 30 19:56:01.483342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.483245 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96550e8d-b40a-43f1-9942-9eb094b9293d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e183ab3d-gvklx\" (UID: \"96550e8d-b40a-43f1-9942-9eb094b9293d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" Oct 30 19:56:01.483519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.483468 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km48s\" (UniqueName: \"kubernetes.io/projected/96550e8d-b40a-43f1-9942-9eb094b9293d-kube-api-access-km48s\") pod \"maas-api-revoke-sub-e183ab3d-gvklx\" (UID: \"96550e8d-b40a-43f1-9942-9eb094b9293d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" Oct 30 19:56:01.483519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.483495 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96550e8d-b40a-43f1-9942-9eb094b9293d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e183ab3d-gvklx\" (UID: \"96550e8d-b40a-43f1-9942-9eb094b9293d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" Oct 30 19:56:01.484043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.484020 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96550e8d-b40a-43f1-9942-9eb094b9293d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e183ab3d-gvklx\" (UID: \"96550e8d-b40a-43f1-9942-9eb094b9293d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" Oct 30 19:56:01.485874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.485857 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96550e8d-b40a-43f1-9942-9eb094b9293d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e183ab3d-gvklx\" (UID: \"96550e8d-b40a-43f1-9942-9eb094b9293d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" Oct 30 19:56:01.501324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.501294 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-km48s\" (UniqueName: \"kubernetes.io/projected/96550e8d-b40a-43f1-9942-9eb094b9293d-kube-api-access-km48s\") pod \"maas-api-revoke-sub-e183ab3d-gvklx\" (UID: \"96550e8d-b40a-43f1-9942-9eb094b9293d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" Oct 30 19:56:01.662664 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.662629 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" Oct 30 19:56:01.774056 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.774024 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl"] Oct 30 19:56:01.774820 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.774796 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8d15b25d-07cf-4e6c-8446-60915a92b92e" containerName="istio-proxy" Oct 30 19:56:01.774820 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.774819 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d15b25d-07cf-4e6c-8446-60915a92b92e" containerName="istio-proxy" Oct 30 19:56:01.775004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.774995 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d15b25d-07cf-4e6c-8446-60915a92b92e" containerName="istio-proxy" Oct 30 19:56:01.779958 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.779904 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" Oct 30 19:56:01.794196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.792273 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl"] Oct 30 19:56:01.890135 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.890091 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrdpr\" (UniqueName: \"kubernetes.io/projected/c48f98ff-5a17-411b-bc08-328bf002e862-kube-api-access-hrdpr\") pod \"maas-api-revoke-sub-e0f32d51-s5hhl\" (UID: \"c48f98ff-5a17-411b-bc08-328bf002e862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" Oct 30 19:56:01.890334 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.890256 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48f98ff-5a17-411b-bc08-328bf002e862-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f32d51-s5hhl\" (UID: \"c48f98ff-5a17-411b-bc08-328bf002e862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" Oct 30 19:56:01.890461 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.890439 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48f98ff-5a17-411b-bc08-328bf002e862-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f32d51-s5hhl\" (UID: \"c48f98ff-5a17-411b-bc08-328bf002e862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" Oct 30 19:56:01.974377 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:56:01.974346 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96550e8d_b40a_43f1_9942_9eb094b9293d.slice/crio-48463e075ce8899747002377c154f69258e1cb920e95336d5d11e31d2a6190fd WatchSource:0}: Error finding container 48463e075ce8899747002377c154f69258e1cb920e95336d5d11e31d2a6190fd: Status 404 returned error can't find the container with id 48463e075ce8899747002377c154f69258e1cb920e95336d5d11e31d2a6190fd Oct 30 19:56:01.975366 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.975343 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx"] Oct 30 19:56:01.992026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.991947 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48f98ff-5a17-411b-bc08-328bf002e862-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f32d51-s5hhl\" (UID: \"c48f98ff-5a17-411b-bc08-328bf002e862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" Oct 30 19:56:01.992179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.992090 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrdpr\" (UniqueName: \"kubernetes.io/projected/c48f98ff-5a17-411b-bc08-328bf002e862-kube-api-access-hrdpr\") pod \"maas-api-revoke-sub-e0f32d51-s5hhl\" (UID: \"c48f98ff-5a17-411b-bc08-328bf002e862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" Oct 30 19:56:01.992261 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.992223 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48f98ff-5a17-411b-bc08-328bf002e862-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f32d51-s5hhl\" (UID: \"c48f98ff-5a17-411b-bc08-328bf002e862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" Oct 30 19:56:01.992944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.992922 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48f98ff-5a17-411b-bc08-328bf002e862-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0f32d51-s5hhl\" (UID: \"c48f98ff-5a17-411b-bc08-328bf002e862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" Oct 30 19:56:01.994484 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:01.994463 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48f98ff-5a17-411b-bc08-328bf002e862-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0f32d51-s5hhl\" (UID: \"c48f98ff-5a17-411b-bc08-328bf002e862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" Oct 30 19:56:02.014639 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:02.014610 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrdpr\" (UniqueName: \"kubernetes.io/projected/c48f98ff-5a17-411b-bc08-328bf002e862-kube-api-access-hrdpr\") pod \"maas-api-revoke-sub-e0f32d51-s5hhl\" (UID: \"c48f98ff-5a17-411b-bc08-328bf002e862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" Oct 30 19:56:02.109341 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:02.109312 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" Oct 30 19:56:02.353636 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:02.353593 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" event={"ID":"96550e8d-b40a-43f1-9942-9eb094b9293d","Type":"ContainerStarted","Data":"48463e075ce8899747002377c154f69258e1cb920e95336d5d11e31d2a6190fd"} Oct 30 19:56:02.433812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:02.432735 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl"] Oct 30 19:56:02.434774 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:56:02.434738 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48f98ff_5a17_411b_bc08_328bf002e862.slice/crio-8c6c9c756c67dfa216790b6931c553b9d1500759e75dfb3cb3e0d64030abb6ac WatchSource:0}: Error finding container 8c6c9c756c67dfa216790b6931c553b9d1500759e75dfb3cb3e0d64030abb6ac: Status 404 returned error can't find the container with id 8c6c9c756c67dfa216790b6931c553b9d1500759e75dfb3cb3e0d64030abb6ac Oct 30 19:56:02.436494 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:02.435198 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:03.366343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.366305 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" event={"ID":"c48f98ff-5a17-411b-bc08-328bf002e862","Type":"ContainerStarted","Data":"d1df54649cb655f0323f9124dbb981b0348aefd0b3a7470cc81e0e8a39419663"} Oct 30 19:56:03.367031 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.366373 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" event={"ID":"c48f98ff-5a17-411b-bc08-328bf002e862","Type":"ContainerStarted","Data":"8c6c9c756c67dfa216790b6931c553b9d1500759e75dfb3cb3e0d64030abb6ac"} Oct 30 19:56:03.370504 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.370478 5091 generic.go:356] "Generic (PLEG): container finished" podID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerID="7b35a104a2c0206cddd13cdd5bad53831309527bfedf7c0908b8201c04b763a6" exitCode=0 Oct 30 19:56:03.370630 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.370543 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" event={"ID":"8b12e993-577a-4dc7-94a0-41c65c7b1fe3","Type":"ContainerDied","Data":"7b35a104a2c0206cddd13cdd5bad53831309527bfedf7c0908b8201c04b763a6"} Oct 30 19:56:03.370630 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.370598 5091 scope.go:122] "RemoveContainer" containerID="c213ab747484af8ee780f4f5f1dc2e460f261e47c573f32380995d70f42cfd30" Oct 30 19:56:03.373566 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.373544 5091 generic.go:356] "Generic (PLEG): container finished" podID="96550e8d-b40a-43f1-9942-9eb094b9293d" containerID="7d3ea2d146498184719c24765b00ea817d004f3a1e135f2b7a10de86bad4c427" exitCode=0 Oct 30 19:56:03.373679 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.373584 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" event={"ID":"96550e8d-b40a-43f1-9942-9eb094b9293d","Type":"ContainerDied","Data":"7d3ea2d146498184719c24765b00ea817d004f3a1e135f2b7a10de86bad4c427"} Oct 30 19:56:03.401713 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.401657 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" podStartSLOduration=2.401643746 podStartE2EDuration="2.401643746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:03.400223035 +0000 UTC m=+1483.095430991" watchObservedRunningTime="2026-10-30 19:56:03.401643746 +0000 UTC m=+1483.096851703" Oct 30 19:56:03.425849 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.425810 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:56:03.519044 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.519008 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-workload-certs\") pod \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " Oct 30 19:56:03.519204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.519124 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-envoy\") pod \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " Oct 30 19:56:03.519204 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.519174 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-kube-api-access-x5fph\" (UniqueName: \"kubernetes.io/projected/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-kube-api-access-x5fph\") pod \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " Oct 30 19:56:03.519357 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.519248 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-workload-socket\") pod \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " Oct 30 19:56:03.519420 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.519357 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istiod-ca-cert\") pod \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " Oct 30 19:56:03.519420 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.519360 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-workload-certs" pod "8b12e993-577a-4dc7-94a0-41c65c7b1fe3" (UID: "8b12e993-577a-4dc7-94a0-41c65c7b1fe3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:03.519420 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.519392 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-credential-socket\") pod \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " Oct 30 19:56:03.519592 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.519453 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-podinfo\") pod \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " Oct 30 19:56:03.519592 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.519510 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-data\") pod \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " Oct 30 19:56:03.519592 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.519574 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-token\" (UniqueName: \"kubernetes.io/projected/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-token\") pod \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\" (UID: \"8b12e993-577a-4dc7-94a0-41c65c7b1fe3\") " Oct 30 19:56:03.519848 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.519615 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-workload-socket" pod "8b12e993-577a-4dc7-94a0-41c65c7b1fe3" (UID: "8b12e993-577a-4dc7-94a0-41c65c7b1fe3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:03.519848 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.519824 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-credential-socket" pod "8b12e993-577a-4dc7-94a0-41c65c7b1fe3" (UID: "8b12e993-577a-4dc7-94a0-41c65c7b1fe3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:03.520391 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.520365 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:03.520514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.520397 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:03.520514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.520414 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:03.520811 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.520780 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-data" pod "8b12e993-577a-4dc7-94a0-41c65c7b1fe3" (UID: "8b12e993-577a-4dc7-94a0-41c65c7b1fe3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:03.521517 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.521491 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istiod-ca-cert" pod "8b12e993-577a-4dc7-94a0-41c65c7b1fe3" (UID: "8b12e993-577a-4dc7-94a0-41c65c7b1fe3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:03.523405 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.523380 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-kube-api-access-x5fph" pod "8b12e993-577a-4dc7-94a0-41c65c7b1fe3" (UID: "8b12e993-577a-4dc7-94a0-41c65c7b1fe3"). InnerVolumeSpecName "kube-api-access-x5fph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:03.524704 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.524679 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-envoy" pod "8b12e993-577a-4dc7-94a0-41c65c7b1fe3" (UID: "8b12e993-577a-4dc7-94a0-41c65c7b1fe3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:03.524704 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.524689 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-podinfo" pod "8b12e993-577a-4dc7-94a0-41c65c7b1fe3" (UID: "8b12e993-577a-4dc7-94a0-41c65c7b1fe3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:56:03.524836 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.524722 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-token" pod "8b12e993-577a-4dc7-94a0-41c65c7b1fe3" (UID: "8b12e993-577a-4dc7-94a0-41c65c7b1fe3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:03.621558 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.621522 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:03.621558 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.621550 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:03.621558 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.621560 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:03.621772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.621570 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5fph\" (UniqueName: \"kubernetes.io/projected/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-kube-api-access-x5fph\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:03.621772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.621581 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:03.621772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.621589 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b12e993-577a-4dc7-94a0-41c65c7b1fe3-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:03.700180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.697500 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:03.891638 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.891566 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h"] Oct 30 19:56:03.892263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.892246 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerName="istio-proxy" Oct 30 19:56:03.892323 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.892264 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerName="istio-proxy" Oct 30 19:56:03.892323 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.892289 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerName="istio-proxy" Oct 30 19:56:03.892323 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.892294 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerName="istio-proxy" Oct 30 19:56:03.892436 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.892420 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerName="istio-proxy" Oct 30 19:56:03.892436 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.892431 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" containerName="istio-proxy" Oct 30 19:56:03.896413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.896396 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" Oct 30 19:56:03.907590 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.907561 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h"] Oct 30 19:56:03.925404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.925373 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cb4162-feeb-4fec-be93-e02fdd499706-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h\" (UID: \"37cb4162-feeb-4fec-be93-e02fdd499706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" Oct 30 19:56:03.925544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.925517 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm5bc\" (UniqueName: \"kubernetes.io/projected/37cb4162-feeb-4fec-be93-e02fdd499706-kube-api-access-zm5bc\") pod \"maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h\" (UID: \"37cb4162-feeb-4fec-be93-e02fdd499706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" Oct 30 19:56:03.925665 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:03.925640 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cb4162-feeb-4fec-be93-e02fdd499706-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h\" (UID: \"37cb4162-feeb-4fec-be93-e02fdd499706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" Oct 30 19:56:04.027243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.027201 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cb4162-feeb-4fec-be93-e02fdd499706-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h\" (UID: \"37cb4162-feeb-4fec-be93-e02fdd499706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" Oct 30 19:56:04.027478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.027283 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm5bc\" (UniqueName: \"kubernetes.io/projected/37cb4162-feeb-4fec-be93-e02fdd499706-kube-api-access-zm5bc\") pod \"maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h\" (UID: \"37cb4162-feeb-4fec-be93-e02fdd499706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" Oct 30 19:56:04.027478 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.027319 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cb4162-feeb-4fec-be93-e02fdd499706-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h\" (UID: \"37cb4162-feeb-4fec-be93-e02fdd499706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" Oct 30 19:56:04.027962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.027940 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cb4162-feeb-4fec-be93-e02fdd499706-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h\" (UID: \"37cb4162-feeb-4fec-be93-e02fdd499706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" Oct 30 19:56:04.029681 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.029660 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cb4162-feeb-4fec-be93-e02fdd499706-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h\" (UID: \"37cb4162-feeb-4fec-be93-e02fdd499706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" Oct 30 19:56:04.042922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.042898 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm5bc\" (UniqueName: \"kubernetes.io/projected/37cb4162-feeb-4fec-be93-e02fdd499706-kube-api-access-zm5bc\") pod \"maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h\" (UID: \"37cb4162-feeb-4fec-be93-e02fdd499706\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" Oct 30 19:56:04.217647 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.217561 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" Oct 30 19:56:04.383658 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.383623 5091 generic.go:356] "Generic (PLEG): container finished" podID="c48f98ff-5a17-411b-bc08-328bf002e862" containerID="d1df54649cb655f0323f9124dbb981b0348aefd0b3a7470cc81e0e8a39419663" exitCode=0 Oct 30 19:56:04.384029 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.383683 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" event={"ID":"c48f98ff-5a17-411b-bc08-328bf002e862","Type":"ContainerDied","Data":"d1df54649cb655f0323f9124dbb981b0348aefd0b3a7470cc81e0e8a39419663"} Oct 30 19:56:04.385443 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.385417 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" Oct 30 19:56:04.385443 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.385432 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l" event={"ID":"8b12e993-577a-4dc7-94a0-41c65c7b1fe3","Type":"ContainerDied","Data":"9cc3acb510662984ac1697d45747a8f75b88418e579f0053d37f0b5d5a92d877"} Oct 30 19:56:04.385617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.385483 5091 scope.go:122] "RemoveContainer" containerID="7b35a104a2c0206cddd13cdd5bad53831309527bfedf7c0908b8201c04b763a6" Oct 30 19:56:04.447626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.447592 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l"] Oct 30 19:56:04.452184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.452116 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-b17e94-openshift-default-666cf59769-nb98l"] Oct 30 19:56:04.521758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.521731 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h"] Oct 30 19:56:04.614748 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.614714 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:04.637892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.637867 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" Oct 30 19:56:04.745053 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.745018 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l"] Oct 30 19:56:04.755792 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.755755 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96550e8d-b40a-43f1-9942-9eb094b9293d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96550e8d-b40a-43f1-9942-9eb094b9293d-maas-api-openshift-ca\") pod \"96550e8d-b40a-43f1-9942-9eb094b9293d\" (UID: \"96550e8d-b40a-43f1-9942-9eb094b9293d\") " Oct 30 19:56:04.755993 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.755886 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96550e8d-b40a-43f1-9942-9eb094b9293d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96550e8d-b40a-43f1-9942-9eb094b9293d-maas-api-xks-ca\") pod \"96550e8d-b40a-43f1-9942-9eb094b9293d\" (UID: \"96550e8d-b40a-43f1-9942-9eb094b9293d\") " Oct 30 19:56:04.756107 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.756078 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96550e8d-b40a-43f1-9942-9eb094b9293d-kube-api-access-km48s\" (UniqueName: \"kubernetes.io/projected/96550e8d-b40a-43f1-9942-9eb094b9293d-kube-api-access-km48s\") pod \"96550e8d-b40a-43f1-9942-9eb094b9293d\" (UID: \"96550e8d-b40a-43f1-9942-9eb094b9293d\") " Oct 30 19:56:04.757314 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.757285 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96550e8d-b40a-43f1-9942-9eb094b9293d-maas-api-openshift-ca" pod "96550e8d-b40a-43f1-9942-9eb094b9293d" (UID: "96550e8d-b40a-43f1-9942-9eb094b9293d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:04.758292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.758269 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb"] Oct 30 19:56:04.759562 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.759530 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96550e8d-b40a-43f1-9942-9eb094b9293d-kube-api-access-km48s" pod "96550e8d-b40a-43f1-9942-9eb094b9293d" (UID: "96550e8d-b40a-43f1-9942-9eb094b9293d"). InnerVolumeSpecName "kube-api-access-km48s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:04.761582 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.761549 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96550e8d-b40a-43f1-9942-9eb094b9293d-maas-api-xks-ca" pod "96550e8d-b40a-43f1-9942-9eb094b9293d" (UID: "96550e8d-b40a-43f1-9942-9eb094b9293d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:04.857860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.857787 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km48s\" (UniqueName: \"kubernetes.io/projected/96550e8d-b40a-43f1-9942-9eb094b9293d-kube-api-access-km48s\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:04.857860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.857826 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96550e8d-b40a-43f1-9942-9eb094b9293d-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:04.857860 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.857844 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96550e8d-b40a-43f1-9942-9eb094b9293d-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:04.901284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:04.901247 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:05.309507 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.308754 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:05.395967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.395932 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" event={"ID":"37cb4162-feeb-4fec-be93-e02fdd499706","Type":"ContainerStarted","Data":"47f9eab5d3887ce0e970a380ba6d0dd44fe9b583b5c1356255d4f64e3842baed"} Oct 30 19:56:05.396411 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.395989 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" event={"ID":"37cb4162-feeb-4fec-be93-e02fdd499706","Type":"ContainerStarted","Data":"01c798dc04dc0809c9ae631c0a90846205d5dc44d093802e46abd52793c37af8"} Oct 30 19:56:05.399048 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.399024 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" Oct 30 19:56:05.399198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.399061 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx" event={"ID":"96550e8d-b40a-43f1-9942-9eb094b9293d","Type":"ContainerDied","Data":"48463e075ce8899747002377c154f69258e1cb920e95336d5d11e31d2a6190fd"} Oct 30 19:56:05.399198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.399106 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48463e075ce8899747002377c154f69258e1cb920e95336d5d11e31d2a6190fd" Oct 30 19:56:05.424917 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.424887 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz"] Oct 30 19:56:05.437101 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.436476 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-a29f33-czsnz"] Oct 30 19:56:05.443514 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.443451 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" podStartSLOduration=2.443433165 podStartE2EDuration="2.443433165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:05.435948383 +0000 UTC m=+1485.131156428" watchObservedRunningTime="2026-10-30 19:56:05.443433165 +0000 UTC m=+1485.138641124" Oct 30 19:56:05.520243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.520125 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:05.596249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.596218 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" Oct 30 19:56:05.691449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.691425 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c48f98ff-5a17-411b-bc08-328bf002e862-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48f98ff-5a17-411b-bc08-328bf002e862-maas-api-xks-ca\") pod \"c48f98ff-5a17-411b-bc08-328bf002e862\" (UID: \"c48f98ff-5a17-411b-bc08-328bf002e862\") " Oct 30 19:56:05.691554 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.691529 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c48f98ff-5a17-411b-bc08-328bf002e862-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48f98ff-5a17-411b-bc08-328bf002e862-maas-api-openshift-ca\") pod \"c48f98ff-5a17-411b-bc08-328bf002e862\" (UID: \"c48f98ff-5a17-411b-bc08-328bf002e862\") " Oct 30 19:56:05.691642 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.691631 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c48f98ff-5a17-411b-bc08-328bf002e862-kube-api-access-hrdpr\" (UniqueName: \"kubernetes.io/projected/c48f98ff-5a17-411b-bc08-328bf002e862-kube-api-access-hrdpr\") pod \"c48f98ff-5a17-411b-bc08-328bf002e862\" (UID: \"c48f98ff-5a17-411b-bc08-328bf002e862\") " Oct 30 19:56:05.691866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.691840 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c48f98ff-5a17-411b-bc08-328bf002e862-maas-api-openshift-ca" pod "c48f98ff-5a17-411b-bc08-328bf002e862" (UID: "c48f98ff-5a17-411b-bc08-328bf002e862"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:05.692612 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.692094 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c48f98ff-5a17-411b-bc08-328bf002e862-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:05.694284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.694262 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c48f98ff-5a17-411b-bc08-328bf002e862-maas-api-xks-ca" pod "c48f98ff-5a17-411b-bc08-328bf002e862" (UID: "c48f98ff-5a17-411b-bc08-328bf002e862"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:05.694374 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.694313 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c48f98ff-5a17-411b-bc08-328bf002e862-kube-api-access-hrdpr" pod "c48f98ff-5a17-411b-bc08-328bf002e862" (UID: "c48f98ff-5a17-411b-bc08-328bf002e862"). InnerVolumeSpecName "kube-api-access-hrdpr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:05.793289 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.793255 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrdpr\" (UniqueName: \"kubernetes.io/projected/c48f98ff-5a17-411b-bc08-328bf002e862-kube-api-access-hrdpr\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:05.793289 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:05.793285 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c48f98ff-5a17-411b-bc08-328bf002e862-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:06.015626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:06.015579 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:06.376242 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:06.372491 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:06.414869 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:06.414843 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" Oct 30 19:56:06.415304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:06.414878 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl" event={"ID":"c48f98ff-5a17-411b-bc08-328bf002e862","Type":"ContainerDied","Data":"8c6c9c756c67dfa216790b6931c553b9d1500759e75dfb3cb3e0d64030abb6ac"} Oct 30 19:56:06.415304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:06.414962 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c6c9c756c67dfa216790b6931c553b9d1500759e75dfb3cb3e0d64030abb6ac" Oct 30 19:56:06.417007 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:06.416974 5091 generic.go:356] "Generic (PLEG): container finished" podID="37cb4162-feeb-4fec-be93-e02fdd499706" containerID="47f9eab5d3887ce0e970a380ba6d0dd44fe9b583b5c1356255d4f64e3842baed" exitCode=0 Oct 30 19:56:06.417183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:06.417069 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" event={"ID":"37cb4162-feeb-4fec-be93-e02fdd499706","Type":"ContainerDied","Data":"47f9eab5d3887ce0e970a380ba6d0dd44fe9b583b5c1356255d4f64e3842baed"} Oct 30 19:56:06.443515 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:06.443491 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="292828eb-f86c-47d1-8113-9213119a3afb" path="/var/lib/kubelet/pods/292828eb-f86c-47d1-8113-9213119a3afb/volumes" Oct 30 19:56:06.443986 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:06.443971 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b12e993-577a-4dc7-94a0-41c65c7b1fe3" path="/var/lib/kubelet/pods/8b12e993-577a-4dc7-94a0-41c65c7b1fe3/volumes" Oct 30 19:56:06.767483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:06.767447 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:07.380813 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:07.380776 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:07.581380 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:07.581354 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" Oct 30 19:56:07.720003 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:07.719926 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37cb4162-feeb-4fec-be93-e02fdd499706-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cb4162-feeb-4fec-be93-e02fdd499706-maas-api-openshift-ca\") pod \"37cb4162-feeb-4fec-be93-e02fdd499706\" (UID: \"37cb4162-feeb-4fec-be93-e02fdd499706\") " Oct 30 19:56:07.720218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:07.720068 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37cb4162-feeb-4fec-be93-e02fdd499706-kube-api-access-zm5bc\" (UniqueName: \"kubernetes.io/projected/37cb4162-feeb-4fec-be93-e02fdd499706-kube-api-access-zm5bc\") pod \"37cb4162-feeb-4fec-be93-e02fdd499706\" (UID: \"37cb4162-feeb-4fec-be93-e02fdd499706\") " Oct 30 19:56:07.720218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:07.720193 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37cb4162-feeb-4fec-be93-e02fdd499706-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cb4162-feeb-4fec-be93-e02fdd499706-maas-api-xks-ca\") pod \"37cb4162-feeb-4fec-be93-e02fdd499706\" (UID: \"37cb4162-feeb-4fec-be93-e02fdd499706\") " Oct 30 19:56:07.720384 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:07.720359 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37cb4162-feeb-4fec-be93-e02fdd499706-maas-api-openshift-ca" pod "37cb4162-feeb-4fec-be93-e02fdd499706" (UID: "37cb4162-feeb-4fec-be93-e02fdd499706"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:07.720676 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:07.720656 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37cb4162-feeb-4fec-be93-e02fdd499706-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:07.722364 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:07.722342 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37cb4162-feeb-4fec-be93-e02fdd499706-kube-api-access-zm5bc" pod "37cb4162-feeb-4fec-be93-e02fdd499706" (UID: "37cb4162-feeb-4fec-be93-e02fdd499706"). InnerVolumeSpecName "kube-api-access-zm5bc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:07.722364 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:07.722354 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37cb4162-feeb-4fec-be93-e02fdd499706-maas-api-xks-ca" pod "37cb4162-feeb-4fec-be93-e02fdd499706" (UID: "37cb4162-feeb-4fec-be93-e02fdd499706"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:07.821705 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:07.821664 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37cb4162-feeb-4fec-be93-e02fdd499706-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:07.821705 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:07.821698 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zm5bc\" (UniqueName: \"kubernetes.io/projected/37cb4162-feeb-4fec-be93-e02fdd499706-kube-api-access-zm5bc\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:08.094660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.094615 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:08.436657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.436589 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" Oct 30 19:56:08.442605 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.442574 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h" event={"ID":"37cb4162-feeb-4fec-be93-e02fdd499706","Type":"ContainerDied","Data":"01c798dc04dc0809c9ae631c0a90846205d5dc44d093802e46abd52793c37af8"} Oct 30 19:56:08.442729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.442628 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01c798dc04dc0809c9ae631c0a90846205d5dc44d093802e46abd52793c37af8" Oct 30 19:56:08.519498 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.519459 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz"] Oct 30 19:56:08.520329 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.520305 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c48f98ff-5a17-411b-bc08-328bf002e862" containerName="revoke-keys" Oct 30 19:56:08.520329 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.520330 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c48f98ff-5a17-411b-bc08-328bf002e862" containerName="revoke-keys" Oct 30 19:56:08.520506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.520359 5091 cpu_manager.go:414] "containerMap: removing container" podUID="37cb4162-feeb-4fec-be93-e02fdd499706" containerName="revoke-keys" Oct 30 19:56:08.520506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.520364 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37cb4162-feeb-4fec-be93-e02fdd499706" containerName="revoke-keys" Oct 30 19:56:08.520506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.520408 5091 cpu_manager.go:414] "containerMap: removing container" podUID="96550e8d-b40a-43f1-9942-9eb094b9293d" containerName="revoke-keys" Oct 30 19:56:08.520506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.520416 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96550e8d-b40a-43f1-9942-9eb094b9293d" containerName="revoke-keys" Oct 30 19:56:08.520731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.520531 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="96550e8d-b40a-43f1-9942-9eb094b9293d" containerName="revoke-keys" Oct 30 19:56:08.520731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.520550 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c48f98ff-5a17-411b-bc08-328bf002e862" containerName="revoke-keys" Oct 30 19:56:08.520731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.520570 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="37cb4162-feeb-4fec-be93-e02fdd499706" containerName="revoke-keys" Oct 30 19:56:08.524415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.524398 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.538470 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.538444 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-c1768b-openshift-default-dockercfg-csb66\"" Oct 30 19:56:08.562275 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.558525 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz"] Oct 30 19:56:08.601364 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.601327 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:08.632567 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.632531 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-envoy\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.632737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.632608 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-token\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.632737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.632669 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-credential-socket\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.632891 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.632798 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bcf3894-d14f-4ee3-a294-63daff87f18a-istiod-ca-cert\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.632891 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.632863 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw9k5\" (UniqueName: \"kubernetes.io/projected/7bcf3894-d14f-4ee3-a294-63daff87f18a-kube-api-access-rw9k5\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.632975 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.632894 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-podinfo\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.633033 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.633014 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-data\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.633079 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.633043 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-workload-socket\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.633114 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.633084 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-workload-certs\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.746218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.735166 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-envoy\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.746218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.735218 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-token\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.746218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.735286 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-credential-socket\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.746218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.735377 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bcf3894-d14f-4ee3-a294-63daff87f18a-istiod-ca-cert\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.746218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.735445 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw9k5\" (UniqueName: \"kubernetes.io/projected/7bcf3894-d14f-4ee3-a294-63daff87f18a-kube-api-access-rw9k5\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.746218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.735483 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-podinfo\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.746218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.735581 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-data\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.746218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.735618 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-workload-socket\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.746218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.735645 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-workload-certs\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.746218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.736165 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-workload-certs\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.746218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.737813 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bcf3894-d14f-4ee3-a294-63daff87f18a-istiod-ca-cert\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.746218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.738058 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-data\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.746218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.738738 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-workload-socket\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.746218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.739876 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-credential-socket\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.782166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.769035 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-token\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.782166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.769907 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-podinfo\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.782166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.770237 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-envoy\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.784168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.782685 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:08.797164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.795509 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw9k5\" (UniqueName: \"kubernetes.io/projected/7bcf3894-d14f-4ee3-a294-63daff87f18a-kube-api-access-rw9k5\") pod \"e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:08.855175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:08.847581 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:09.179798 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:09.179772 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz"] Oct 30 19:56:09.181919 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:56:09.181894 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bcf3894_d14f_4ee3_a294_63daff87f18a.slice/crio-918527cfc3becb6038a87147aabaa704382ba93cd054a9b3ef2467189f81b345 WatchSource:0}: Error finding container 918527cfc3becb6038a87147aabaa704382ba93cd054a9b3ef2467189f81b345: Status 404 returned error can't find the container with id 918527cfc3becb6038a87147aabaa704382ba93cd054a9b3ef2467189f81b345 Oct 30 19:56:09.184401 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:09.184370 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:09.184504 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:09.184433 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:09.184504 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:09.184466 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:09.446994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:09.446890 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" event={"ID":"7bcf3894-d14f-4ee3-a294-63daff87f18a","Type":"ContainerStarted","Data":"6b71d3f026897317d1c110c31a1297a30e18c867e421af0910c645e42208eab3"} Oct 30 19:56:09.446994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:09.446938 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" event={"ID":"7bcf3894-d14f-4ee3-a294-63daff87f18a","Type":"ContainerStarted","Data":"918527cfc3becb6038a87147aabaa704382ba93cd054a9b3ef2467189f81b345"} Oct 30 19:56:09.479452 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:09.479396 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" podStartSLOduration=1.479383444 podStartE2EDuration="1.479383444s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:09.473847063 +0000 UTC m=+1489.169055022" watchObservedRunningTime="2026-10-30 19:56:09.479383444 +0000 UTC m=+1489.174591401" Oct 30 19:56:09.650193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:09.647255 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:09.745929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:09.745835 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" podUID="3f73a998-807d-4652-8c3a-28b4f6d42390" containerName="payload-pre-processing" containerID="cri-o://3d2859d3610d8ff67d756156c6fc58980bcce43ead1fccf8c327e649a8709b99" gracePeriod=25 Oct 30 19:56:09.759679 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:09.759640 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" podUID="02feb7af-17b1-472a-81d4-a5265781f30c" containerName="payload-processing" containerID="cri-o://421151e16a377defdd053f0c132d7de7a43d0076d12e00ead83faf3fb8e32d1d" gracePeriod=25 Oct 30 19:56:09.848731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:09.848704 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:10.054325 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.054296 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" Oct 30 19:56:10.087540 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.086708 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" Oct 30 19:56:10.156489 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.156457 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f73a998-807d-4652-8c3a-28b4f6d42390-kube-api-access-p5qmq\" (UniqueName: \"kubernetes.io/projected/3f73a998-807d-4652-8c3a-28b4f6d42390-kube-api-access-p5qmq\") pod \"3f73a998-807d-4652-8c3a-28b4f6d42390\" (UID: \"3f73a998-807d-4652-8c3a-28b4f6d42390\") " Oct 30 19:56:10.156667 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.156509 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f73a998-807d-4652-8c3a-28b4f6d42390-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f73a998-807d-4652-8c3a-28b4f6d42390-plugins-config-volume\") pod \"3f73a998-807d-4652-8c3a-28b4f6d42390\" (UID: \"3f73a998-807d-4652-8c3a-28b4f6d42390\") " Oct 30 19:56:10.157461 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.157373 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f73a998-807d-4652-8c3a-28b4f6d42390-plugins-config-volume" pod "3f73a998-807d-4652-8c3a-28b4f6d42390" (UID: "3f73a998-807d-4652-8c3a-28b4f6d42390"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:10.159233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.159209 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f73a998-807d-4652-8c3a-28b4f6d42390-kube-api-access-p5qmq" pod "3f73a998-807d-4652-8c3a-28b4f6d42390" (UID: "3f73a998-807d-4652-8c3a-28b4f6d42390"). InnerVolumeSpecName "kube-api-access-p5qmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:10.258422 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.258343 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02feb7af-17b1-472a-81d4-a5265781f30c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02feb7af-17b1-472a-81d4-a5265781f30c-plugins-config-volume\") pod \"02feb7af-17b1-472a-81d4-a5265781f30c\" (UID: \"02feb7af-17b1-472a-81d4-a5265781f30c\") " Oct 30 19:56:10.258571 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.258450 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02feb7af-17b1-472a-81d4-a5265781f30c-kube-api-access-rfp57\" (UniqueName: \"kubernetes.io/projected/02feb7af-17b1-472a-81d4-a5265781f30c-kube-api-access-rfp57\") pod \"02feb7af-17b1-472a-81d4-a5265781f30c\" (UID: \"02feb7af-17b1-472a-81d4-a5265781f30c\") " Oct 30 19:56:10.258766 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.258738 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02feb7af-17b1-472a-81d4-a5265781f30c-plugins-config-volume" pod "02feb7af-17b1-472a-81d4-a5265781f30c" (UID: "02feb7af-17b1-472a-81d4-a5265781f30c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:10.259140 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.259115 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5qmq\" (UniqueName: \"kubernetes.io/projected/3f73a998-807d-4652-8c3a-28b4f6d42390-kube-api-access-p5qmq\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:10.259140 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.259159 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02feb7af-17b1-472a-81d4-a5265781f30c-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:10.259330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.259176 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f73a998-807d-4652-8c3a-28b4f6d42390-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:10.260658 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.260634 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02feb7af-17b1-472a-81d4-a5265781f30c-kube-api-access-rfp57" pod "02feb7af-17b1-472a-81d4-a5265781f30c" (UID: "02feb7af-17b1-472a-81d4-a5265781f30c"). InnerVolumeSpecName "kube-api-access-rfp57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:10.360387 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.360345 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfp57\" (UniqueName: \"kubernetes.io/projected/02feb7af-17b1-472a-81d4-a5265781f30c-kube-api-access-rfp57\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:10.460002 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.459973 5091 generic.go:356] "Generic (PLEG): container finished" podID="02feb7af-17b1-472a-81d4-a5265781f30c" containerID="421151e16a377defdd053f0c132d7de7a43d0076d12e00ead83faf3fb8e32d1d" exitCode=0 Oct 30 19:56:10.460141 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.460044 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" Oct 30 19:56:10.460141 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.460059 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" event={"ID":"02feb7af-17b1-472a-81d4-a5265781f30c","Type":"ContainerDied","Data":"421151e16a377defdd053f0c132d7de7a43d0076d12e00ead83faf3fb8e32d1d"} Oct 30 19:56:10.460141 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.460122 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb" event={"ID":"02feb7af-17b1-472a-81d4-a5265781f30c","Type":"ContainerDied","Data":"d3f2a44db3d61cd6c8d1ee26ea65cccfb3727d0995bdb86d659b8758770db9fc"} Oct 30 19:56:10.460360 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.460279 5091 scope.go:122] "RemoveContainer" containerID="421151e16a377defdd053f0c132d7de7a43d0076d12e00ead83faf3fb8e32d1d" Oct 30 19:56:10.463004 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.462978 5091 generic.go:356] "Generic (PLEG): container finished" podID="3f73a998-807d-4652-8c3a-28b4f6d42390" containerID="3d2859d3610d8ff67d756156c6fc58980bcce43ead1fccf8c327e649a8709b99" exitCode=0 Oct 30 19:56:10.463100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.463031 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" Oct 30 19:56:10.463264 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.463109 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" event={"ID":"3f73a998-807d-4652-8c3a-28b4f6d42390","Type":"ContainerDied","Data":"3d2859d3610d8ff67d756156c6fc58980bcce43ead1fccf8c327e649a8709b99"} Oct 30 19:56:10.463264 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.463198 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l" event={"ID":"3f73a998-807d-4652-8c3a-28b4f6d42390","Type":"ContainerDied","Data":"e9ba0da1a7b5fe9c27d2c2bafbffef72462ae5be9dc6aa446abf0910f8da1231"} Oct 30 19:56:10.477587 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.477567 5091 scope.go:122] "RemoveContainer" containerID="421151e16a377defdd053f0c132d7de7a43d0076d12e00ead83faf3fb8e32d1d" Oct 30 19:56:10.477897 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:56:10.477872 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421151e16a377defdd053f0c132d7de7a43d0076d12e00ead83faf3fb8e32d1d\": container with ID starting with 421151e16a377defdd053f0c132d7de7a43d0076d12e00ead83faf3fb8e32d1d not found: ID does not exist" containerID="421151e16a377defdd053f0c132d7de7a43d0076d12e00ead83faf3fb8e32d1d" Oct 30 19:56:10.478025 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.477905 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421151e16a377defdd053f0c132d7de7a43d0076d12e00ead83faf3fb8e32d1d"} err="failed to get container status \"421151e16a377defdd053f0c132d7de7a43d0076d12e00ead83faf3fb8e32d1d\": rpc error: code = NotFound desc = could not find container \"421151e16a377defdd053f0c132d7de7a43d0076d12e00ead83faf3fb8e32d1d\": container with ID starting with 421151e16a377defdd053f0c132d7de7a43d0076d12e00ead83faf3fb8e32d1d not found: ID does not exist" Oct 30 19:56:10.478025 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.477926 5091 scope.go:122] "RemoveContainer" containerID="3d2859d3610d8ff67d756156c6fc58980bcce43ead1fccf8c327e649a8709b99" Oct 30 19:56:10.490407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.490385 5091 scope.go:122] "RemoveContainer" containerID="3d2859d3610d8ff67d756156c6fc58980bcce43ead1fccf8c327e649a8709b99" Oct 30 19:56:10.490714 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:56:10.490694 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d2859d3610d8ff67d756156c6fc58980bcce43ead1fccf8c327e649a8709b99\": container with ID starting with 3d2859d3610d8ff67d756156c6fc58980bcce43ead1fccf8c327e649a8709b99 not found: ID does not exist" containerID="3d2859d3610d8ff67d756156c6fc58980bcce43ead1fccf8c327e649a8709b99" Oct 30 19:56:10.490781 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.490725 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d2859d3610d8ff67d756156c6fc58980bcce43ead1fccf8c327e649a8709b99"} err="failed to get container status \"3d2859d3610d8ff67d756156c6fc58980bcce43ead1fccf8c327e649a8709b99\": rpc error: code = NotFound desc = could not find container \"3d2859d3610d8ff67d756156c6fc58980bcce43ead1fccf8c327e649a8709b99\": container with ID starting with 3d2859d3610d8ff67d756156c6fc58980bcce43ead1fccf8c327e649a8709b99 not found: ID does not exist" Oct 30 19:56:10.498766 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.498742 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l"] Oct 30 19:56:10.502785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.502764 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a29f33-b5dddf8c-ntf8l"] Oct 30 19:56:10.520925 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.520878 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb"] Oct 30 19:56:10.524503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.524485 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a29f33-5ffb5696f4-v88tb"] Oct 30 19:56:10.873215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.859706 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:10.919631 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:10.919573 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:11.477210 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:11.477180 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:11.478458 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:11.478437 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:12.022730 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:12.022689 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:12.272815 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:12.272717 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:12.441337 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:12.441306 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02feb7af-17b1-472a-81d4-a5265781f30c" path="/var/lib/kubelet/pods/02feb7af-17b1-472a-81d4-a5265781f30c/volumes" Oct 30 19:56:12.441750 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:12.441735 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f73a998-807d-4652-8c3a-28b4f6d42390" path="/var/lib/kubelet/pods/3f73a998-807d-4652-8c3a-28b4f6d42390/volumes" Oct 30 19:56:12.940195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:12.940156 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:13.222448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.222354 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:13.425494 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.425464 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:13.498918 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.498895 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v"] Oct 30 19:56:13.499507 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.499493 5091 cpu_manager.go:414] "containerMap: removing container" podUID="02feb7af-17b1-472a-81d4-a5265781f30c" containerName="payload-processing" Oct 30 19:56:13.499559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.499509 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02feb7af-17b1-472a-81d4-a5265781f30c" containerName="payload-processing" Oct 30 19:56:13.499559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.499545 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3f73a998-807d-4652-8c3a-28b4f6d42390" containerName="payload-pre-processing" Oct 30 19:56:13.499559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.499551 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f73a998-807d-4652-8c3a-28b4f6d42390" containerName="payload-pre-processing" Oct 30 19:56:13.499663 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.499642 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="02feb7af-17b1-472a-81d4-a5265781f30c" containerName="payload-processing" Oct 30 19:56:13.499663 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.499657 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f73a998-807d-4652-8c3a-28b4f6d42390" containerName="payload-pre-processing" Oct 30 19:56:13.505276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.505254 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" Oct 30 19:56:13.516976 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.516954 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v"] Oct 30 19:56:13.521903 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.521881 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-c1768b\"" Oct 30 19:56:13.532459 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.532421 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-c1768b-dockercfg-gxm7h\"" Oct 30 19:56:13.564015 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.563975 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6"] Oct 30 19:56:13.581781 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.581656 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6"] Oct 30 19:56:13.582071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.582052 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" Oct 30 19:56:13.628000 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.627944 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt8zz\" (UniqueName: \"kubernetes.io/projected/763c8469-7822-4c84-ab15-67e20ddbc251-kube-api-access-jt8zz\") pod \"payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v\" (UID: \"763c8469-7822-4c84-ab15-67e20ddbc251\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" Oct 30 19:56:13.628358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.628038 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c22cd7b-df94-4afa-83be-594cee6a2701-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6\" (UID: \"5c22cd7b-df94-4afa-83be-594cee6a2701\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" Oct 30 19:56:13.628358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.628158 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkjp6\" (UniqueName: \"kubernetes.io/projected/5c22cd7b-df94-4afa-83be-594cee6a2701-kube-api-access-rkjp6\") pod \"payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6\" (UID: \"5c22cd7b-df94-4afa-83be-594cee6a2701\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" Oct 30 19:56:13.628358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.628375 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763c8469-7822-4c84-ab15-67e20ddbc251-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v\" (UID: \"763c8469-7822-4c84-ab15-67e20ddbc251\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" Oct 30 19:56:13.687201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.683540 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h"] Oct 30 19:56:13.700407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.700375 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h"] Oct 30 19:56:13.700611 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.700506 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.718791 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.717354 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-b24a1a-openshift-default-dockercfg-h5n8n\"" Oct 30 19:56:13.729612 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.729581 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkjp6\" (UniqueName: \"kubernetes.io/projected/5c22cd7b-df94-4afa-83be-594cee6a2701-kube-api-access-rkjp6\") pod \"payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6\" (UID: \"5c22cd7b-df94-4afa-83be-594cee6a2701\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" Oct 30 19:56:13.729751 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.729739 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763c8469-7822-4c84-ab15-67e20ddbc251-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v\" (UID: \"763c8469-7822-4c84-ab15-67e20ddbc251\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" Oct 30 19:56:13.729829 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.729817 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt8zz\" (UniqueName: \"kubernetes.io/projected/763c8469-7822-4c84-ab15-67e20ddbc251-kube-api-access-jt8zz\") pod \"payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v\" (UID: \"763c8469-7822-4c84-ab15-67e20ddbc251\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" Oct 30 19:56:13.729872 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.729858 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c22cd7b-df94-4afa-83be-594cee6a2701-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6\" (UID: \"5c22cd7b-df94-4afa-83be-594cee6a2701\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" Oct 30 19:56:13.730597 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.730577 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c22cd7b-df94-4afa-83be-594cee6a2701-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6\" (UID: \"5c22cd7b-df94-4afa-83be-594cee6a2701\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" Oct 30 19:56:13.730670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.730628 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763c8469-7822-4c84-ab15-67e20ddbc251-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v\" (UID: \"763c8469-7822-4c84-ab15-67e20ddbc251\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" Oct 30 19:56:13.748027 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.747986 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkjp6\" (UniqueName: \"kubernetes.io/projected/5c22cd7b-df94-4afa-83be-594cee6a2701-kube-api-access-rkjp6\") pod \"payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6\" (UID: \"5c22cd7b-df94-4afa-83be-594cee6a2701\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" Oct 30 19:56:13.748748 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.748708 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt8zz\" (UniqueName: \"kubernetes.io/projected/763c8469-7822-4c84-ab15-67e20ddbc251-kube-api-access-jt8zz\") pod \"payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v\" (UID: \"763c8469-7822-4c84-ab15-67e20ddbc251\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" Oct 30 19:56:13.758881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.758752 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:13.822711 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.822668 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" Oct 30 19:56:13.831529 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.831502 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-workload-socket\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.831647 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.831533 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f206f771-c29d-41db-8768-12d2feefe4e3-istio-token\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.831694 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.831649 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-credential-socket\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.831735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.831711 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ps5g\" (UniqueName: \"kubernetes.io/projected/f206f771-c29d-41db-8768-12d2feefe4e3-kube-api-access-8ps5g\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.831804 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.831785 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-istio-envoy\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.831850 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.831838 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f206f771-c29d-41db-8768-12d2feefe4e3-istiod-ca-cert\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.831996 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.831983 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-istio-data\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.832049 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.832012 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-workload-certs\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.832049 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.832036 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f206f771-c29d-41db-8768-12d2feefe4e3-istio-podinfo\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.917385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.917350 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" Oct 30 19:56:13.937559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.935880 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-istio-data\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.937559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.935937 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-workload-certs\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.937559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.935968 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f206f771-c29d-41db-8768-12d2feefe4e3-istio-podinfo\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.937559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.935998 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-workload-socket\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.937559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.936019 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f206f771-c29d-41db-8768-12d2feefe4e3-istio-token\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.937559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.936087 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-credential-socket\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.937559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.936239 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ps5g\" (UniqueName: \"kubernetes.io/projected/f206f771-c29d-41db-8768-12d2feefe4e3-kube-api-access-8ps5g\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.937559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.936290 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-istio-envoy\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.937559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.936339 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f206f771-c29d-41db-8768-12d2feefe4e3-istiod-ca-cert\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.937559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.937182 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f206f771-c29d-41db-8768-12d2feefe4e3-istiod-ca-cert\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.937559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.937496 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-workload-socket\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.937559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.937512 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-workload-certs\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.938250 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.937659 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-credential-socket\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.938250 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.937697 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-istio-data\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.939698 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.939672 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-istio-envoy\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.940990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.940962 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f206f771-c29d-41db-8768-12d2feefe4e3-istio-podinfo\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.969483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.969303 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ps5g\" (UniqueName: \"kubernetes.io/projected/f206f771-c29d-41db-8768-12d2feefe4e3-kube-api-access-8ps5g\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:13.969483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:13.969409 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f206f771-c29d-41db-8768-12d2feefe4e3-istio-token\") pod \"e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:14.018595 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:14.017212 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:14.108254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:14.108175 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:14.270572 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:14.270485 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v"] Oct 30 19:56:14.397426 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:14.396976 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:14.474166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:14.471641 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6"] Oct 30 19:56:14.521175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:14.521034 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" event={"ID":"5c22cd7b-df94-4afa-83be-594cee6a2701","Type":"ContainerStarted","Data":"dee82afc04527585bf008d8c656ad7f4e8c9ffea77effcab757ef10496a6cf8d"} Oct 30 19:56:14.534468 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:14.526127 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" event={"ID":"763c8469-7822-4c84-ab15-67e20ddbc251","Type":"ContainerStarted","Data":"05e380c6af095e88002c374f23b853401027802039d92e24cde4da6b8b9a8412"} Oct 30 19:56:14.534468 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:14.526218 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" event={"ID":"763c8469-7822-4c84-ab15-67e20ddbc251","Type":"ContainerStarted","Data":"edebbc708ae30bf77b17d6d7bac1c33be6ed51e909f841544795ddff72552bfb"} Oct 30 19:56:14.534468 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:14.527285 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" Oct 30 19:56:14.600485 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:14.600370 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:14.841211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:14.837395 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" podStartSLOduration=1.8373732550000001 podStartE2EDuration="1.837373255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:14.557438388 +0000 UTC m=+1494.252646347" watchObservedRunningTime="2026-10-30 19:56:14.837373255 +0000 UTC m=+1494.532581215" Oct 30 19:56:14.841211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:14.839505 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h"] Oct 30 19:56:14.878896 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:14.867183 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:14.878896 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:14.867271 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:14.878896 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:14.867314 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:15.506180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:15.505828 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:15.614252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:15.614213 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" event={"ID":"f206f771-c29d-41db-8768-12d2feefe4e3","Type":"ContainerStarted","Data":"8165d039203b13ecfef0568b2ff185b4e100ab5341ebf9306bdf750f4cb14819"} Oct 30 19:56:15.614535 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:15.614497 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" event={"ID":"f206f771-c29d-41db-8768-12d2feefe4e3","Type":"ContainerStarted","Data":"22291d3700b0460e42d6fec0347f2b5c30e0f572be1858b5cf9f90806c536a44"} Oct 30 19:56:15.934777 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:15.933541 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:15.935235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:15.935072 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" event={"ID":"5c22cd7b-df94-4afa-83be-594cee6a2701","Type":"ContainerStarted","Data":"b5f98e6fc7b696628abac67e541937c78cd88c87f10152dd36323c9d8c355067"} Oct 30 19:56:15.935235 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:15.935161 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" Oct 30 19:56:16.061646 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:16.052059 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2"] Oct 30 19:56:16.061646 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:16.061291 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:16.061646 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:16.061567 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" podUID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" containerName="istio-proxy" containerID="cri-o://7aa3069b2f5607b1aadfb82a6851d55ed03154cef57075c24f42705c8d855dd1" gracePeriod=30 Oct 30 19:56:16.070520 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:16.067859 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 19:56:16.070520 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:16.067921 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 19:56:16.126778 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:16.126677 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:16.159220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:16.156886 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" podStartSLOduration=3.156862791 podStartE2EDuration="3.156862791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:15.72518249 +0000 UTC m=+1495.420390449" watchObservedRunningTime="2026-10-30 19:56:16.156862791 +0000 UTC m=+1495.852070749" Oct 30 19:56:16.164711 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:16.164677 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:16.224607 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:16.224495 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:16.248172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:16.246880 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" podStartSLOduration=3.246856667 podStartE2EDuration="3.246856667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:15.987875394 +0000 UTC m=+1495.683083354" watchObservedRunningTime="2026-10-30 19:56:16.246856667 +0000 UTC m=+1495.942064626" Oct 30 19:56:16.330304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:16.329250 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:16.506430 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:56:16.479513 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:56:16.651043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:16.650480 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 19:56:17.042540 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:17.042497 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:17.344345 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:17.344241 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:17.345121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:17.345094 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:56:17.346986 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:17.346960 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:18.071170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:18.055703 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:18.352802 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:18.352568 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:18.353270 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:18.353235 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:18.371199 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:18.354964 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 19:56:18.404567 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:18.403753 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:18.407345 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:18.407323 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:18.408197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:18.408179 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:56:18.516767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:18.516669 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:18.854176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:18.851928 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:19.168878 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:19.168786 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:19.470222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:19.469719 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:19.783726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:19.783669 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:19.796410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:19.796384 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h"] Oct 30 19:56:19.803858 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:19.803830 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-f03120-tqz5h"] Oct 30 19:56:19.949783 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:19.949733 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:20.191236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.191101 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:20.391716 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.384594 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:20.569717 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.569658 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:20.577565 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.577533 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37cb4162-feeb-4fec-be93-e02fdd499706" path="/var/lib/kubelet/pods/37cb4162-feeb-4fec-be93-e02fdd499706/volumes" Oct 30 19:56:20.585364 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.585312 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp"] Oct 30 19:56:20.592959 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.592932 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" Oct 30 19:56:20.596352 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.596326 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp"] Oct 30 19:56:20.624612 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.624588 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:56:20.624749 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.624648 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:56:20.681139 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.681106 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e917c4e9-9a21-48fa-b866-291661ad2b22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c4f759c-hnksp\" (UID: \"e917c4e9-9a21-48fa-b866-291661ad2b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" Oct 30 19:56:20.681328 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.681249 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt2ds\" (UniqueName: \"kubernetes.io/projected/e917c4e9-9a21-48fa-b866-291661ad2b22-kube-api-access-tt2ds\") pod \"maas-api-revoke-sub-6c4f759c-hnksp\" (UID: \"e917c4e9-9a21-48fa-b866-291661ad2b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" Oct 30 19:56:20.681328 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.681298 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e917c4e9-9a21-48fa-b866-291661ad2b22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c4f759c-hnksp\" (UID: \"e917c4e9-9a21-48fa-b866-291661ad2b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" Oct 30 19:56:20.782668 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.782624 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e917c4e9-9a21-48fa-b866-291661ad2b22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c4f759c-hnksp\" (UID: \"e917c4e9-9a21-48fa-b866-291661ad2b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" Oct 30 19:56:20.782864 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.782703 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt2ds\" (UniqueName: \"kubernetes.io/projected/e917c4e9-9a21-48fa-b866-291661ad2b22-kube-api-access-tt2ds\") pod \"maas-api-revoke-sub-6c4f759c-hnksp\" (UID: \"e917c4e9-9a21-48fa-b866-291661ad2b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" Oct 30 19:56:20.782864 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.782740 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e917c4e9-9a21-48fa-b866-291661ad2b22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c4f759c-hnksp\" (UID: \"e917c4e9-9a21-48fa-b866-291661ad2b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" Oct 30 19:56:20.783550 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.783520 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e917c4e9-9a21-48fa-b866-291661ad2b22-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c4f759c-hnksp\" (UID: \"e917c4e9-9a21-48fa-b866-291661ad2b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" Oct 30 19:56:20.786759 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.786738 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:56:20.795616 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.795591 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e917c4e9-9a21-48fa-b866-291661ad2b22-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c4f759c-hnksp\" (UID: \"e917c4e9-9a21-48fa-b866-291661ad2b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" Oct 30 19:56:20.797953 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.797929 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt2ds\" (UniqueName: \"kubernetes.io/projected/e917c4e9-9a21-48fa-b866-291661ad2b22-kube-api-access-tt2ds\") pod \"maas-api-revoke-sub-6c4f759c-hnksp\" (UID: \"e917c4e9-9a21-48fa-b866-291661ad2b22\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" Oct 30 19:56:20.798076 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.798015 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:56:20.805457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.805432 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 19:56:20.917700 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.917619 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:56:20.924101 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:20.924062 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" Oct 30 19:56:21.039358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.039328 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:21.277430 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.277329 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:21.422271 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.422238 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp"] Oct 30 19:56:21.592022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.591994 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:56:21.599067 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.599030 5091 generic.go:356] "Generic (PLEG): container finished" podID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" containerID="7aa3069b2f5607b1aadfb82a6851d55ed03154cef57075c24f42705c8d855dd1" exitCode=0 Oct 30 19:56:21.599343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.599205 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" event={"ID":"57f477a5-ec74-470b-bb7e-afd4d690e8c7","Type":"ContainerDied","Data":"7aa3069b2f5607b1aadfb82a6851d55ed03154cef57075c24f42705c8d855dd1"} Oct 30 19:56:21.599343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.599264 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" event={"ID":"57f477a5-ec74-470b-bb7e-afd4d690e8c7","Type":"ContainerDied","Data":"2c1b2982a19ceb7e6463939720b2851d3c3d03168fcbdb23d6b14fa9e848030e"} Oct 30 19:56:21.599343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.599296 5091 scope.go:122] "RemoveContainer" containerID="7aa3069b2f5607b1aadfb82a6851d55ed03154cef57075c24f42705c8d855dd1" Oct 30 19:56:21.599562 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.599297 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2" Oct 30 19:56:21.602043 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.602017 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" event={"ID":"e917c4e9-9a21-48fa-b866-291661ad2b22","Type":"ContainerStarted","Data":"bd2dcbb890a7ba6c99662b7eca30b2e086c2ce720be7c6dd31ef2439cddf2d9a"} Oct 30 19:56:21.613054 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.613028 5091 scope.go:122] "RemoveContainer" containerID="95dddeac43c00e2a488aa748dd39e4ada5cf271b8855707a4bddede54450ba48" Oct 30 19:56:21.625504 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.625481 5091 scope.go:122] "RemoveContainer" containerID="7aa3069b2f5607b1aadfb82a6851d55ed03154cef57075c24f42705c8d855dd1" Oct 30 19:56:21.626246 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:56:21.626216 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7aa3069b2f5607b1aadfb82a6851d55ed03154cef57075c24f42705c8d855dd1\": container with ID starting with 7aa3069b2f5607b1aadfb82a6851d55ed03154cef57075c24f42705c8d855dd1 not found: ID does not exist" containerID="7aa3069b2f5607b1aadfb82a6851d55ed03154cef57075c24f42705c8d855dd1" Oct 30 19:56:21.626443 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.626252 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7aa3069b2f5607b1aadfb82a6851d55ed03154cef57075c24f42705c8d855dd1"} err="failed to get container status \"7aa3069b2f5607b1aadfb82a6851d55ed03154cef57075c24f42705c8d855dd1\": rpc error: code = NotFound desc = could not find container \"7aa3069b2f5607b1aadfb82a6851d55ed03154cef57075c24f42705c8d855dd1\": container with ID starting with 7aa3069b2f5607b1aadfb82a6851d55ed03154cef57075c24f42705c8d855dd1 not found: ID does not exist" Oct 30 19:56:21.626443 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.626271 5091 scope.go:122] "RemoveContainer" containerID="95dddeac43c00e2a488aa748dd39e4ada5cf271b8855707a4bddede54450ba48" Oct 30 19:56:21.626661 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:56:21.626630 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95dddeac43c00e2a488aa748dd39e4ada5cf271b8855707a4bddede54450ba48\": container with ID starting with 95dddeac43c00e2a488aa748dd39e4ada5cf271b8855707a4bddede54450ba48 not found: ID does not exist" containerID="95dddeac43c00e2a488aa748dd39e4ada5cf271b8855707a4bddede54450ba48" Oct 30 19:56:21.626779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.626669 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95dddeac43c00e2a488aa748dd39e4ada5cf271b8855707a4bddede54450ba48"} err="failed to get container status \"95dddeac43c00e2a488aa748dd39e4ada5cf271b8855707a4bddede54450ba48\": rpc error: code = NotFound desc = could not find container \"95dddeac43c00e2a488aa748dd39e4ada5cf271b8855707a4bddede54450ba48\": container with ID starting with 95dddeac43c00e2a488aa748dd39e4ada5cf271b8855707a4bddede54450ba48 not found: ID does not exist" Oct 30 19:56:21.700980 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.700902 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istiod-ca-cert\") pod \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " Oct 30 19:56:21.701126 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.700992 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-workload-socket\") pod \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " Oct 30 19:56:21.701126 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.701099 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-data\") pod \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " Oct 30 19:56:21.701222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.701167 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-workload-certs\") pod \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " Oct 30 19:56:21.701265 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.701231 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-envoy\") pod \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " Oct 30 19:56:21.701353 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.701309 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-workload-socket" pod "57f477a5-ec74-470b-bb7e-afd4d690e8c7" (UID: "57f477a5-ec74-470b-bb7e-afd4d690e8c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:21.701353 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.701333 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-podinfo\") pod \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " Oct 30 19:56:21.701575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.701383 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-data" pod "57f477a5-ec74-470b-bb7e-afd4d690e8c7" (UID: "57f477a5-ec74-470b-bb7e-afd4d690e8c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:21.701575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.701423 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f477a5-ec74-470b-bb7e-afd4d690e8c7-kube-api-access-2rbbd\" (UniqueName: \"kubernetes.io/projected/57f477a5-ec74-470b-bb7e-afd4d690e8c7-kube-api-access-2rbbd\") pod \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " Oct 30 19:56:21.701575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.701426 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istiod-ca-cert" pod "57f477a5-ec74-470b-bb7e-afd4d690e8c7" (UID: "57f477a5-ec74-470b-bb7e-afd4d690e8c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:21.701575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.701469 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-credential-socket\") pod \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " Oct 30 19:56:21.701575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.701508 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-workload-certs" pod "57f477a5-ec74-470b-bb7e-afd4d690e8c7" (UID: "57f477a5-ec74-470b-bb7e-afd4d690e8c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:21.701575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.701575 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-token\" (UniqueName: \"kubernetes.io/projected/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-token\") pod \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\" (UID: \"57f477a5-ec74-470b-bb7e-afd4d690e8c7\") " Oct 30 19:56:21.701868 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.701755 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-credential-socket" pod "57f477a5-ec74-470b-bb7e-afd4d690e8c7" (UID: "57f477a5-ec74-470b-bb7e-afd4d690e8c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:21.702139 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.702117 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:21.702266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.702158 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:21.702266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.702173 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:21.702266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.702186 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:21.702266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.702199 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:21.704074 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.704047 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-envoy" pod "57f477a5-ec74-470b-bb7e-afd4d690e8c7" (UID: "57f477a5-ec74-470b-bb7e-afd4d690e8c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:21.704254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.704201 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-podinfo" pod "57f477a5-ec74-470b-bb7e-afd4d690e8c7" (UID: "57f477a5-ec74-470b-bb7e-afd4d690e8c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:56:21.704384 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.704360 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f477a5-ec74-470b-bb7e-afd4d690e8c7-kube-api-access-2rbbd" pod "57f477a5-ec74-470b-bb7e-afd4d690e8c7" (UID: "57f477a5-ec74-470b-bb7e-afd4d690e8c7"). InnerVolumeSpecName "kube-api-access-2rbbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:21.704918 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.704891 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-token" pod "57f477a5-ec74-470b-bb7e-afd4d690e8c7" (UID: "57f477a5-ec74-470b-bb7e-afd4d690e8c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:21.778442 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.778404 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:21.803956 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.803921 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:21.803956 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.803952 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:21.804185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.803965 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rbbd\" (UniqueName: \"kubernetes.io/projected/57f477a5-ec74-470b-bb7e-afd4d690e8c7-kube-api-access-2rbbd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:21.804185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.803980 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57f477a5-ec74-470b-bb7e-afd4d690e8c7-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:21.931137 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.931111 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2"] Oct 30 19:56:21.935975 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:21.935953 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-a29f33-openshift-default-5b4bcc765-c6gm2"] Oct 30 19:56:22.209301 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:22.209270 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:22.349131 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:22.349093 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:22.442294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:22.442264 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" path="/var/lib/kubelet/pods/57f477a5-ec74-470b-bb7e-afd4d690e8c7/volumes" Oct 30 19:56:22.611662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:22.611573 5091 generic.go:356] "Generic (PLEG): container finished" podID="e917c4e9-9a21-48fa-b866-291661ad2b22" containerID="7ca5363585d104d67cec2b6cba60ccad600b69e3c5a551a5e1c8b657eb4b44d0" exitCode=0 Oct 30 19:56:22.611662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:22.611640 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" event={"ID":"e917c4e9-9a21-48fa-b866-291661ad2b22","Type":"ContainerDied","Data":"7ca5363585d104d67cec2b6cba60ccad600b69e3c5a551a5e1c8b657eb4b44d0"} Oct 30 19:56:22.793315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:22.793282 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:23.801208 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:23.801141 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" Oct 30 19:56:23.879627 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:23.879480 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:23.937315 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:23.937287 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e917c4e9-9a21-48fa-b866-291661ad2b22-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e917c4e9-9a21-48fa-b866-291661ad2b22-maas-api-openshift-ca\") pod \"e917c4e9-9a21-48fa-b866-291661ad2b22\" (UID: \"e917c4e9-9a21-48fa-b866-291661ad2b22\") " Oct 30 19:56:23.937540 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:23.937522 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e917c4e9-9a21-48fa-b866-291661ad2b22-kube-api-access-tt2ds\" (UniqueName: \"kubernetes.io/projected/e917c4e9-9a21-48fa-b866-291661ad2b22-kube-api-access-tt2ds\") pod \"e917c4e9-9a21-48fa-b866-291661ad2b22\" (UID: \"e917c4e9-9a21-48fa-b866-291661ad2b22\") " Oct 30 19:56:23.937618 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:23.937564 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e917c4e9-9a21-48fa-b866-291661ad2b22-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e917c4e9-9a21-48fa-b866-291661ad2b22-maas-api-xks-ca\") pod \"e917c4e9-9a21-48fa-b866-291661ad2b22\" (UID: \"e917c4e9-9a21-48fa-b866-291661ad2b22\") " Oct 30 19:56:23.937671 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:23.937651 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e917c4e9-9a21-48fa-b866-291661ad2b22-maas-api-openshift-ca" pod "e917c4e9-9a21-48fa-b866-291661ad2b22" (UID: "e917c4e9-9a21-48fa-b866-291661ad2b22"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:23.938193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:23.938132 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e917c4e9-9a21-48fa-b866-291661ad2b22-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:23.939708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:23.939679 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e917c4e9-9a21-48fa-b866-291661ad2b22-maas-api-xks-ca" pod "e917c4e9-9a21-48fa-b866-291661ad2b22" (UID: "e917c4e9-9a21-48fa-b866-291661ad2b22"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:23.939708 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:23.939685 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e917c4e9-9a21-48fa-b866-291661ad2b22-kube-api-access-tt2ds" pod "e917c4e9-9a21-48fa-b866-291661ad2b22" (UID: "e917c4e9-9a21-48fa-b866-291661ad2b22"). InnerVolumeSpecName "kube-api-access-tt2ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:24.059170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:24.050696 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt2ds\" (UniqueName: \"kubernetes.io/projected/e917c4e9-9a21-48fa-b866-291661ad2b22-kube-api-access-tt2ds\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:24.059170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:24.050734 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e917c4e9-9a21-48fa-b866-291661ad2b22-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:24.101279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:24.101241 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:24.304619 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:24.304581 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:24.630383 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:24.630308 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" Oct 30 19:56:24.630383 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:24.630345 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp" event={"ID":"e917c4e9-9a21-48fa-b866-291661ad2b22","Type":"ContainerDied","Data":"bd2dcbb890a7ba6c99662b7eca30b2e086c2ce720be7c6dd31ef2439cddf2d9a"} Oct 30 19:56:24.630619 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:24.630392 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd2dcbb890a7ba6c99662b7eca30b2e086c2ce720be7c6dd31ef2439cddf2d9a" Oct 30 19:56:25.193175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:25.181742 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:26.226367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:26.217161 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:27.017475 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:27.017444 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" Oct 30 19:56:27.017648 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:27.017571 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" Oct 30 19:56:27.123499 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:27.123450 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:27.628319 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:27.628289 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:27.797009 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:27.796971 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:27.952914 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:27.952825 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd"] Oct 30 19:56:27.953117 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:27.953089 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" podUID="2bff8956-093c-4a78-a281-8571789127a7" containerName="istio-proxy" containerID="cri-o://8704e0ac2dd4bd7d9cc142cfdf13918342a82975233e1e14c43ff18bf852d989" gracePeriod=30 Oct 30 19:56:28.046071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:28.046025 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:28.202733 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:28.202698 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:29.008125 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:29.008087 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:29.169171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:29.168435 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:30.279660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:30.279622 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:30.454466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:30.454429 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:31.429270 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:31.429219 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:32.585978 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:32.585938 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:33.227108 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.227079 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5"] Oct 30 19:56:33.227854 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.227835 5091 cpu_manager.go:414] "containerMap: removing container" podUID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" containerName="istio-proxy" Oct 30 19:56:33.227929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.227856 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" containerName="istio-proxy" Oct 30 19:56:33.227929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.227876 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e917c4e9-9a21-48fa-b866-291661ad2b22" containerName="revoke-keys" Oct 30 19:56:33.227929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.227881 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e917c4e9-9a21-48fa-b866-291661ad2b22" containerName="revoke-keys" Oct 30 19:56:33.227929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.227889 5091 cpu_manager.go:414] "containerMap: removing container" podUID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" containerName="istio-proxy" Oct 30 19:56:33.227929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.227894 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" containerName="istio-proxy" Oct 30 19:56:33.228129 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.228001 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" containerName="istio-proxy" Oct 30 19:56:33.228129 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.228011 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e917c4e9-9a21-48fa-b866-291661ad2b22" containerName="revoke-keys" Oct 30 19:56:33.232786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.232764 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" Oct 30 19:56:33.240922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.240895 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5"] Oct 30 19:56:33.241155 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.241129 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:56:33.241825 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.241809 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:56:33.269921 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.269892 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:56:33.321973 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.321943 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-workload-certs\") pod \"2bff8956-093c-4a78-a281-8571789127a7\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " Oct 30 19:56:33.322170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.322041 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-istio-data\") pod \"2bff8956-093c-4a78-a281-8571789127a7\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " Oct 30 19:56:33.322170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.322082 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-istio-envoy\") pod \"2bff8956-093c-4a78-a281-8571789127a7\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " Oct 30 19:56:33.322170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.322124 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bff8956-093c-4a78-a281-8571789127a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bff8956-093c-4a78-a281-8571789127a7-istiod-ca-cert\") pod \"2bff8956-093c-4a78-a281-8571789127a7\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " Oct 30 19:56:33.322348 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.322174 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bff8956-093c-4a78-a281-8571789127a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bff8956-093c-4a78-a281-8571789127a7-istio-podinfo\") pod \"2bff8956-093c-4a78-a281-8571789127a7\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " Oct 30 19:56:33.322348 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.322259 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-credential-socket\") pod \"2bff8956-093c-4a78-a281-8571789127a7\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " Oct 30 19:56:33.322348 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.322296 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bff8956-093c-4a78-a281-8571789127a7-kube-api-access-h5c47\" (UniqueName: \"kubernetes.io/projected/2bff8956-093c-4a78-a281-8571789127a7-kube-api-access-h5c47\") pod \"2bff8956-093c-4a78-a281-8571789127a7\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " Oct 30 19:56:33.322508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.322347 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-workload-socket\") pod \"2bff8956-093c-4a78-a281-8571789127a7\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " Oct 30 19:56:33.322508 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.322454 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bff8956-093c-4a78-a281-8571789127a7-istio-token\" (UniqueName: \"kubernetes.io/projected/2bff8956-093c-4a78-a281-8571789127a7-istio-token\") pod \"2bff8956-093c-4a78-a281-8571789127a7\" (UID: \"2bff8956-093c-4a78-a281-8571789127a7\") " Oct 30 19:56:33.322613 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.322540 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-istio-data" pod "2bff8956-093c-4a78-a281-8571789127a7" (UID: "2bff8956-093c-4a78-a281-8571789127a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:33.322613 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.322558 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bff8956-093c-4a78-a281-8571789127a7-istiod-ca-cert" pod "2bff8956-093c-4a78-a281-8571789127a7" (UID: "2bff8956-093c-4a78-a281-8571789127a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:33.322720 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.322696 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ee4d261-m86d5\" (UID: \"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" Oct 30 19:56:33.322781 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.322749 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ee4d261-m86d5\" (UID: \"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" Oct 30 19:56:33.322781 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.322768 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-credential-socket" pod "2bff8956-093c-4a78-a281-8571789127a7" (UID: "2bff8956-093c-4a78-a281-8571789127a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:33.322967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.322945 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-workload-socket" pod "2bff8956-093c-4a78-a281-8571789127a7" (UID: "2bff8956-093c-4a78-a281-8571789127a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:33.322967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.322954 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjgq4\" (UniqueName: \"kubernetes.io/projected/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-kube-api-access-kjgq4\") pod \"maas-api-revoke-sub-8ee4d261-m86d5\" (UID: \"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" Oct 30 19:56:33.323281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.323258 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-workload-certs" pod "2bff8956-093c-4a78-a281-8571789127a7" (UID: "2bff8956-093c-4a78-a281-8571789127a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:33.323492 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.323473 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:33.323560 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.323499 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bff8956-093c-4a78-a281-8571789127a7-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:33.323560 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.323514 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:33.323560 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.323528 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:33.323560 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.323542 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:33.325552 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.325529 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bff8956-093c-4a78-a281-8571789127a7-istio-podinfo" pod "2bff8956-093c-4a78-a281-8571789127a7" (UID: "2bff8956-093c-4a78-a281-8571789127a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:56:33.326925 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.326897 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bff8956-093c-4a78-a281-8571789127a7-istio-token" pod "2bff8956-093c-4a78-a281-8571789127a7" (UID: "2bff8956-093c-4a78-a281-8571789127a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:33.327185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.327137 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-istio-envoy" pod "2bff8956-093c-4a78-a281-8571789127a7" (UID: "2bff8956-093c-4a78-a281-8571789127a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:56:33.327299 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.327280 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bff8956-093c-4a78-a281-8571789127a7-kube-api-access-h5c47" pod "2bff8956-093c-4a78-a281-8571789127a7" (UID: "2bff8956-093c-4a78-a281-8571789127a7"). InnerVolumeSpecName "kube-api-access-h5c47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:33.359022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.358938 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:33.425242 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.425202 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjgq4\" (UniqueName: \"kubernetes.io/projected/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-kube-api-access-kjgq4\") pod \"maas-api-revoke-sub-8ee4d261-m86d5\" (UID: \"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" Oct 30 19:56:33.425435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.425349 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ee4d261-m86d5\" (UID: \"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" Oct 30 19:56:33.425435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.425397 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ee4d261-m86d5\" (UID: \"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" Oct 30 19:56:33.425559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.425486 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bff8956-093c-4a78-a281-8571789127a7-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:33.425559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.425502 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bff8956-093c-4a78-a281-8571789127a7-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:33.425559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.425520 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bff8956-093c-4a78-a281-8571789127a7-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:33.425559 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.425535 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5c47\" (UniqueName: \"kubernetes.io/projected/2bff8956-093c-4a78-a281-8571789127a7-kube-api-access-h5c47\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:33.426094 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.426066 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8ee4d261-m86d5\" (UID: \"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" Oct 30 19:56:33.427834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.427810 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8ee4d261-m86d5\" (UID: \"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" Oct 30 19:56:33.439511 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.439488 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjgq4\" (UniqueName: \"kubernetes.io/projected/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-kube-api-access-kjgq4\") pod \"maas-api-revoke-sub-8ee4d261-m86d5\" (UID: \"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" Oct 30 19:56:33.570395 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.570359 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" Oct 30 19:56:33.650082 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.641231 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:33.707713 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.707677 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b"] Oct 30 19:56:33.708691 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.708667 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2bff8956-093c-4a78-a281-8571789127a7" containerName="istio-proxy" Oct 30 19:56:33.708691 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.708689 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bff8956-093c-4a78-a281-8571789127a7" containerName="istio-proxy" Oct 30 19:56:33.708691 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.708704 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2bff8956-093c-4a78-a281-8571789127a7" containerName="istio-proxy" Oct 30 19:56:33.708950 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.708713 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bff8956-093c-4a78-a281-8571789127a7" containerName="istio-proxy" Oct 30 19:56:33.708950 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.708862 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bff8956-093c-4a78-a281-8571789127a7" containerName="istio-proxy" Oct 30 19:56:33.708950 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.708874 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bff8956-093c-4a78-a281-8571789127a7" containerName="istio-proxy" Oct 30 19:56:33.708950 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.708888 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="57f477a5-ec74-470b-bb7e-afd4d690e8c7" containerName="istio-proxy" Oct 30 19:56:33.714486 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.714462 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.720920 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.720893 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-b7431c-openshift-default-dockercfg-kcbtj\"" Oct 30 19:56:33.728952 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.728921 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-workload-socket\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.729075 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.728968 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbaef91-57be-4511-a038-a607def818b5-istiod-ca-cert\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.729075 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.729037 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-istio-envoy\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.729227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.729080 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbaef91-57be-4511-a038-a607def818b5-istio-podinfo\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.729227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.729106 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-credential-socket\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.729227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.729189 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89xwg\" (UniqueName: \"kubernetes.io/projected/0fbaef91-57be-4511-a038-a607def818b5-kube-api-access-89xwg\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.729392 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.729272 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-istio-data\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.729392 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.729309 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-workload-certs\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.729392 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.729336 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbaef91-57be-4511-a038-a607def818b5-istio-token\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.747351 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.744565 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b"] Oct 30 19:56:33.778879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.778841 5091 generic.go:356] "Generic (PLEG): container finished" podID="2bff8956-093c-4a78-a281-8571789127a7" containerID="8704e0ac2dd4bd7d9cc142cfdf13918342a82975233e1e14c43ff18bf852d989" exitCode=0 Oct 30 19:56:33.779030 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.778917 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" Oct 30 19:56:33.779075 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.779043 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" event={"ID":"2bff8956-093c-4a78-a281-8571789127a7","Type":"ContainerDied","Data":"8704e0ac2dd4bd7d9cc142cfdf13918342a82975233e1e14c43ff18bf852d989"} Oct 30 19:56:33.779113 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.779087 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd" event={"ID":"2bff8956-093c-4a78-a281-8571789127a7","Type":"ContainerDied","Data":"5c7330de0e69f575f65f9eb6c5b8bd22fee895383274048943b6718a2caa2647"} Oct 30 19:56:33.779172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.779116 5091 scope.go:122] "RemoveContainer" containerID="8704e0ac2dd4bd7d9cc142cfdf13918342a82975233e1e14c43ff18bf852d989" Oct 30 19:56:33.802769 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.802724 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:33.803193 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.803130 5091 scope.go:122] "RemoveContainer" containerID="daaa5ef9b5bc570c59b69658ec8d3fc45424a35ba2b8d89db66a46dbb8df6743" Oct 30 19:56:33.814955 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.814939 5091 scope.go:122] "RemoveContainer" containerID="8704e0ac2dd4bd7d9cc142cfdf13918342a82975233e1e14c43ff18bf852d989" Oct 30 19:56:33.815242 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:56:33.815212 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8704e0ac2dd4bd7d9cc142cfdf13918342a82975233e1e14c43ff18bf852d989\": container with ID starting with 8704e0ac2dd4bd7d9cc142cfdf13918342a82975233e1e14c43ff18bf852d989 not found: ID does not exist" containerID="8704e0ac2dd4bd7d9cc142cfdf13918342a82975233e1e14c43ff18bf852d989" Oct 30 19:56:33.815326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.815252 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8704e0ac2dd4bd7d9cc142cfdf13918342a82975233e1e14c43ff18bf852d989"} err="failed to get container status \"8704e0ac2dd4bd7d9cc142cfdf13918342a82975233e1e14c43ff18bf852d989\": rpc error: code = NotFound desc = could not find container \"8704e0ac2dd4bd7d9cc142cfdf13918342a82975233e1e14c43ff18bf852d989\": container with ID starting with 8704e0ac2dd4bd7d9cc142cfdf13918342a82975233e1e14c43ff18bf852d989 not found: ID does not exist" Oct 30 19:56:33.815326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.815275 5091 scope.go:122] "RemoveContainer" containerID="daaa5ef9b5bc570c59b69658ec8d3fc45424a35ba2b8d89db66a46dbb8df6743" Oct 30 19:56:33.815558 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:56:33.815537 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daaa5ef9b5bc570c59b69658ec8d3fc45424a35ba2b8d89db66a46dbb8df6743\": container with ID starting with daaa5ef9b5bc570c59b69658ec8d3fc45424a35ba2b8d89db66a46dbb8df6743 not found: ID does not exist" containerID="daaa5ef9b5bc570c59b69658ec8d3fc45424a35ba2b8d89db66a46dbb8df6743" Oct 30 19:56:33.815635 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.815561 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daaa5ef9b5bc570c59b69658ec8d3fc45424a35ba2b8d89db66a46dbb8df6743"} err="failed to get container status \"daaa5ef9b5bc570c59b69658ec8d3fc45424a35ba2b8d89db66a46dbb8df6743\": rpc error: code = NotFound desc = could not find container \"daaa5ef9b5bc570c59b69658ec8d3fc45424a35ba2b8d89db66a46dbb8df6743\": container with ID starting with daaa5ef9b5bc570c59b69658ec8d3fc45424a35ba2b8d89db66a46dbb8df6743 not found: ID does not exist" Oct 30 19:56:33.825997 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.825948 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd"] Oct 30 19:56:33.831111 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.831089 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89xwg\" (UniqueName: \"kubernetes.io/projected/0fbaef91-57be-4511-a038-a607def818b5-kube-api-access-89xwg\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.831234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.831215 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-istio-data\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.831308 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.831255 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-workload-certs\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.831308 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.831288 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbaef91-57be-4511-a038-a607def818b5-istio-token\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.831726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.831468 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-workload-socket\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.831726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.831662 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbaef91-57be-4511-a038-a607def818b5-istiod-ca-cert\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.831877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.831786 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-workload-socket\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.831877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.831800 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-istio-envoy\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.831877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.831804 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-workload-certs\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.831877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.831868 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbaef91-57be-4511-a038-a607def818b5-istio-podinfo\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.832095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.831897 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-credential-socket\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.832575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.832248 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-credential-socket\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.832575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.832560 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbaef91-57be-4511-a038-a607def818b5-istiod-ca-cert\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.832759 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.832740 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-istio-data\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.835071 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.835045 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-f03120-openshift-default-6c67ff9d5c-gxmtd"] Oct 30 19:56:33.835564 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.835544 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-istio-envoy\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.837095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.837070 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbaef91-57be-4511-a038-a607def818b5-istio-podinfo\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.844518 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.844494 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbaef91-57be-4511-a038-a607def818b5-istio-token\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:33.844782 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:33.844766 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89xwg\" (UniqueName: \"kubernetes.io/projected/0fbaef91-57be-4511-a038-a607def818b5-kube-api-access-89xwg\") pod \"e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:34.078375 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:34.076003 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:34.141419 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:34.140012 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5"] Oct 30 19:56:34.148102 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:34.148070 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:34.442340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:34.442311 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bff8956-093c-4a78-a281-8571789127a7" path="/var/lib/kubelet/pods/2bff8956-093c-4a78-a281-8571789127a7/volumes" Oct 30 19:56:34.459683 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:34.459645 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b"] Oct 30 19:56:34.461992 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:34.461959 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:34.462220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:34.462035 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:34.462220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:34.462080 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:34.744225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:34.741464 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:34.796633 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:34.796600 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" event={"ID":"0fbaef91-57be-4511-a038-a607def818b5","Type":"ContainerStarted","Data":"928ff57d39d92e480d79933914c6f7521efe79ac550a7792249ed935e9647e38"} Oct 30 19:56:34.796866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:34.796845 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" event={"ID":"0fbaef91-57be-4511-a038-a607def818b5","Type":"ContainerStarted","Data":"6b77a0a757cd2d0ef396afcc8de56acdc6694904750a640d498ee79e1d828dea"} Oct 30 19:56:34.798698 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:34.798657 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" event={"ID":"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a","Type":"ContainerStarted","Data":"43d6f39613b6a59b4bfaaf9d734ef10d483c144ec20cb5b0d204c1f75a7ec63f"} Oct 30 19:56:34.798844 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:34.798708 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" event={"ID":"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a","Type":"ContainerStarted","Data":"de0e2ac0772e6d7ea7ef2faa809791b1ff66a316f189fc456ae47f7f2bbbf008"} Oct 30 19:56:34.838904 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:34.838848 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" podStartSLOduration=1.838831382 podStartE2EDuration="1.838831382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:34.833921549 +0000 UTC m=+1514.529129507" watchObservedRunningTime="2026-10-30 19:56:34.838831382 +0000 UTC m=+1514.534039341" Oct 30 19:56:34.867552 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:34.867489 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" podStartSLOduration=1.867471966 podStartE2EDuration="1.867471966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:34.863508685 +0000 UTC m=+1514.558716667" watchObservedRunningTime="2026-10-30 19:56:34.867471966 +0000 UTC m=+1514.562679925" Oct 30 19:56:35.102159 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:35.102120 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:35.107649 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:35.107621 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:35.841298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:35.839239 5091 generic.go:356] "Generic (PLEG): container finished" podID="9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a" containerID="43d6f39613b6a59b4bfaaf9d734ef10d483c144ec20cb5b0d204c1f75a7ec63f" exitCode=0 Oct 30 19:56:35.841298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:35.841070 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" event={"ID":"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a","Type":"ContainerDied","Data":"43d6f39613b6a59b4bfaaf9d734ef10d483c144ec20cb5b0d204c1f75a7ec63f"} Oct 30 19:56:35.841298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:35.841132 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:35.850442 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:35.850362 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:56:35.880225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:35.876089 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:37.028633 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:37.028607 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" Oct 30 19:56:37.034385 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:37.034363 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-kube-api-access-kjgq4\" (UniqueName: \"kubernetes.io/projected/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-kube-api-access-kjgq4\") pod \"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a\" (UID: \"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a\") " Oct 30 19:56:37.034555 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:37.034541 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-maas-api-xks-ca\") pod \"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a\" (UID: \"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a\") " Oct 30 19:56:37.034630 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:37.034564 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-maas-api-openshift-ca\") pod \"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a\" (UID: \"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a\") " Oct 30 19:56:37.034944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:37.034919 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-maas-api-openshift-ca" pod "9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a" (UID: "9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:37.036426 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:37.036405 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-kube-api-access-kjgq4" pod "9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a" (UID: "9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a"). InnerVolumeSpecName "kube-api-access-kjgq4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:37.036675 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:37.036649 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-maas-api-xks-ca" pod "9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a" (UID: "9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:37.136962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:37.136875 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjgq4\" (UniqueName: \"kubernetes.io/projected/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-kube-api-access-kjgq4\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:37.136962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:37.136915 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:37.136962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:37.136948 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:37.223604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:37.223565 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:37.458129 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:37.458029 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:37.862781 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:37.862753 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" Oct 30 19:56:37.862951 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:37.862787 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5" event={"ID":"9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a","Type":"ContainerDied","Data":"de0e2ac0772e6d7ea7ef2faa809791b1ff66a316f189fc456ae47f7f2bbbf008"} Oct 30 19:56:37.862951 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:37.862841 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de0e2ac0772e6d7ea7ef2faa809791b1ff66a316f189fc456ae47f7f2bbbf008" Oct 30 19:56:38.675210 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.673180 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:38.775054 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.774996 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5"] Oct 30 19:56:38.776095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.776068 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a" containerName="revoke-keys" Oct 30 19:56:38.776095 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.776094 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a" containerName="revoke-keys" Oct 30 19:56:38.776388 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.776262 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a" containerName="revoke-keys" Oct 30 19:56:38.780855 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.780832 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx"] Oct 30 19:56:38.780991 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.780957 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" Oct 30 19:56:38.785957 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.785933 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" Oct 30 19:56:38.814769 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.812311 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-b7431c\"" Oct 30 19:56:38.814769 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.812542 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-b7431c-dockercfg-cmnfc\"" Oct 30 19:56:38.829219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.823359 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx"] Oct 30 19:56:38.829219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.827908 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5"] Oct 30 19:56:38.897224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.886734 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2pfk\" (UniqueName: \"kubernetes.io/projected/d7db6874-2d87-4709-9545-0f29a1532a6c-kube-api-access-n2pfk\") pod \"payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx\" (UID: \"d7db6874-2d87-4709-9545-0f29a1532a6c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" Oct 30 19:56:38.897625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.897588 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/002df7b8-201d-45a9-be26-076d2b887a7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5\" (UID: \"002df7b8-201d-45a9-be26-076d2b887a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" Oct 30 19:56:38.897858 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.897822 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7db6874-2d87-4709-9545-0f29a1532a6c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx\" (UID: \"d7db6874-2d87-4709-9545-0f29a1532a6c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" Oct 30 19:56:38.898246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:38.898210 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jf89\" (UniqueName: \"kubernetes.io/projected/002df7b8-201d-45a9-be26-076d2b887a7b-kube-api-access-2jf89\") pod \"payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5\" (UID: \"002df7b8-201d-45a9-be26-076d2b887a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" Oct 30 19:56:39.106931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.065021 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jf89\" (UniqueName: \"kubernetes.io/projected/002df7b8-201d-45a9-be26-076d2b887a7b-kube-api-access-2jf89\") pod \"payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5\" (UID: \"002df7b8-201d-45a9-be26-076d2b887a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" Oct 30 19:56:39.128430 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.128389 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2pfk\" (UniqueName: \"kubernetes.io/projected/d7db6874-2d87-4709-9545-0f29a1532a6c-kube-api-access-n2pfk\") pod \"payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx\" (UID: \"d7db6874-2d87-4709-9545-0f29a1532a6c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" Oct 30 19:56:39.129447 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.129412 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/002df7b8-201d-45a9-be26-076d2b887a7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5\" (UID: \"002df7b8-201d-45a9-be26-076d2b887a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" Oct 30 19:56:39.129728 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.129708 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7db6874-2d87-4709-9545-0f29a1532a6c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx\" (UID: \"d7db6874-2d87-4709-9545-0f29a1532a6c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" Oct 30 19:56:39.143173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.130939 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7db6874-2d87-4709-9545-0f29a1532a6c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx\" (UID: \"d7db6874-2d87-4709-9545-0f29a1532a6c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" Oct 30 19:56:39.144270 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.144242 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/002df7b8-201d-45a9-be26-076d2b887a7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5\" (UID: \"002df7b8-201d-45a9-be26-076d2b887a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" Oct 30 19:56:39.165179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.158093 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:39.199169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.189889 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jf89\" (UniqueName: \"kubernetes.io/projected/002df7b8-201d-45a9-be26-076d2b887a7b-kube-api-access-2jf89\") pod \"payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5\" (UID: \"002df7b8-201d-45a9-be26-076d2b887a7b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" Oct 30 19:56:39.211165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.207878 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2pfk\" (UniqueName: \"kubernetes.io/projected/d7db6874-2d87-4709-9545-0f29a1532a6c-kube-api-access-n2pfk\") pod \"payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx\" (UID: \"d7db6874-2d87-4709-9545-0f29a1532a6c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" Oct 30 19:56:39.297903 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.297864 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2"] Oct 30 19:56:39.316765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.315757 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" Oct 30 19:56:39.323687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.323516 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:56:39.323687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.323655 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:56:39.354141 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.342169 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2\" (UID: \"4a3d573e-3eff-4eef-9c8a-164f7851c6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" Oct 30 19:56:39.354141 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.342269 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2\" (UID: \"4a3d573e-3eff-4eef-9c8a-164f7851c6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" Oct 30 19:56:39.354141 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.342348 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqsz8\" (UniqueName: \"kubernetes.io/projected/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-kube-api-access-mqsz8\") pod \"maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2\" (UID: \"4a3d573e-3eff-4eef-9c8a-164f7851c6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" Oct 30 19:56:39.369211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.363700 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2"] Oct 30 19:56:39.413198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.401527 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" Oct 30 19:56:39.448431 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.448389 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" Oct 30 19:56:39.492934 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.449560 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2\" (UID: \"4a3d573e-3eff-4eef-9c8a-164f7851c6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" Oct 30 19:56:39.878817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.829333 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2\" (UID: \"4a3d573e-3eff-4eef-9c8a-164f7851c6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" Oct 30 19:56:39.878817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.829436 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqsz8\" (UniqueName: \"kubernetes.io/projected/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-kube-api-access-mqsz8\") pod \"maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2\" (UID: \"4a3d573e-3eff-4eef-9c8a-164f7851c6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" Oct 30 19:56:39.878817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.573640 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2\" (UID: \"4a3d573e-3eff-4eef-9c8a-164f7851c6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" Oct 30 19:56:39.878817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.831061 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2\" (UID: \"4a3d573e-3eff-4eef-9c8a-164f7851c6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" Oct 30 19:56:39.944182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.914367 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqsz8\" (UniqueName: \"kubernetes.io/projected/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-kube-api-access-mqsz8\") pod \"maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2\" (UID: \"4a3d573e-3eff-4eef-9c8a-164f7851c6f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" Oct 30 19:56:39.985465 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:39.985207 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" Oct 30 19:56:40.427947 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:40.427409 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:40.648044 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:40.647362 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 19:56:41.178875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:41.178818 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:41.381765 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:41.322020 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:41.499249 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:41.465474 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:41.568256 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:41.564684 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:56:41.720138 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:41.655414 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:41.720138 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:41.655851 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:56:42.323884 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:42.323801 5091 patch_prober.go:29] interesting pod/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:56:42.723170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:42.710214 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" podUID="d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:42.723170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:42.637935 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 19:56:42.723170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:42.324608 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:56:42.723170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:42.682745 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 19:56:43.126707 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:43.104804 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:43.193170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:43.182357 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" podUID="540a152c-2be5-4253-b272-586b34835043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:43.217077 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:43.209786 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc" podUID="f1fc93c8-2259-4138-9691-4c9f4e758a01" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:43.256280 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:43.245257 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:56:43.302320 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:43.302285 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:56:43.302610 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:43.302592 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:56:43.302749 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:43.302738 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 19:56:43.321600 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:43.321568 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2"] Oct 30 19:56:43.331248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:43.331228 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5"] Oct 30 19:56:43.337313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:43.337287 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:43.337958 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:43.337933 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx"] Oct 30 19:56:43.727731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:43.727692 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" event={"ID":"d7db6874-2d87-4709-9545-0f29a1532a6c","Type":"ContainerStarted","Data":"c9cb1001d11da4406b76a16ea16366fe7be34c92a6a1cd249041035355944801"} Oct 30 19:56:43.729899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:43.729865 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" event={"ID":"002df7b8-201d-45a9-be26-076d2b887a7b","Type":"ContainerStarted","Data":"f273e950b8fcf4bd70520a7be21b209fcf085afc975547d4543ba766aa1e4cb5"} Oct 30 19:56:43.733370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:43.733308 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" event={"ID":"4a3d573e-3eff-4eef-9c8a-164f7851c6f2","Type":"ContainerStarted","Data":"1f7bccf38e447832a892634ed2f6e99ffabcadf44df66699289e7dda3be36c84"} Oct 30 19:56:44.140472 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:44.136397 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:44.397467 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:44.397384 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:44.901165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:44.898577 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:45.034533 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.031255 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g"] Oct 30 19:56:45.037011 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.036988 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g"] Oct 30 19:56:45.037181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.037024 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" event={"ID":"d7db6874-2d87-4709-9545-0f29a1532a6c","Type":"ContainerStarted","Data":"501ee98893a5b2731381939287fc18f41199f33e95ea797bf31bf0b6ddbdf772"} Oct 30 19:56:45.037181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.037050 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" Oct 30 19:56:45.037181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.037124 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.038446 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.038408 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" event={"ID":"002df7b8-201d-45a9-be26-076d2b887a7b","Type":"ContainerStarted","Data":"fed957e828c7039bedfa8650c9d549e3003beac340e0e6966e1faadb5e7850db"} Oct 30 19:56:45.038568 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.038550 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" Oct 30 19:56:45.041161 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.041121 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" event={"ID":"4a3d573e-3eff-4eef-9c8a-164f7851c6f2","Type":"ContainerStarted","Data":"79b5de0ceafbe42a7641e12bf794886aad7b4adfc4fa3f87a8853838f54cef2a"} Oct 30 19:56:45.042483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.042465 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-7eaeed-openshift-default-dockercfg-7hrlh\"" Oct 30 19:56:45.126117 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.126057 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" podStartSLOduration=7.126039691 podStartE2EDuration="7.126039691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:45.06843603 +0000 UTC m=+1524.763643989" watchObservedRunningTime="2026-10-30 19:56:45.126039691 +0000 UTC m=+1524.821247650" Oct 30 19:56:45.132599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.132554 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" podStartSLOduration=6.132534523 podStartE2EDuration="6.132534523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:45.120603958 +0000 UTC m=+1524.815811920" watchObservedRunningTime="2026-10-30 19:56:45.132534523 +0000 UTC m=+1524.827742481" Oct 30 19:56:45.173255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.171087 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-data\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.173255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.171423 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istiod-ca-cert\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.173255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.171469 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-envoy\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.173255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.171522 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-token\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.173255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.171554 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-workload-socket\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.173255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.171580 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-workload-certs\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.173255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.171656 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v57tr\" (UniqueName: \"kubernetes.io/projected/0153a5c2-c41c-4dd5-9bc2-035455025bd1-kube-api-access-v57tr\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.173255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.171723 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-podinfo\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.173255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.171796 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-credential-socket\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.218170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.217375 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" podStartSLOduration=7.217357482 podStartE2EDuration="7.217357482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:45.160366009 +0000 UTC m=+1524.855573971" watchObservedRunningTime="2026-10-30 19:56:45.217357482 +0000 UTC m=+1524.912565443" Oct 30 19:56:45.275171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.274347 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-credential-socket\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.275171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.274423 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-data\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.275171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.274667 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istiod-ca-cert\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.275171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.274707 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-envoy\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.275171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.274758 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-token\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.275171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.274789 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-workload-socket\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.275171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.274813 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-workload-certs\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.275171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.274881 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v57tr\" (UniqueName: \"kubernetes.io/projected/0153a5c2-c41c-4dd5-9bc2-035455025bd1-kube-api-access-v57tr\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.275171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.274950 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-podinfo\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.288114 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.288070 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-data\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.290454 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.289046 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istiod-ca-cert\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.290680 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.289344 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-workload-socket\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.307172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.289890 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-credential-socket\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.307563 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.290132 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-workload-certs\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.356844 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.356424 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-envoy\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.366353 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.359448 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-token\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.372269 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.367902 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-podinfo\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.407176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.401339 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v57tr\" (UniqueName: \"kubernetes.io/projected/0153a5c2-c41c-4dd5-9bc2-035455025bd1-kube-api-access-v57tr\") pod \"e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.418175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.416372 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:45.511166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:45.510710 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:46.002583 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:46.002549 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:46.119221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:46.118240 5091 generic.go:356] "Generic (PLEG): container finished" podID="4a3d573e-3eff-4eef-9c8a-164f7851c6f2" containerID="79b5de0ceafbe42a7641e12bf794886aad7b4adfc4fa3f87a8853838f54cef2a" exitCode=0 Oct 30 19:56:46.121214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:46.119619 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" event={"ID":"4a3d573e-3eff-4eef-9c8a-164f7851c6f2","Type":"ContainerDied","Data":"79b5de0ceafbe42a7641e12bf794886aad7b4adfc4fa3f87a8853838f54cef2a"} Oct 30 19:56:46.125000 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:56:46.124976 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:56:46.316021 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:46.315994 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g"] Oct 30 19:56:46.321974 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:46.321940 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:46.322073 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:46.322007 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:46.322073 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:46.322038 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:46.873851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:46.873791 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:47.130024 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.129935 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" event={"ID":"0153a5c2-c41c-4dd5-9bc2-035455025bd1","Type":"ContainerStarted","Data":"ccbd6495cf6cda2472174039f2a1ced45112c359f21843030002c9dec0ef3fe3"} Oct 30 19:56:47.130215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.130039 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" event={"ID":"0153a5c2-c41c-4dd5-9bc2-035455025bd1","Type":"ContainerStarted","Data":"88dc36fc2cca0d83c2605b2141b75cf42fa1cbb07967c3c11f752e504950be1d"} Oct 30 19:56:47.192828 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.192321 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" podStartSLOduration=3.192297831 podStartE2EDuration="3.192297831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:47.166550681 +0000 UTC m=+1526.861758668" watchObservedRunningTime="2026-10-30 19:56:47.192297831 +0000 UTC m=+1526.887505789" Oct 30 19:56:47.418522 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.418438 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:47.425175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.425118 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:47.490871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.490838 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" Oct 30 19:56:47.584701 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.584664 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-maas-api-xks-ca\") pod \"4a3d573e-3eff-4eef-9c8a-164f7851c6f2\" (UID: \"4a3d573e-3eff-4eef-9c8a-164f7851c6f2\") " Oct 30 19:56:47.584876 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.584765 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-maas-api-openshift-ca\") pod \"4a3d573e-3eff-4eef-9c8a-164f7851c6f2\" (UID: \"4a3d573e-3eff-4eef-9c8a-164f7851c6f2\") " Oct 30 19:56:47.585342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.585061 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-kube-api-access-mqsz8\" (UniqueName: \"kubernetes.io/projected/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-kube-api-access-mqsz8\") pod \"4a3d573e-3eff-4eef-9c8a-164f7851c6f2\" (UID: \"4a3d573e-3eff-4eef-9c8a-164f7851c6f2\") " Oct 30 19:56:47.586009 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.585953 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-maas-api-openshift-ca" pod "4a3d573e-3eff-4eef-9c8a-164f7851c6f2" (UID: "4a3d573e-3eff-4eef-9c8a-164f7851c6f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:47.587370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.587329 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-maas-api-xks-ca" pod "4a3d573e-3eff-4eef-9c8a-164f7851c6f2" (UID: "4a3d573e-3eff-4eef-9c8a-164f7851c6f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:56:47.589588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.589563 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-kube-api-access-mqsz8" pod "4a3d573e-3eff-4eef-9c8a-164f7851c6f2" (UID: "4a3d573e-3eff-4eef-9c8a-164f7851c6f2"). InnerVolumeSpecName "kube-api-access-mqsz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:47.687710 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.687614 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:47.687710 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.687656 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqsz8\" (UniqueName: \"kubernetes.io/projected/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-kube-api-access-mqsz8\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:47.687710 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.687672 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a3d573e-3eff-4eef-9c8a-164f7851c6f2-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:47.870174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:47.856039 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:48.140777 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:48.140745 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" Oct 30 19:56:48.140777 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:48.140762 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2" event={"ID":"4a3d573e-3eff-4eef-9c8a-164f7851c6f2","Type":"ContainerDied","Data":"1f7bccf38e447832a892634ed2f6e99ffabcadf44df66699289e7dda3be36c84"} Oct 30 19:56:48.141330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:48.140807 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f7bccf38e447832a892634ed2f6e99ffabcadf44df66699289e7dda3be36c84" Oct 30 19:56:48.141330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:48.141172 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:48.142507 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:48.142487 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:56:48.581397 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:48.581330 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:48.761966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:48.761920 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:48.912449 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:48.912316 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:49.076067 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.076028 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:49.247245 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.247071 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:49.403487 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.403453 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m"] Oct 30 19:56:49.405096 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.405072 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4a3d573e-3eff-4eef-9c8a-164f7851c6f2" containerName="revoke-keys" Oct 30 19:56:49.405281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.405264 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a3d573e-3eff-4eef-9c8a-164f7851c6f2" containerName="revoke-keys" Oct 30 19:56:49.405627 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.405610 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a3d573e-3eff-4eef-9c8a-164f7851c6f2" containerName="revoke-keys" Oct 30 19:56:49.410736 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.410712 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.420444 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.420421 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m"] Oct 30 19:56:49.420927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.420906 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-54d4c4-openshift-default-dockercfg-khhgg\"" Oct 30 19:56:49.454309 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.454275 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:56:49.454496 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.454332 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:56:49.509629 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.509596 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:49.538479 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.536998 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj"] Oct 30 19:56:49.541716 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.541686 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-workload-socket\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.541839 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.541760 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj4vd\" (UniqueName: \"kubernetes.io/projected/c222e9d2-821b-4c1c-92d2-88297be8f52d-kube-api-access-wj4vd\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.541839 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.541816 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-podinfo\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.541967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.541947 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-workload-certs\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.542164 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.542126 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c222e9d2-821b-4c1c-92d2-88297be8f52d-istiod-ca-cert\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.542259 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.542222 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-credential-socket\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.542310 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.542264 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-token\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.542411 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.542389 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-data\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.542494 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.542436 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-envoy\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.648170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.643664 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c222e9d2-821b-4c1c-92d2-88297be8f52d-istiod-ca-cert\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.648170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.643735 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-credential-socket\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.648170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.643763 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-token\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.648170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.643803 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-data\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.648170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.643831 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-envoy\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.648170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.643888 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-workload-socket\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.648170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.643932 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj4vd\" (UniqueName: \"kubernetes.io/projected/c222e9d2-821b-4c1c-92d2-88297be8f52d-kube-api-access-wj4vd\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.648170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.643966 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-podinfo\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.648170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.644007 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-workload-certs\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.648170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.644458 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-workload-certs\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.648170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.644980 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c222e9d2-821b-4c1c-92d2-88297be8f52d-istiod-ca-cert\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.648170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.646345 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-credential-socket\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.648170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.646993 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-workload-socket\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.648170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.648071 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-data\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.655970 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.655922 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-envoy\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.697608 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.687408 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq"] Oct 30 19:56:49.697608 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.693436 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-podinfo\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.731592 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.731424 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:49.736785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.736728 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:49.751353 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.751322 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq"] Oct 30 19:56:49.753850 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.753732 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj4vd\" (UniqueName: \"kubernetes.io/projected/c222e9d2-821b-4c1c-92d2-88297be8f52d-kube-api-access-wj4vd\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.755333 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.755302 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-token\") pod \"e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:49.798684 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.798649 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-779fa3-openshift-default-dockercfg-ccggk\"" Oct 30 19:56:49.860463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.860422 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872504eb-20ef-411c-b526-61e05a15a9c4-istio-token\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:49.860659 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.860501 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-istio-envoy\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:49.860659 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.860597 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4645\" (UniqueName: \"kubernetes.io/projected/872504eb-20ef-411c-b526-61e05a15a9c4-kube-api-access-s4645\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:49.860843 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.860818 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-workload-socket\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:49.860913 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.860870 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-credential-socket\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:49.860975 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.860929 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872504eb-20ef-411c-b526-61e05a15a9c4-istio-podinfo\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:49.861038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.861014 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-workload-certs\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:49.861097 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.861079 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872504eb-20ef-411c-b526-61e05a15a9c4-istiod-ca-cert\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:49.861188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.861170 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-istio-data\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:49.993172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.970856 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-workload-socket\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:49.993172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.970908 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-credential-socket\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:49.993172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.970967 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872504eb-20ef-411c-b526-61e05a15a9c4-istio-podinfo\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:49.993172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.981626 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-workload-socket\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:49.993172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.981910 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-credential-socket\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:50.022214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.993680 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-workload-certs\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:50.022214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.993747 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872504eb-20ef-411c-b526-61e05a15a9c4-istiod-ca-cert\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:50.022214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.993827 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-istio-data\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:50.022214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.993884 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872504eb-20ef-411c-b526-61e05a15a9c4-istio-token\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:50.022214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.993939 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-istio-envoy\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:50.022214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.993990 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4645\" (UniqueName: \"kubernetes.io/projected/872504eb-20ef-411c-b526-61e05a15a9c4-kube-api-access-s4645\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:50.022214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.995559 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-istio-data\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:50.022214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.995785 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-workload-certs\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:50.022214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:49.996393 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872504eb-20ef-411c-b526-61e05a15a9c4-istiod-ca-cert\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:50.022214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.011898 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872504eb-20ef-411c-b526-61e05a15a9c4-istio-podinfo\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:50.037113 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.023832 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:50.037499 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.034470 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-istio-envoy\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:50.051246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.044743 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872504eb-20ef-411c-b526-61e05a15a9c4-istio-token\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:50.051246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.047957 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:50.051246 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.049821 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4645\" (UniqueName: \"kubernetes.io/projected/872504eb-20ef-411c-b526-61e05a15a9c4-kube-api-access-s4645\") pod \"e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:50.121857 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.121266 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:50.209364 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.209320 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:50.527120 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:56:50.527085 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod872504eb_20ef_411c_b526_61e05a15a9c4.slice/crio-40655185fad6456e015f451368ec7b4be22516ddd9520a7189dd6c702a308de0 WatchSource:0}: Error finding container 40655185fad6456e015f451368ec7b4be22516ddd9520a7189dd6c702a308de0: Status 404 returned error can't find the container with id 40655185fad6456e015f451368ec7b4be22516ddd9520a7189dd6c702a308de0 Oct 30 19:56:50.530418 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.530303 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:50.530418 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.530367 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:50.530418 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.530395 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:50.530717 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.530692 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq"] Oct 30 19:56:50.532365 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:56:50.532302 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc222e9d2_821b_4c1c_92d2_88297be8f52d.slice/crio-cbe66a692da1fa9bcf48c8e3d201084a6439889654b2c51eb89e60f56c459e7f WatchSource:0}: Error finding container cbe66a692da1fa9bcf48c8e3d201084a6439889654b2c51eb89e60f56c459e7f: Status 404 returned error can't find the container with id cbe66a692da1fa9bcf48c8e3d201084a6439889654b2c51eb89e60f56c459e7f Oct 30 19:56:50.535423 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.535389 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:50.535535 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.535466 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:50.535535 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.535511 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:50.536121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.536071 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m"] Oct 30 19:56:50.870219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:50.870002 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.213225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:51.209486 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.241168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:51.240574 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" event={"ID":"c222e9d2-821b-4c1c-92d2-88297be8f52d","Type":"ContainerStarted","Data":"7f5d4e0b0ede7cef957dc6d33cfe4195cd797972df41d573d4b6d2102d1f7dcc"} Oct 30 19:56:51.241168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:51.240646 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" event={"ID":"c222e9d2-821b-4c1c-92d2-88297be8f52d","Type":"ContainerStarted","Data":"cbe66a692da1fa9bcf48c8e3d201084a6439889654b2c51eb89e60f56c459e7f"} Oct 30 19:56:51.309209 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:51.304476 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" podStartSLOduration=2.3044592440000002 podStartE2EDuration="2.304459244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:51.297595051 +0000 UTC m=+1530.992803021" watchObservedRunningTime="2026-10-30 19:56:51.304459244 +0000 UTC m=+1530.999667203" Oct 30 19:56:51.309209 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:51.307163 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" event={"ID":"872504eb-20ef-411c-b526-61e05a15a9c4","Type":"ContainerStarted","Data":"8fdff38d73f5853ed799dedad4be18e5e92ca0a473371521325500a93cc767a7"} Oct 30 19:56:51.309209 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:51.307223 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" event={"ID":"872504eb-20ef-411c-b526-61e05a15a9c4","Type":"ContainerStarted","Data":"40655185fad6456e015f451368ec7b4be22516ddd9520a7189dd6c702a308de0"} Oct 30 19:56:51.582201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:51.579236 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.743138 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:51.743091 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:51.947222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:51.944649 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:52.056171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:52.053344 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:52.125281 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:52.118518 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:52.137122 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:52.128495 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:52.152762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:52.152518 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:52.154760 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:52.154713 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" podStartSLOduration=3.154698387 podStartE2EDuration="3.154698387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:56:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:56:51.364238774 +0000 UTC m=+1531.059446733" watchObservedRunningTime="2026-10-30 19:56:52.154698387 +0000 UTC m=+1531.849906361" Oct 30 19:56:52.319012 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:52.318972 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:52.319242 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:52.319036 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:52.319966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:52.319940 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:56:52.320075 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:52.320011 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:56:52.629657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:52.626974 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:53.339014 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:53.338978 5091 generic.go:356] "Generic (PLEG): container finished" podID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerID="6b71d3f026897317d1c110c31a1297a30e18c867e421af0910c645e42208eab3" exitCode=0 Oct 30 19:56:53.339310 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:53.339029 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" event={"ID":"7bcf3894-d14f-4ee3-a294-63daff87f18a","Type":"ContainerDied","Data":"6b71d3f026897317d1c110c31a1297a30e18c867e421af0910c645e42208eab3"} Oct 30 19:56:53.339599 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:53.339573 5091 scope.go:122] "RemoveContainer" containerID="6b71d3f026897317d1c110c31a1297a30e18c867e421af0910c645e42208eab3" Oct 30 19:56:53.342284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:53.341517 5091 generic.go:356] "Generic (PLEG): container finished" podID="f206f771-c29d-41db-8768-12d2feefe4e3" containerID="8165d039203b13ecfef0568b2ff185b4e100ab5341ebf9306bdf750f4cb14819" exitCode=0 Oct 30 19:56:53.342284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:53.341563 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:53.342284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:53.341642 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:53.342284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:53.341684 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:53.342284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:53.341796 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" event={"ID":"f206f771-c29d-41db-8768-12d2feefe4e3","Type":"ContainerDied","Data":"8165d039203b13ecfef0568b2ff185b4e100ab5341ebf9306bdf750f4cb14819"} Oct 30 19:56:53.342549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:53.342394 5091 scope.go:122] "RemoveContainer" containerID="8165d039203b13ecfef0568b2ff185b4e100ab5341ebf9306bdf750f4cb14819" Oct 30 19:56:53.344127 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:53.344103 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:53.344371 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:53.344342 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:53.344485 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:53.344400 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:56:53.535026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:53.534967 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:53.708232 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:53.706881 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:53.891747 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:53.891711 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:54.082776 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:54.082741 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:54.101776 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:54.101737 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2"] Oct 30 19:56:54.109480 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:54.109432 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-927bf8-swwt2"] Oct 30 19:56:54.279896 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:54.279846 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:54.400234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:54.388625 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" event={"ID":"f206f771-c29d-41db-8768-12d2feefe4e3","Type":"ContainerStarted","Data":"dca0e8d59dc560059e21e189ccf25b63fd4c4b307a1843209d4f15ca8cb47e60"} Oct 30 19:56:54.404359 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:54.404326 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" event={"ID":"7bcf3894-d14f-4ee3-a294-63daff87f18a","Type":"ContainerStarted","Data":"aa6a60c3ff94cfa847a6442dbaadfaae911b4937378cfb0a6e490662bb81709e"} Oct 30 19:56:54.443431 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:54.443403 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a3d573e-3eff-4eef-9c8a-164f7851c6f2" path="/var/lib/kubelet/pods/4a3d573e-3eff-4eef-9c8a-164f7851c6f2/volumes" Oct 30 19:56:54.468745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:54.468701 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:54.537801 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:54.537733 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" podUID="b4277311-2a23-4736-99a8-29b0690cbeb4" containerName="payload-processing" containerID="cri-o://81cc912fabb0c5ced91ae414fd9f5ba788b619d6e8f9360f9cf785e603abab65" gracePeriod=25 Oct 30 19:56:54.679263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:54.679048 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:54.857732 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:54.848192 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:54.857732 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:54.849449 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 19:56:54.857732 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:54.849498 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" podUID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 19:56:55.023100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.023010 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:56:55.023424 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.023402 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 19:56:55.023563 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.023542 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 19:56:55.030114 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.030039 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" Oct 30 19:56:55.121906 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.121851 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:55.138225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.132740 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4277311-2a23-4736-99a8-29b0690cbeb4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4277311-2a23-4736-99a8-29b0690cbeb4-plugins-config-volume\") pod \"b4277311-2a23-4736-99a8-29b0690cbeb4\" (UID: \"b4277311-2a23-4736-99a8-29b0690cbeb4\") " Oct 30 19:56:55.138225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.133370 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4277311-2a23-4736-99a8-29b0690cbeb4-kube-api-access-72z9q\" (UniqueName: \"kubernetes.io/projected/b4277311-2a23-4736-99a8-29b0690cbeb4-kube-api-access-72z9q\") pod \"b4277311-2a23-4736-99a8-29b0690cbeb4\" (UID: \"b4277311-2a23-4736-99a8-29b0690cbeb4\") " Oct 30 19:56:55.138225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.134638 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4277311-2a23-4736-99a8-29b0690cbeb4-plugins-config-volume" pod "b4277311-2a23-4736-99a8-29b0690cbeb4" (UID: "b4277311-2a23-4736-99a8-29b0690cbeb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:56:55.138225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.136694 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4277311-2a23-4736-99a8-29b0690cbeb4-kube-api-access-72z9q" pod "b4277311-2a23-4736-99a8-29b0690cbeb4" (UID: "b4277311-2a23-4736-99a8-29b0690cbeb4"). InnerVolumeSpecName "kube-api-access-72z9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:56:55.420489 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.311065 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:55.420489 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.413531 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4277311-2a23-4736-99a8-29b0690cbeb4-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:55.420489 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.413559 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72z9q\" (UniqueName: \"kubernetes.io/projected/b4277311-2a23-4736-99a8-29b0690cbeb4-kube-api-access-72z9q\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:56:55.625992 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.625960 5091 generic.go:356] "Generic (PLEG): container finished" podID="b4277311-2a23-4736-99a8-29b0690cbeb4" containerID="81cc912fabb0c5ced91ae414fd9f5ba788b619d6e8f9360f9cf785e603abab65" exitCode=0 Oct 30 19:56:55.627354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.627321 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" event={"ID":"b4277311-2a23-4736-99a8-29b0690cbeb4","Type":"ContainerDied","Data":"81cc912fabb0c5ced91ae414fd9f5ba788b619d6e8f9360f9cf785e603abab65"} Oct 30 19:56:55.627437 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.627393 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" event={"ID":"b4277311-2a23-4736-99a8-29b0690cbeb4","Type":"ContainerDied","Data":"3f46e8edb8e9c63055c2c8d88eb9e769afa748601d10e3e22cab4693f82abee7"} Oct 30 19:56:55.627437 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.627418 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj" Oct 30 19:56:55.627527 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.627424 5091 scope.go:122] "RemoveContainer" containerID="81cc912fabb0c5ced91ae414fd9f5ba788b619d6e8f9360f9cf785e603abab65" Oct 30 19:56:55.714202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.711642 5091 scope.go:122] "RemoveContainer" containerID="81cc912fabb0c5ced91ae414fd9f5ba788b619d6e8f9360f9cf785e603abab65" Oct 30 19:56:55.714202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.711766 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj"] Oct 30 19:56:55.716472 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.716335 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-927bf8-6df68dcfdd-6rppj"] Oct 30 19:56:55.716793 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:56:55.716737 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81cc912fabb0c5ced91ae414fd9f5ba788b619d6e8f9360f9cf785e603abab65\": container with ID starting with 81cc912fabb0c5ced91ae414fd9f5ba788b619d6e8f9360f9cf785e603abab65 not found: ID does not exist" containerID="81cc912fabb0c5ced91ae414fd9f5ba788b619d6e8f9360f9cf785e603abab65" Oct 30 19:56:55.716899 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.716782 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81cc912fabb0c5ced91ae414fd9f5ba788b619d6e8f9360f9cf785e603abab65"} err="failed to get container status \"81cc912fabb0c5ced91ae414fd9f5ba788b619d6e8f9360f9cf785e603abab65\": rpc error: code = NotFound desc = could not find container \"81cc912fabb0c5ced91ae414fd9f5ba788b619d6e8f9360f9cf785e603abab65\": container with ID starting with 81cc912fabb0c5ced91ae414fd9f5ba788b619d6e8f9360f9cf785e603abab65 not found: ID does not exist" Oct 30 19:56:55.881427 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.881388 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 19:56:55.882279 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:55.882242 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" podUID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 19:56:56.017317 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:56.014454 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:56.022706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:56.022675 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 19:56:56.022875 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:56.022732 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 19:56:56.128691 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:56.128445 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" Oct 30 19:56:56.134808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:56.133619 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" Oct 30 19:56:56.222227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:56.218665 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:56.422316 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:56.418168 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:56.620575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:56.616496 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4277311-2a23-4736-99a8-29b0690cbeb4" path="/var/lib/kubelet/pods/b4277311-2a23-4736-99a8-29b0690cbeb4/volumes" Oct 30 19:56:56.620575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:56.616995 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:56:56.852076 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:56.852043 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 19:56:56.852406 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:56.852376 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" podUID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 19:56:57.022283 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:57.022234 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 19:56:57.022662 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:57.022294 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 19:56:57.421123 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:57.421083 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:57.851171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:57.850140 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 19:56:57.851171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:57.850223 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" podUID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 19:56:58.023565 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:58.023525 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 19:56:58.024044 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:58.023582 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 19:56:58.279418 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:58.279378 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:58.856554 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:58.856510 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 19:56:58.856873 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:58.856846 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" podUID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 19:56:59.022418 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:59.022386 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 19:56:59.022605 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:59.022438 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 19:56:59.087173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:59.086497 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:56:59.860178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:59.859760 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Oct 30 19:56:59.860178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:56:59.859837 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" podUID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Oct 30 19:57:00.038212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:00.037274 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" start-of-body= Oct 30 19:57:00.038212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:00.037344 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.209:15021/healthz/ready\": dial tcp 10.132.0.209:15021: connect: connection refused" Oct 30 19:57:00.133799 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:00.133617 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:00.314300 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:00.314263 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:00.463620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:00.463403 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:00.862301 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:00.862269 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:57:00.863590 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:00.863567 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:57:01.026887 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:01.026857 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:57:01.027361 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:01.027342 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:57:01.028054 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:01.028037 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 19:57:01.381178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:01.373138 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:02.315343 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:02.315205 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:02.883673 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:02.881969 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:03.026353 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.026308 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:03.244582 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.244493 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:03.439304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.439271 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:03.722831 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.722797 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6"] Oct 30 19:57:03.723518 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.723503 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b4277311-2a23-4736-99a8-29b0690cbeb4" containerName="payload-processing" Oct 30 19:57:03.723578 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.723520 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4277311-2a23-4736-99a8-29b0690cbeb4" containerName="payload-processing" Oct 30 19:57:03.723631 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.723621 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4277311-2a23-4736-99a8-29b0690cbeb4" containerName="payload-processing" Oct 30 19:57:03.727364 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.727346 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" Oct 30 19:57:03.742690 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.742658 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:57:03.743033 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.743008 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:57:03.767188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.767160 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6"] Oct 30 19:57:03.806753 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.806713 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b79555cf-280f-4128-9ee3-f28fb48815a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6\" (UID: \"b79555cf-280f-4128-9ee3-f28fb48815a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" Oct 30 19:57:03.806919 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.806897 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcwcj\" (UniqueName: \"kubernetes.io/projected/b79555cf-280f-4128-9ee3-f28fb48815a0-kube-api-access-gcwcj\") pod \"maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6\" (UID: \"b79555cf-280f-4128-9ee3-f28fb48815a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" Oct 30 19:57:03.807070 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.807055 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b79555cf-280f-4128-9ee3-f28fb48815a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6\" (UID: \"b79555cf-280f-4128-9ee3-f28fb48815a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" Oct 30 19:57:03.908802 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.908769 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b79555cf-280f-4128-9ee3-f28fb48815a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6\" (UID: \"b79555cf-280f-4128-9ee3-f28fb48815a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" Oct 30 19:57:03.909207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.908926 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcwcj\" (UniqueName: \"kubernetes.io/projected/b79555cf-280f-4128-9ee3-f28fb48815a0-kube-api-access-gcwcj\") pod \"maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6\" (UID: \"b79555cf-280f-4128-9ee3-f28fb48815a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" Oct 30 19:57:03.909207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.909121 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b79555cf-280f-4128-9ee3-f28fb48815a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6\" (UID: \"b79555cf-280f-4128-9ee3-f28fb48815a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" Oct 30 19:57:03.909524 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.909504 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b79555cf-280f-4128-9ee3-f28fb48815a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6\" (UID: \"b79555cf-280f-4128-9ee3-f28fb48815a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" Oct 30 19:57:03.911675 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.911654 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b79555cf-280f-4128-9ee3-f28fb48815a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6\" (UID: \"b79555cf-280f-4128-9ee3-f28fb48815a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" Oct 30 19:57:03.923504 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:03.923470 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcwcj\" (UniqueName: \"kubernetes.io/projected/b79555cf-280f-4128-9ee3-f28fb48815a0-kube-api-access-gcwcj\") pod \"maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6\" (UID: \"b79555cf-280f-4128-9ee3-f28fb48815a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" Oct 30 19:57:04.064076 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:04.064045 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" Oct 30 19:57:04.528171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:04.507752 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6"] Oct 30 19:57:04.784454 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:04.771775 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:04.848170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:04.847888 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" event={"ID":"b79555cf-280f-4128-9ee3-f28fb48815a0","Type":"ContainerStarted","Data":"e0eaf2f9b470cf3e40c46359d16756e90de915cbb838a01bb616a440626e32b9"} Oct 30 19:57:04.995834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:04.995800 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:05.862778 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:05.862738 5091 generic.go:356] "Generic (PLEG): container finished" podID="b79555cf-280f-4128-9ee3-f28fb48815a0" containerID="e7b03ad774680fa3712dbd4480b3a4fac8f5d21c8191ca2470b971f08a1b1fe3" exitCode=0 Oct 30 19:57:05.863017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:05.862999 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" event={"ID":"b79555cf-280f-4128-9ee3-f28fb48815a0","Type":"ContainerDied","Data":"e7b03ad774680fa3712dbd4480b3a4fac8f5d21c8191ca2470b971f08a1b1fe3"} Oct 30 19:57:06.038672 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:06.038536 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:06.229628 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:06.229592 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:07.097792 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:07.097754 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:07.283990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:07.283965 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" Oct 30 19:57:07.314773 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:07.314745 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b79555cf-280f-4128-9ee3-f28fb48815a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b79555cf-280f-4128-9ee3-f28fb48815a0-maas-api-xks-ca\") pod \"b79555cf-280f-4128-9ee3-f28fb48815a0\" (UID: \"b79555cf-280f-4128-9ee3-f28fb48815a0\") " Oct 30 19:57:07.314917 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:07.314876 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b79555cf-280f-4128-9ee3-f28fb48815a0-kube-api-access-gcwcj\" (UniqueName: \"kubernetes.io/projected/b79555cf-280f-4128-9ee3-f28fb48815a0-kube-api-access-gcwcj\") pod \"b79555cf-280f-4128-9ee3-f28fb48815a0\" (UID: \"b79555cf-280f-4128-9ee3-f28fb48815a0\") " Oct 30 19:57:07.314990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:07.314930 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b79555cf-280f-4128-9ee3-f28fb48815a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b79555cf-280f-4128-9ee3-f28fb48815a0-maas-api-openshift-ca\") pod \"b79555cf-280f-4128-9ee3-f28fb48815a0\" (UID: \"b79555cf-280f-4128-9ee3-f28fb48815a0\") " Oct 30 19:57:07.315400 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:07.315370 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b79555cf-280f-4128-9ee3-f28fb48815a0-maas-api-openshift-ca" pod "b79555cf-280f-4128-9ee3-f28fb48815a0" (UID: "b79555cf-280f-4128-9ee3-f28fb48815a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:07.315531 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:07.315511 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b79555cf-280f-4128-9ee3-f28fb48815a0-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:07.317334 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:07.317307 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b79555cf-280f-4128-9ee3-f28fb48815a0-maas-api-xks-ca" pod "b79555cf-280f-4128-9ee3-f28fb48815a0" (UID: "b79555cf-280f-4128-9ee3-f28fb48815a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:07.317334 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:07.317323 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79555cf-280f-4128-9ee3-f28fb48815a0-kube-api-access-gcwcj" pod "b79555cf-280f-4128-9ee3-f28fb48815a0" (UID: "b79555cf-280f-4128-9ee3-f28fb48815a0"). InnerVolumeSpecName "kube-api-access-gcwcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:07.417609 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:07.417569 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b79555cf-280f-4128-9ee3-f28fb48815a0-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:07.417609 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:07.417604 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcwcj\" (UniqueName: \"kubernetes.io/projected/b79555cf-280f-4128-9ee3-f28fb48815a0-kube-api-access-gcwcj\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:07.707722 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:07.707634 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:07.982243 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:07.982135 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" Oct 30 19:57:07.992088 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:07.981944 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6" event={"ID":"b79555cf-280f-4128-9ee3-f28fb48815a0","Type":"ContainerDied","Data":"e0eaf2f9b470cf3e40c46359d16756e90de915cbb838a01bb616a440626e32b9"} Oct 30 19:57:07.992376 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:07.992348 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0eaf2f9b470cf3e40c46359d16756e90de915cbb838a01bb616a440626e32b9" Oct 30 19:57:08.134108 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.134067 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:08.415452 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.415403 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:08.575169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.574396 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv"] Oct 30 19:57:08.589901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.575464 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b79555cf-280f-4128-9ee3-f28fb48815a0" containerName="revoke-keys" Oct 30 19:57:08.589901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.583292 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b79555cf-280f-4128-9ee3-f28fb48815a0" containerName="revoke-keys" Oct 30 19:57:08.589901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.583606 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b79555cf-280f-4128-9ee3-f28fb48815a0" containerName="revoke-keys" Oct 30 19:57:08.592513 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.592461 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.600007 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.599976 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv"] Oct 30 19:57:08.612557 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.607820 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-7cbcbb-openshift-default-dockercfg-ggq85\"" Oct 30 19:57:08.657131 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.657084 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-envoy\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.657351 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.657252 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wbmr\" (UniqueName: \"kubernetes.io/projected/9d750d29-4fd8-4bd5-b879-ff33ba319afc-kube-api-access-9wbmr\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.657351 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.657314 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-podinfo\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.657522 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.657484 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-workload-socket\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.657588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.657565 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-workload-certs\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.657650 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.657602 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-credential-socket\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.657650 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.657627 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-token\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.657767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.657660 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-data\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.657767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.657685 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istiod-ca-cert\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.692762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.692031 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:08.768181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.768001 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wbmr\" (UniqueName: \"kubernetes.io/projected/9d750d29-4fd8-4bd5-b879-ff33ba319afc-kube-api-access-9wbmr\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.768181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.768071 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-podinfo\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.768471 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.768270 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-workload-socket\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.768471 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.768331 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-workload-certs\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.768471 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.768364 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-credential-socket\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.768471 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.768388 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-token\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.768471 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.768417 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-data\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.768471 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.768443 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istiod-ca-cert\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.768471 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.768473 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-envoy\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.781743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.781705 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-workload-certs\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.782368 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.782340 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-credential-socket\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.783078 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.783054 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-data\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.794697 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.793875 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istiod-ca-cert\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.794880 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.794084 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-envoy\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.797483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.797453 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-workload-socket\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.801356 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.801306 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wbmr\" (UniqueName: \"kubernetes.io/projected/9d750d29-4fd8-4bd5-b879-ff33ba319afc-kube-api-access-9wbmr\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.802044 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.802023 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-token\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.818173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.817377 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-podinfo\") pod \"e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:08.879183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.872568 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:08.948252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:08.948164 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:09.380377 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:09.380096 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv"] Oct 30 19:57:09.383680 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:57:09.383651 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d750d29_4fd8_4bd5_b879_ff33ba319afc.slice/crio-3aa850ecf43c5af6c4530114576fb562aa96bb53627dd78a2b012385ac112e65 WatchSource:0}: Error finding container 3aa850ecf43c5af6c4530114576fb562aa96bb53627dd78a2b012385ac112e65: Status 404 returned error can't find the container with id 3aa850ecf43c5af6c4530114576fb562aa96bb53627dd78a2b012385ac112e65 Oct 30 19:57:09.387175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:09.387105 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:09.387255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:09.387207 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:09.387312 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:09.387252 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:09.694630 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:09.689907 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:10.039720 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:10.039685 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" event={"ID":"9d750d29-4fd8-4bd5-b879-ff33ba319afc","Type":"ContainerStarted","Data":"03b46fa6422e9f1fa0bfffa7e75b2f5fb1197c945e6d84456ea2933b605dae84"} Oct 30 19:57:10.039882 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:10.039742 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" event={"ID":"9d750d29-4fd8-4bd5-b879-ff33ba319afc","Type":"ContainerStarted","Data":"3aa850ecf43c5af6c4530114576fb562aa96bb53627dd78a2b012385ac112e65"} Oct 30 19:57:10.077170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:10.071793 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" podStartSLOduration=2.071774813 podStartE2EDuration="2.071774813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:10.070695765 +0000 UTC m=+1549.765903724" watchObservedRunningTime="2026-10-30 19:57:10.071774813 +0000 UTC m=+1549.766982786" Oct 30 19:57:10.628175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:10.627956 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:10.828186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:10.828139 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:10.952217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:10.949443 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:10.976166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:10.966416 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:11.072173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:11.070496 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:11.079252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:11.079209 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:57:11.715171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:11.714091 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:12.597160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:12.597105 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:12.933734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:12.933656 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:13.143526 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:13.143491 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:13.943667 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:13.943640 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:14.138824 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:14.138784 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:14.605157 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:14.605107 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6"] Oct 30 19:57:14.621438 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:14.621406 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v"] Oct 30 19:57:14.787598 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:14.787561 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:15.107901 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:15.107866 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:15.299966 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:15.299924 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:15.751422 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:15.751388 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:15.978084 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:57:15.978046 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:57:16.585547 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:16.585484 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:16.850057 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:16.849938 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:17.000199 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:16.999258 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:17.012991 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:17.012954 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6"] Oct 30 19:57:17.020833 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:17.020802 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-c1768b-7f8f6"] Oct 30 19:57:17.260687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:17.260656 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:17.620443 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:17.620348 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:18.150737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:18.150671 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:18.442136 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:18.442064 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b79555cf-280f-4128-9ee3-f28fb48815a0" path="/var/lib/kubelet/pods/b79555cf-280f-4128-9ee3-f28fb48815a0/volumes" Oct 30 19:57:19.197939 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:19.197725 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:19.285625 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:19.285594 5091 generic.go:356] "Generic (PLEG): container finished" podID="0fbaef91-57be-4511-a038-a607def818b5" containerID="928ff57d39d92e480d79933914c6f7521efe79ac550a7792249ed935e9647e38" exitCode=0 Oct 30 19:57:19.285799 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:19.285626 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" event={"ID":"0fbaef91-57be-4511-a038-a607def818b5","Type":"ContainerDied","Data":"928ff57d39d92e480d79933914c6f7521efe79ac550a7792249ed935e9647e38"} Oct 30 19:57:19.286035 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:19.286021 5091 scope.go:122] "RemoveContainer" containerID="928ff57d39d92e480d79933914c6f7521efe79ac550a7792249ed935e9647e38" Oct 30 19:57:19.287666 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:19.287637 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:19.287790 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:19.287706 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:19.287790 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:19.287742 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:19.606408 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:19.606288 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" podUID="5c22cd7b-df94-4afa-83be-594cee6a2701" containerName="payload-processing" containerID="cri-o://b5f98e6fc7b696628abac67e541937c78cd88c87f10152dd36323c9d8c355067" gracePeriod=25 Oct 30 19:57:19.622367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:19.622330 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" podUID="763c8469-7822-4c84-ab15-67e20ddbc251" containerName="payload-pre-processing" containerID="cri-o://05e380c6af095e88002c374f23b853401027802039d92e24cde4da6b8b9a8412" gracePeriod=25 Oct 30 19:57:19.733718 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:57:19.733688 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763c8469_7822_4c84_ab15_67e20ddbc251.slice/crio-conmon-05e380c6af095e88002c374f23b853401027802039d92e24cde4da6b8b9a8412.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763c8469_7822_4c84_ab15_67e20ddbc251.slice/crio-05e380c6af095e88002c374f23b853401027802039d92e24cde4da6b8b9a8412.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:57:19.908387 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:19.904207 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:20.103423 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.074762 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" Oct 30 19:57:20.227221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.210588 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c22cd7b-df94-4afa-83be-594cee6a2701-kube-api-access-rkjp6\" (UniqueName: \"kubernetes.io/projected/5c22cd7b-df94-4afa-83be-594cee6a2701-kube-api-access-rkjp6\") pod \"5c22cd7b-df94-4afa-83be-594cee6a2701\" (UID: \"5c22cd7b-df94-4afa-83be-594cee6a2701\") " Oct 30 19:57:20.227221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.210820 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c22cd7b-df94-4afa-83be-594cee6a2701-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c22cd7b-df94-4afa-83be-594cee6a2701-plugins-config-volume\") pod \"5c22cd7b-df94-4afa-83be-594cee6a2701\" (UID: \"5c22cd7b-df94-4afa-83be-594cee6a2701\") " Oct 30 19:57:20.231165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.230375 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c22cd7b-df94-4afa-83be-594cee6a2701-plugins-config-volume" pod "5c22cd7b-df94-4afa-83be-594cee6a2701" (UID: "5c22cd7b-df94-4afa-83be-594cee6a2701"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:20.240382 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.240332 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c22cd7b-df94-4afa-83be-594cee6a2701-kube-api-access-rkjp6" pod "5c22cd7b-df94-4afa-83be-594cee6a2701" (UID: "5c22cd7b-df94-4afa-83be-594cee6a2701"). InnerVolumeSpecName "kube-api-access-rkjp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:20.298806 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.298616 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:20.314907 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.312715 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c22cd7b-df94-4afa-83be-594cee6a2701-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:20.314907 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.312808 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkjp6\" (UniqueName: \"kubernetes.io/projected/5c22cd7b-df94-4afa-83be-594cee6a2701-kube-api-access-rkjp6\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:20.385170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.371309 5091 generic.go:356] "Generic (PLEG): container finished" podID="5c22cd7b-df94-4afa-83be-594cee6a2701" containerID="b5f98e6fc7b696628abac67e541937c78cd88c87f10152dd36323c9d8c355067" exitCode=0 Oct 30 19:57:20.385170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.371437 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" event={"ID":"5c22cd7b-df94-4afa-83be-594cee6a2701","Type":"ContainerDied","Data":"b5f98e6fc7b696628abac67e541937c78cd88c87f10152dd36323c9d8c355067"} Oct 30 19:57:20.385170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.371494 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" event={"ID":"5c22cd7b-df94-4afa-83be-594cee6a2701","Type":"ContainerDied","Data":"dee82afc04527585bf008d8c656ad7f4e8c9ffea77effcab757ef10496a6cf8d"} Oct 30 19:57:20.385170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.371525 5091 scope.go:122] "RemoveContainer" containerID="b5f98e6fc7b696628abac67e541937c78cd88c87f10152dd36323c9d8c355067" Oct 30 19:57:20.385170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.371697 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6" Oct 30 19:57:20.400484 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.400445 5091 generic.go:356] "Generic (PLEG): container finished" podID="763c8469-7822-4c84-ab15-67e20ddbc251" containerID="05e380c6af095e88002c374f23b853401027802039d92e24cde4da6b8b9a8412" exitCode=0 Oct 30 19:57:20.400667 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.400621 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" event={"ID":"763c8469-7822-4c84-ab15-67e20ddbc251","Type":"ContainerDied","Data":"05e380c6af095e88002c374f23b853401027802039d92e24cde4da6b8b9a8412"} Oct 30 19:57:20.403574 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.403545 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" event={"ID":"0fbaef91-57be-4511-a038-a607def818b5","Type":"ContainerStarted","Data":"3d02e820397d0c5e6c9b8d41196e997da72b575f0aa1c3b5168d254ab55841ec"} Oct 30 19:57:20.408416 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.408396 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" Oct 30 19:57:20.417011 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.416978 5091 scope.go:122] "RemoveContainer" containerID="b5f98e6fc7b696628abac67e541937c78cd88c87f10152dd36323c9d8c355067" Oct 30 19:57:20.418596 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:57:20.418393 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5f98e6fc7b696628abac67e541937c78cd88c87f10152dd36323c9d8c355067\": container with ID starting with b5f98e6fc7b696628abac67e541937c78cd88c87f10152dd36323c9d8c355067 not found: ID does not exist" containerID="b5f98e6fc7b696628abac67e541937c78cd88c87f10152dd36323c9d8c355067" Oct 30 19:57:20.418596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.418434 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5f98e6fc7b696628abac67e541937c78cd88c87f10152dd36323c9d8c355067"} err="failed to get container status \"b5f98e6fc7b696628abac67e541937c78cd88c87f10152dd36323c9d8c355067\": rpc error: code = NotFound desc = could not find container \"b5f98e6fc7b696628abac67e541937c78cd88c87f10152dd36323c9d8c355067\": container with ID starting with b5f98e6fc7b696628abac67e541937c78cd88c87f10152dd36323c9d8c355067 not found: ID does not exist" Oct 30 19:57:20.501201 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.501135 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:20.505493 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.505466 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6"] Oct 30 19:57:20.509801 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.509772 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-c1768b-7784fb67fb-jnzk6"] Oct 30 19:57:20.518928 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.518904 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763c8469-7822-4c84-ab15-67e20ddbc251-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763c8469-7822-4c84-ab15-67e20ddbc251-plugins-config-volume\") pod \"763c8469-7822-4c84-ab15-67e20ddbc251\" (UID: \"763c8469-7822-4c84-ab15-67e20ddbc251\") " Oct 30 19:57:20.519188 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.519168 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763c8469-7822-4c84-ab15-67e20ddbc251-kube-api-access-jt8zz\" (UniqueName: \"kubernetes.io/projected/763c8469-7822-4c84-ab15-67e20ddbc251-kube-api-access-jt8zz\") pod \"763c8469-7822-4c84-ab15-67e20ddbc251\" (UID: \"763c8469-7822-4c84-ab15-67e20ddbc251\") " Oct 30 19:57:20.519367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.519345 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763c8469-7822-4c84-ab15-67e20ddbc251-plugins-config-volume" pod "763c8469-7822-4c84-ab15-67e20ddbc251" (UID: "763c8469-7822-4c84-ab15-67e20ddbc251"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:20.519856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.519830 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/763c8469-7822-4c84-ab15-67e20ddbc251-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:20.521602 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.521574 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763c8469-7822-4c84-ab15-67e20ddbc251-kube-api-access-jt8zz" pod "763c8469-7822-4c84-ab15-67e20ddbc251" (UID: "763c8469-7822-4c84-ab15-67e20ddbc251"). InnerVolumeSpecName "kube-api-access-jt8zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:20.620797 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.620763 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt8zz\" (UniqueName: \"kubernetes.io/projected/763c8469-7822-4c84-ab15-67e20ddbc251-kube-api-access-jt8zz\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:20.743848 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.743812 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f"] Oct 30 19:57:20.744964 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.744937 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5c22cd7b-df94-4afa-83be-594cee6a2701" containerName="payload-processing" Oct 30 19:57:20.744964 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.744962 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c22cd7b-df94-4afa-83be-594cee6a2701" containerName="payload-processing" Oct 30 19:57:20.745182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.744983 5091 cpu_manager.go:414] "containerMap: removing container" podUID="763c8469-7822-4c84-ab15-67e20ddbc251" containerName="payload-pre-processing" Oct 30 19:57:20.745182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.744989 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763c8469-7822-4c84-ab15-67e20ddbc251" containerName="payload-pre-processing" Oct 30 19:57:20.745182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.745127 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="763c8469-7822-4c84-ab15-67e20ddbc251" containerName="payload-pre-processing" Oct 30 19:57:20.745419 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.745204 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c22cd7b-df94-4afa-83be-594cee6a2701" containerName="payload-processing" Oct 30 19:57:20.794934 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.794797 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f"] Oct 30 19:57:20.795070 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.794945 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.803025 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.802975 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-981ee5-openshift-default-dockercfg-dqzjm\"" Oct 30 19:57:20.823423 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.823387 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-envoy\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.823553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.823431 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-workload-socket\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.823553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.823464 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-credential-socket\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.823553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.823539 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-token\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.823690 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.823665 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96mkg\" (UniqueName: \"kubernetes.io/projected/3d5e6444-233c-4c4e-b82f-559b46e46a38-kube-api-access-96mkg\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.823732 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.823698 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-podinfo\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.823819 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.823806 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-data\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.823870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.823827 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5e6444-233c-4c4e-b82f-559b46e46a38-istiod-ca-cert\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.823870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.823843 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-workload-certs\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.841796 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.841772 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:57:20.871171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.863235 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:20.943171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.934561 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-envoy\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.943171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.934614 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-workload-socket\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.943171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.934675 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-credential-socket\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.943171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.934725 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-token\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.943171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.934928 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96mkg\" (UniqueName: \"kubernetes.io/projected/3d5e6444-233c-4c4e-b82f-559b46e46a38-kube-api-access-96mkg\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.943171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.934991 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-podinfo\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.943171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.935344 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-workload-socket\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.970944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.967385 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-data\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.970944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.967456 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5e6444-233c-4c4e-b82f-559b46e46a38-istiod-ca-cert\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.970944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.967486 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-workload-certs\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:20.993227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.992818 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-data\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:21.021491 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.994429 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5e6444-233c-4c4e-b82f-559b46e46a38-istiod-ca-cert\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:21.021491 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:20.994710 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-credential-socket\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:21.021491 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.020801 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-workload-certs\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:21.070214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.062012 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-token\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:21.116737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.104461 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:57:21.116737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.104663 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 19:57:21.116737 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.104715 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" podUID="0fbaef91-57be-4511-a038-a607def818b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 19:57:21.140546 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.135273 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-envoy\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:21.159318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.141485 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:21.160477 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.160386 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96mkg\" (UniqueName: \"kubernetes.io/projected/3d5e6444-233c-4c4e-b82f-559b46e46a38-kube-api-access-96mkg\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:21.177539 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.172041 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-podinfo\") pod \"e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:21.427533 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.427449 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:21.527175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.526845 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" event={"ID":"763c8469-7822-4c84-ab15-67e20ddbc251","Type":"ContainerDied","Data":"edebbc708ae30bf77b17d6d7bac1c33be6ed51e909f841544795ddff72552bfb"} Oct 30 19:57:21.527175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.526912 5091 scope.go:122] "RemoveContainer" containerID="05e380c6af095e88002c374f23b853401027802039d92e24cde4da6b8b9a8412" Oct 30 19:57:21.559437 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.555163 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v" Oct 30 19:57:21.732159 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.732111 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v"] Oct 30 19:57:21.741810 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.741770 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-c1768b-5dd5d79667-94d6v"] Oct 30 19:57:21.911771 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:21.911655 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:22.016673 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:57:22.016640 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5e6444_233c_4c4e_b82f_559b46e46a38.slice/crio-98d84e65a7f29b73c729921679f7e179a81a5ce9192267620324f7317626e6be WatchSource:0}: Error finding container 98d84e65a7f29b73c729921679f7e179a81a5ce9192267620324f7317626e6be: Status 404 returned error can't find the container with id 98d84e65a7f29b73c729921679f7e179a81a5ce9192267620324f7317626e6be Oct 30 19:57:22.019417 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:22.019393 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f"] Oct 30 19:57:22.019519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:22.019490 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:22.019562 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:22.019551 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:22.019597 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:22.019579 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:22.042817 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:22.042794 5091 scope.go:122] "RemoveContainer" containerID="04e995e6dfbf9ca2835804ac9d7aaa59daff860111b95a9b05bfaeffce314786" Oct 30 19:57:22.081183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:22.081161 5091 scope.go:122] "RemoveContainer" containerID="891af3d7fa9403472eb00775eaae66bae9ebb0b6124cfc54085d05c40a16d42f" Oct 30 19:57:22.093626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:22.093602 5091 scope.go:122] "RemoveContainer" containerID="1cd417639ed5bb86c22564f705d0d4650d37285688bd6915e91b3428ee192c04" Oct 30 19:57:22.102627 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:22.102598 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 19:57:22.102767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:22.102656 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" podUID="0fbaef91-57be-4511-a038-a607def818b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 19:57:22.119602 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:22.119276 5091 scope.go:122] "RemoveContainer" containerID="c1ab07600d46f69733408a5b8f19d7c99b299b410c4ec5255b15210fe5291698" Oct 30 19:57:22.446225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:22.445289 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c22cd7b-df94-4afa-83be-594cee6a2701" path="/var/lib/kubelet/pods/5c22cd7b-df94-4afa-83be-594cee6a2701/volumes" Oct 30 19:57:22.446225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:22.445905 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763c8469-7822-4c84-ab15-67e20ddbc251" path="/var/lib/kubelet/pods/763c8469-7822-4c84-ab15-67e20ddbc251/volumes" Oct 30 19:57:22.537354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:22.537323 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" event={"ID":"3d5e6444-233c-4c4e-b82f-559b46e46a38","Type":"ContainerStarted","Data":"4ca9093b95260f9f6ff042b6e39d3dac8c32c65ef715112cebd541e805140ce5"} Oct 30 19:57:22.537528 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:22.537380 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" event={"ID":"3d5e6444-233c-4c4e-b82f-559b46e46a38","Type":"ContainerStarted","Data":"98d84e65a7f29b73c729921679f7e179a81a5ce9192267620324f7317626e6be"} Oct 30 19:57:22.589190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:22.589108 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" podStartSLOduration=2.589092434 podStartE2EDuration="2.589092434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:22.586698145 +0000 UTC m=+1562.281906103" watchObservedRunningTime="2026-10-30 19:57:22.589092434 +0000 UTC m=+1562.284300391" Oct 30 19:57:23.101881 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:23.101843 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 19:57:23.102060 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:23.101905 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" podUID="0fbaef91-57be-4511-a038-a607def818b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 19:57:23.189180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:23.186527 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:23.432384 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:23.432298 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:23.438688 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:23.438660 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:23.563171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:23.563025 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:23.575170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:23.570368 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:57:24.059362 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:24.059331 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc"] Oct 30 19:57:24.078173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:24.072068 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2ed83de-zx5cc"] Oct 30 19:57:24.102350 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:24.102316 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 19:57:24.102505 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:24.102374 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" podUID="0fbaef91-57be-4511-a038-a607def818b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 19:57:24.271170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:24.270729 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:24.443267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:24.443192 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4220b4fa-54d6-4fc4-bc66-24f89798c1bf" path="/var/lib/kubelet/pods/4220b4fa-54d6-4fc4-bc66-24f89798c1bf/volumes" Oct 30 19:57:24.473169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:24.472654 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:25.088954 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:25.088806 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:25.102444 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:25.102411 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" start-of-body= Oct 30 19:57:25.102612 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:25.102467 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" podUID="0fbaef91-57be-4511-a038-a607def818b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.212:15021/healthz/ready\": dial tcp 10.132.0.212:15021: connect: connection refused" Oct 30 19:57:25.485487 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:25.484007 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:25.668904 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:25.668735 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:26.106333 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.106302 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:57:26.107506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.107485 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 19:57:26.463257 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.462998 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:26.544589 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.544557 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v"] Oct 30 19:57:26.551390 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.551254 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" Oct 30 19:57:26.584125 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.584053 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-981ee5-dockercfg-sgjm6\"" Oct 30 19:57:26.599240 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.599212 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v"] Oct 30 19:57:26.617763 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.617735 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-981ee5\"" Oct 30 19:57:26.678838 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.678797 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ndb\" (UniqueName: \"kubernetes.io/projected/18693447-e4c1-4975-be81-368f7b3d54ed-kube-api-access-m7ndb\") pod \"payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v\" (UID: \"18693447-e4c1-4975-be81-368f7b3d54ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" Oct 30 19:57:26.679588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.679320 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18693447-e4c1-4975-be81-368f7b3d54ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v\" (UID: \"18693447-e4c1-4975-be81-368f7b3d54ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" Oct 30 19:57:26.695502 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.695471 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:26.781100 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.781059 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18693447-e4c1-4975-be81-368f7b3d54ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v\" (UID: \"18693447-e4c1-4975-be81-368f7b3d54ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" Oct 30 19:57:26.781295 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.781193 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ndb\" (UniqueName: \"kubernetes.io/projected/18693447-e4c1-4975-be81-368f7b3d54ed-kube-api-access-m7ndb\") pod \"payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v\" (UID: \"18693447-e4c1-4975-be81-368f7b3d54ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" Oct 30 19:57:26.781975 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.781948 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18693447-e4c1-4975-be81-368f7b3d54ed-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v\" (UID: \"18693447-e4c1-4975-be81-368f7b3d54ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" Oct 30 19:57:26.797587 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.797553 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ndb\" (UniqueName: \"kubernetes.io/projected/18693447-e4c1-4975-be81-368f7b3d54ed-kube-api-access-m7ndb\") pod \"payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v\" (UID: \"18693447-e4c1-4975-be81-368f7b3d54ed\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" Oct 30 19:57:26.899177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:26.898719 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" Oct 30 19:57:27.457409 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:27.441685 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:27.457409 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:27.451096 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v"] Oct 30 19:57:27.607065 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:27.606915 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:27.645741 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:27.645705 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" event={"ID":"18693447-e4c1-4975-be81-368f7b3d54ed","Type":"ContainerStarted","Data":"9db49a4c7a5c19f47fc89422c25e8defb60a39aa355f44caef1d5ccbc006b608"} Oct 30 19:57:27.645932 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:27.645770 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" event={"ID":"18693447-e4c1-4975-be81-368f7b3d54ed","Type":"ContainerStarted","Data":"96817b6efbf20be92fbf1af064c89d7c723d6518ffed68952d27634c4764eba4"} Oct 30 19:57:27.645932 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:27.645922 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" Oct 30 19:57:27.675990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:27.675939 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" podStartSLOduration=1.675926531 podStartE2EDuration="1.675926531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:27.670628036 +0000 UTC m=+1567.365836006" watchObservedRunningTime="2026-10-30 19:57:27.675926531 +0000 UTC m=+1567.371134493" Oct 30 19:57:27.940772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:27.940737 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz"] Oct 30 19:57:27.941062 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:27.941007 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" podUID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerName="istio-proxy" containerID="cri-o://aa6a60c3ff94cfa847a6442dbaadfaae911b4937378cfb0a6e490662bb81709e" gracePeriod=30 Oct 30 19:57:28.103214 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:28.092832 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:29.005287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:29.004928 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:29.960018 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:29.959965 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:30.467035 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:30.466990 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:30.866615 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:30.866522 5091 patch_prober.go:29] interesting pod/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 19:57:30.866615 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:30.866584 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" podUID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 19:57:31.479871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:31.479837 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:32.688168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:32.682047 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:33.640366 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.640331 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:57:33.758192 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.756103 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-token\" (UniqueName: \"kubernetes.io/projected/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-token\") pod \"7bcf3894-d14f-4ee3-a294-63daff87f18a\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " Oct 30 19:57:33.789212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.772439 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-credential-socket\") pod \"7bcf3894-d14f-4ee3-a294-63daff87f18a\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " Oct 30 19:57:33.789212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.772504 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-workload-certs\") pod \"7bcf3894-d14f-4ee3-a294-63daff87f18a\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " Oct 30 19:57:33.789212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.772631 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-podinfo\") pod \"7bcf3894-d14f-4ee3-a294-63daff87f18a\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " Oct 30 19:57:33.789212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.772662 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-envoy\") pod \"7bcf3894-d14f-4ee3-a294-63daff87f18a\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " Oct 30 19:57:33.789212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.772731 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bcf3894-d14f-4ee3-a294-63daff87f18a-kube-api-access-rw9k5\" (UniqueName: \"kubernetes.io/projected/7bcf3894-d14f-4ee3-a294-63daff87f18a-kube-api-access-rw9k5\") pod \"7bcf3894-d14f-4ee3-a294-63daff87f18a\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " Oct 30 19:57:33.789212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.772764 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-workload-socket\") pod \"7bcf3894-d14f-4ee3-a294-63daff87f18a\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " Oct 30 19:57:33.789212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.772824 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-data\") pod \"7bcf3894-d14f-4ee3-a294-63daff87f18a\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " Oct 30 19:57:33.789212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.772905 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bcf3894-d14f-4ee3-a294-63daff87f18a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bcf3894-d14f-4ee3-a294-63daff87f18a-istiod-ca-cert\") pod \"7bcf3894-d14f-4ee3-a294-63daff87f18a\" (UID: \"7bcf3894-d14f-4ee3-a294-63daff87f18a\") " Oct 30 19:57:33.862173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.795535 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-workload-certs" pod "7bcf3894-d14f-4ee3-a294-63daff87f18a" (UID: "7bcf3894-d14f-4ee3-a294-63daff87f18a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:33.897228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.863316 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-data" pod "7bcf3894-d14f-4ee3-a294-63daff87f18a" (UID: "7bcf3894-d14f-4ee3-a294-63daff87f18a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:33.897228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.864258 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-credential-socket" pod "7bcf3894-d14f-4ee3-a294-63daff87f18a" (UID: "7bcf3894-d14f-4ee3-a294-63daff87f18a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:33.897228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.864324 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-workload-socket" pod "7bcf3894-d14f-4ee3-a294-63daff87f18a" (UID: "7bcf3894-d14f-4ee3-a294-63daff87f18a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:33.916175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.911862 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:33.916175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.911902 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:33.916175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.911918 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:33.916175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:33.911933 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:34.163222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.156385 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bcf3894-d14f-4ee3-a294-63daff87f18a-istiod-ca-cert" pod "7bcf3894-d14f-4ee3-a294-63daff87f18a" (UID: "7bcf3894-d14f-4ee3-a294-63daff87f18a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:34.163222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.160739 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:34.169212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.163453 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7bcf3894-d14f-4ee3-a294-63daff87f18a-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:34.300990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.300951 5091 generic.go:356] "Generic (PLEG): container finished" podID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerID="aa6a60c3ff94cfa847a6442dbaadfaae911b4937378cfb0a6e490662bb81709e" exitCode=0 Oct 30 19:57:34.301420 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.301396 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" Oct 30 19:57:34.301712 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.301685 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" event={"ID":"7bcf3894-d14f-4ee3-a294-63daff87f18a","Type":"ContainerDied","Data":"aa6a60c3ff94cfa847a6442dbaadfaae911b4937378cfb0a6e490662bb81709e"} Oct 30 19:57:34.301885 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.301868 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz" event={"ID":"7bcf3894-d14f-4ee3-a294-63daff87f18a","Type":"ContainerDied","Data":"918527cfc3becb6038a87147aabaa704382ba93cd054a9b3ef2467189f81b345"} Oct 30 19:57:34.302017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.302004 5091 scope.go:122] "RemoveContainer" containerID="aa6a60c3ff94cfa847a6442dbaadfaae911b4937378cfb0a6e490662bb81709e" Oct 30 19:57:34.485173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.475447 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-token" pod "7bcf3894-d14f-4ee3-a294-63daff87f18a" (UID: "7bcf3894-d14f-4ee3-a294-63daff87f18a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:34.485173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.475537 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bcf3894-d14f-4ee3-a294-63daff87f18a-kube-api-access-rw9k5" pod "7bcf3894-d14f-4ee3-a294-63daff87f18a" (UID: "7bcf3894-d14f-4ee3-a294-63daff87f18a"). InnerVolumeSpecName "kube-api-access-rw9k5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:34.485173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.475591 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-podinfo" pod "7bcf3894-d14f-4ee3-a294-63daff87f18a" (UID: "7bcf3894-d14f-4ee3-a294-63daff87f18a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:57:34.485173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.482697 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-envoy" pod "7bcf3894-d14f-4ee3-a294-63daff87f18a" (UID: "7bcf3894-d14f-4ee3-a294-63daff87f18a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:57:34.572170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.545913 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:34.572170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.545953 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:34.572170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.545973 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw9k5\" (UniqueName: \"kubernetes.io/projected/7bcf3894-d14f-4ee3-a294-63daff87f18a-kube-api-access-rw9k5\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:34.572170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.545988 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7bcf3894-d14f-4ee3-a294-63daff87f18a-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:34.653439 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.653391 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:34.852169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:34.836332 5091 scope.go:122] "RemoveContainer" containerID="6b71d3f026897317d1c110c31a1297a30e18c867e421af0910c645e42208eab3" Oct 30 19:57:35.006285 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.006214 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz"] Oct 30 19:57:35.018229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.018196 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-c1768b-openshift-default-567bf88dc7-cqntz"] Oct 30 19:57:35.089210 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.070096 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4"] Oct 30 19:57:35.111207 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.094136 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerName="istio-proxy" Oct 30 19:57:35.111412 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.111396 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerName="istio-proxy" Oct 30 19:57:35.111825 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.111808 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerName="istio-proxy" Oct 30 19:57:35.122622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.122597 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" Oct 30 19:57:35.129586 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.129473 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:57:35.131929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.131907 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:57:35.139310 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.139288 5091 scope.go:122] "RemoveContainer" containerID="aa6a60c3ff94cfa847a6442dbaadfaae911b4937378cfb0a6e490662bb81709e" Oct 30 19:57:35.146197 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:57:35.139892 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa6a60c3ff94cfa847a6442dbaadfaae911b4937378cfb0a6e490662bb81709e\": container with ID starting with aa6a60c3ff94cfa847a6442dbaadfaae911b4937378cfb0a6e490662bb81709e not found: ID does not exist" containerID="aa6a60c3ff94cfa847a6442dbaadfaae911b4937378cfb0a6e490662bb81709e" Oct 30 19:57:35.146389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.146356 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa6a60c3ff94cfa847a6442dbaadfaae911b4937378cfb0a6e490662bb81709e"} err="failed to get container status \"aa6a60c3ff94cfa847a6442dbaadfaae911b4937378cfb0a6e490662bb81709e\": rpc error: code = NotFound desc = could not find container \"aa6a60c3ff94cfa847a6442dbaadfaae911b4937378cfb0a6e490662bb81709e\": container with ID starting with aa6a60c3ff94cfa847a6442dbaadfaae911b4937378cfb0a6e490662bb81709e not found: ID does not exist" Oct 30 19:57:35.146505 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.146493 5091 scope.go:122] "RemoveContainer" containerID="6b71d3f026897317d1c110c31a1297a30e18c867e421af0910c645e42208eab3" Oct 30 19:57:35.202726 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:57:35.178277 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b71d3f026897317d1c110c31a1297a30e18c867e421af0910c645e42208eab3\": container with ID starting with 6b71d3f026897317d1c110c31a1297a30e18c867e421af0910c645e42208eab3 not found: ID does not exist" containerID="6b71d3f026897317d1c110c31a1297a30e18c867e421af0910c645e42208eab3" Oct 30 19:57:35.202726 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.178322 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b71d3f026897317d1c110c31a1297a30e18c867e421af0910c645e42208eab3"} err="failed to get container status \"6b71d3f026897317d1c110c31a1297a30e18c867e421af0910c645e42208eab3\": rpc error: code = NotFound desc = could not find container \"6b71d3f026897317d1c110c31a1297a30e18c867e421af0910c645e42208eab3\": container with ID starting with 6b71d3f026897317d1c110c31a1297a30e18c867e421af0910c645e42208eab3 not found: ID does not exist" Oct 30 19:57:35.204005 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.203974 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4"] Oct 30 19:57:35.216170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.208738 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45dg\" (UniqueName: \"kubernetes.io/projected/28374e9f-8887-431c-a94b-8ced74f433c6-kube-api-access-g45dg\") pod \"maas-api-revoke-sub-6a905328-zmjp4\" (UID: \"28374e9f-8887-431c-a94b-8ced74f433c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" Oct 30 19:57:35.241504 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.227269 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28374e9f-8887-431c-a94b-8ced74f433c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a905328-zmjp4\" (UID: \"28374e9f-8887-431c-a94b-8ced74f433c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" Oct 30 19:57:35.241504 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.227328 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28374e9f-8887-431c-a94b-8ced74f433c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a905328-zmjp4\" (UID: \"28374e9f-8887-431c-a94b-8ced74f433c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" Oct 30 19:57:35.343353 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.343251 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45dg\" (UniqueName: \"kubernetes.io/projected/28374e9f-8887-431c-a94b-8ced74f433c6-kube-api-access-g45dg\") pod \"maas-api-revoke-sub-6a905328-zmjp4\" (UID: \"28374e9f-8887-431c-a94b-8ced74f433c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" Oct 30 19:57:35.394350 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.394316 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28374e9f-8887-431c-a94b-8ced74f433c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a905328-zmjp4\" (UID: \"28374e9f-8887-431c-a94b-8ced74f433c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" Oct 30 19:57:35.394619 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.394601 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28374e9f-8887-431c-a94b-8ced74f433c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a905328-zmjp4\" (UID: \"28374e9f-8887-431c-a94b-8ced74f433c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" Oct 30 19:57:35.411170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.395269 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28374e9f-8887-431c-a94b-8ced74f433c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a905328-zmjp4\" (UID: \"28374e9f-8887-431c-a94b-8ced74f433c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" Oct 30 19:57:35.511179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.510372 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45dg\" (UniqueName: \"kubernetes.io/projected/28374e9f-8887-431c-a94b-8ced74f433c6-kube-api-access-g45dg\") pod \"maas-api-revoke-sub-6a905328-zmjp4\" (UID: \"28374e9f-8887-431c-a94b-8ced74f433c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" Oct 30 19:57:35.511179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.510905 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28374e9f-8887-431c-a94b-8ced74f433c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a905328-zmjp4\" (UID: \"28374e9f-8887-431c-a94b-8ced74f433c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" Oct 30 19:57:35.596869 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.594773 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:35.603644 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:35.601958 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" Oct 30 19:57:36.484373 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:36.483997 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:36.763953 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:36.763914 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bcf3894-d14f-4ee3-a294-63daff87f18a" path="/var/lib/kubelet/pods/7bcf3894-d14f-4ee3-a294-63daff87f18a/volumes" Oct 30 19:57:37.056247 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:57:37.052317 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28374e9f_8887_431c_a94b_8ced74f433c6.slice/crio-67c72f7a4b606155af2bf7269561cf5bd04c750a25ddd75bdaacf624a7b76107 WatchSource:0}: Error finding container 67c72f7a4b606155af2bf7269561cf5bd04c750a25ddd75bdaacf624a7b76107: Status 404 returned error can't find the container with id 67c72f7a4b606155af2bf7269561cf5bd04c750a25ddd75bdaacf624a7b76107 Oct 30 19:57:37.231263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:37.225278 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:57:37.257194 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:37.245273 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:57:37.364232 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:37.342819 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 30 19:57:37.380967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:37.380609 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4"] Oct 30 19:57:37.397663 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:37.397626 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:37.913026 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:37.912985 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" event={"ID":"28374e9f-8887-431c-a94b-8ced74f433c6","Type":"ContainerStarted","Data":"67c72f7a4b606155af2bf7269561cf5bd04c750a25ddd75bdaacf624a7b76107"} Oct 30 19:57:38.305172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:38.301804 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:38.710892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:38.692180 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" Oct 30 19:57:39.039622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:39.039560 5091 generic.go:356] "Generic (PLEG): container finished" podID="28374e9f-8887-431c-a94b-8ced74f433c6" containerID="1c9cf0c7574d4ff724fc6ad7483cd81c08c3c2f7c0e0d43ed1d14d5561c6b593" exitCode=0 Oct 30 19:57:39.040075 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:39.039846 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" event={"ID":"28374e9f-8887-431c-a94b-8ced74f433c6","Type":"ContainerDied","Data":"1c9cf0c7574d4ff724fc6ad7483cd81c08c3c2f7c0e0d43ed1d14d5561c6b593"} Oct 30 19:57:39.365534 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:39.365436 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:39.532563 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:39.532525 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:39.730669 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:39.730578 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:40.270903 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:40.270877 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" Oct 30 19:57:40.359218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:40.349380 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28374e9f-8887-431c-a94b-8ced74f433c6-kube-api-access-g45dg\" (UniqueName: \"kubernetes.io/projected/28374e9f-8887-431c-a94b-8ced74f433c6-kube-api-access-g45dg\") pod \"28374e9f-8887-431c-a94b-8ced74f433c6\" (UID: \"28374e9f-8887-431c-a94b-8ced74f433c6\") " Oct 30 19:57:40.359218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:40.349673 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28374e9f-8887-431c-a94b-8ced74f433c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28374e9f-8887-431c-a94b-8ced74f433c6-maas-api-xks-ca\") pod \"28374e9f-8887-431c-a94b-8ced74f433c6\" (UID: \"28374e9f-8887-431c-a94b-8ced74f433c6\") " Oct 30 19:57:40.359218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:40.349755 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28374e9f-8887-431c-a94b-8ced74f433c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28374e9f-8887-431c-a94b-8ced74f433c6-maas-api-openshift-ca\") pod \"28374e9f-8887-431c-a94b-8ced74f433c6\" (UID: \"28374e9f-8887-431c-a94b-8ced74f433c6\") " Oct 30 19:57:40.359218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:40.355174 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28374e9f-8887-431c-a94b-8ced74f433c6-maas-api-openshift-ca" pod "28374e9f-8887-431c-a94b-8ced74f433c6" (UID: "28374e9f-8887-431c-a94b-8ced74f433c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:40.368226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:40.364305 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:40.390244 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:40.390107 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28374e9f-8887-431c-a94b-8ced74f433c6-kube-api-access-g45dg" pod "28374e9f-8887-431c-a94b-8ced74f433c6" (UID: "28374e9f-8887-431c-a94b-8ced74f433c6"). InnerVolumeSpecName "kube-api-access-g45dg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:40.390920 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:40.390883 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28374e9f-8887-431c-a94b-8ced74f433c6-maas-api-xks-ca" pod "28374e9f-8887-431c-a94b-8ced74f433c6" (UID: "28374e9f-8887-431c-a94b-8ced74f433c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:40.455667 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:40.455594 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28374e9f-8887-431c-a94b-8ced74f433c6-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:40.455667 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:40.455628 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28374e9f-8887-431c-a94b-8ced74f433c6-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:40.455667 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:40.455643 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g45dg\" (UniqueName: \"kubernetes.io/projected/28374e9f-8887-431c-a94b-8ced74f433c6-kube-api-access-g45dg\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:41.085815 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:41.085777 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" event={"ID":"28374e9f-8887-431c-a94b-8ced74f433c6","Type":"ContainerDied","Data":"67c72f7a4b606155af2bf7269561cf5bd04c750a25ddd75bdaacf624a7b76107"} Oct 30 19:57:41.086041 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:41.085840 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67c72f7a4b606155af2bf7269561cf5bd04c750a25ddd75bdaacf624a7b76107" Oct 30 19:57:41.086041 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:41.085910 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4" Oct 30 19:57:41.385848 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:41.385701 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:42.623203 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:42.619310 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:43.939270 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:43.939100 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:45.035192 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.034816 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.362844 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx"] Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.363588 5091 cpu_manager.go:414] "containerMap: removing container" podUID="28374e9f-8887-431c-a94b-8ced74f433c6" containerName="revoke-keys" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.363604 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28374e9f-8887-431c-a94b-8ced74f433c6" containerName="revoke-keys" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.363615 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerName="istio-proxy" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.363620 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerName="istio-proxy" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.363724 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="28374e9f-8887-431c-a94b-8ced74f433c6" containerName="revoke-keys" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.363732 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bcf3894-d14f-4ee3-a294-63daff87f18a" containerName="istio-proxy" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.376459 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.378090 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx"] Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.411091 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.411202 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.496290 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nffmp\" (UniqueName: \"kubernetes.io/projected/4014c09e-3d4f-4a8d-afde-97c482579676-kube-api-access-nffmp\") pod \"maas-api-revoke-sub-cddc9f7b-qd4zx\" (UID: \"4014c09e-3d4f-4a8d-afde-97c482579676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.496348 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4014c09e-3d4f-4a8d-afde-97c482579676-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cddc9f7b-qd4zx\" (UID: \"4014c09e-3d4f-4a8d-afde-97c482579676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.496577 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4014c09e-3d4f-4a8d-afde-97c482579676-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cddc9f7b-qd4zx\" (UID: \"4014c09e-3d4f-4a8d-afde-97c482579676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.597795 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4014c09e-3d4f-4a8d-afde-97c482579676-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cddc9f7b-qd4zx\" (UID: \"4014c09e-3d4f-4a8d-afde-97c482579676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.597883 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nffmp\" (UniqueName: \"kubernetes.io/projected/4014c09e-3d4f-4a8d-afde-97c482579676-kube-api-access-nffmp\") pod \"maas-api-revoke-sub-cddc9f7b-qd4zx\" (UID: \"4014c09e-3d4f-4a8d-afde-97c482579676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.597924 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4014c09e-3d4f-4a8d-afde-97c482579676-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cddc9f7b-qd4zx\" (UID: \"4014c09e-3d4f-4a8d-afde-97c482579676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.598746 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4014c09e-3d4f-4a8d-afde-97c482579676-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cddc9f7b-qd4zx\" (UID: \"4014c09e-3d4f-4a8d-afde-97c482579676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.600047 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4014c09e-3d4f-4a8d-afde-97c482579676-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cddc9f7b-qd4zx\" (UID: \"4014c09e-3d4f-4a8d-afde-97c482579676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.613167 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nffmp\" (UniqueName: \"kubernetes.io/projected/4014c09e-3d4f-4a8d-afde-97c482579676-kube-api-access-nffmp\") pod \"maas-api-revoke-sub-cddc9f7b-qd4zx\" (UID: \"4014c09e-3d4f-4a8d-afde-97c482579676\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:45.707895 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:57:46.033654 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:46.045377 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:57:46.053054 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4014c09e_3d4f_4a8d_afde_97c482579676.slice/crio-7e9ef33b4ee81725c0654216555ef07f9f7a1056ffa8e291d1b8f8643b46776a WatchSource:0}: Error finding container 7e9ef33b4ee81725c0654216555ef07f9f7a1056ffa8e291d1b8f8643b46776a: Status 404 returned error can't find the container with id 7e9ef33b4ee81725c0654216555ef07f9f7a1056ffa8e291d1b8f8643b46776a Oct 30 19:57:46.112786 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:46.054089 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx"] Oct 30 19:57:46.157882 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:46.157856 5091 generic.go:356] "Generic (PLEG): container finished" podID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" containerID="ccbd6495cf6cda2472174039f2a1ced45112c359f21843030002c9dec0ef3fe3" exitCode=0 Oct 30 19:57:46.158055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:46.157910 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" event={"ID":"0153a5c2-c41c-4dd5-9bc2-035455025bd1","Type":"ContainerDied","Data":"ccbd6495cf6cda2472174039f2a1ced45112c359f21843030002c9dec0ef3fe3"} Oct 30 19:57:46.158476 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:46.158457 5091 scope.go:122] "RemoveContainer" containerID="ccbd6495cf6cda2472174039f2a1ced45112c359f21843030002c9dec0ef3fe3" Oct 30 19:57:46.159527 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:46.159503 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" event={"ID":"4014c09e-3d4f-4a8d-afde-97c482579676","Type":"ContainerStarted","Data":"7e9ef33b4ee81725c0654216555ef07f9f7a1056ffa8e291d1b8f8643b46776a"} Oct 30 19:57:46.258621 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:46.257321 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:46.258621 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:46.257417 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:46.258621 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:46.257467 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:47.202136 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.202018 5091 generic.go:356] "Generic (PLEG): container finished" podID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerID="7f5d4e0b0ede7cef957dc6d33cfe4195cd797972df41d573d4b6d2102d1f7dcc" exitCode=0 Oct 30 19:57:47.202136 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.202053 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" event={"ID":"c222e9d2-821b-4c1c-92d2-88297be8f52d","Type":"ContainerDied","Data":"7f5d4e0b0ede7cef957dc6d33cfe4195cd797972df41d573d4b6d2102d1f7dcc"} Oct 30 19:57:47.202990 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.202636 5091 scope.go:122] "RemoveContainer" containerID="7f5d4e0b0ede7cef957dc6d33cfe4195cd797972df41d573d4b6d2102d1f7dcc" Oct 30 19:57:47.205212 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.205179 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:47.205389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.205266 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:47.205389 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.205312 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:47.208162 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.206287 5091 generic.go:356] "Generic (PLEG): container finished" podID="872504eb-20ef-411c-b526-61e05a15a9c4" containerID="8fdff38d73f5853ed799dedad4be18e5e92ca0a473371521325500a93cc767a7" exitCode=0 Oct 30 19:57:47.208162 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.206470 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" event={"ID":"872504eb-20ef-411c-b526-61e05a15a9c4","Type":"ContainerDied","Data":"8fdff38d73f5853ed799dedad4be18e5e92ca0a473371521325500a93cc767a7"} Oct 30 19:57:47.208162 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.206875 5091 scope.go:122] "RemoveContainer" containerID="8fdff38d73f5853ed799dedad4be18e5e92ca0a473371521325500a93cc767a7" Oct 30 19:57:47.212369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.209274 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:47.212369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.209345 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:47.212369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.209393 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:47.212369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.210904 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" event={"ID":"0153a5c2-c41c-4dd5-9bc2-035455025bd1","Type":"ContainerStarted","Data":"6b06c26c1c600f57eb3fc7b0360f3b7795a1c2fd23f0a99dba456635cbe0bfb2"} Oct 30 19:57:47.250400 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.218192 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" event={"ID":"4014c09e-3d4f-4a8d-afde-97c482579676","Type":"ContainerStarted","Data":"e39752ca634098647dd8be4bca1c86a9ab6e4f99d6b7e40ce96e05d3c6cd89d1"} Oct 30 19:57:47.305503 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.301732 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt"] Oct 30 19:57:47.319501 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.319473 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.336722 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.336701 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-779fa3\"/\"default-dockercfg-rgzpl\"" Oct 30 19:57:47.341265 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.341239 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt"] Oct 30 19:57:47.342291 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.342269 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-779fa3\"/\"auth-test-model-779fa3-kserve-self-signed-certs\"" Oct 30 19:57:47.342791 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.342773 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-779fa3\"/\"kube-root-ca.crt\"" Oct 30 19:57:47.401238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.400399 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-model-cache\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.401238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.400473 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6794b918-54df-4217-b67a-26fec2eb5646-tls-certs\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.401238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.400541 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-tmp-dir\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.401238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.400635 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6ds7\" (UniqueName: \"kubernetes.io/projected/6794b918-54df-4217-b67a-26fec2eb5646-kube-api-access-f6ds7\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.401238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.400731 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-dshm\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.401238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.400867 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-home\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.425804 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.425777 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:57:47.425947 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.425922 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 19:57:47.426037 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.426002 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" podUID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 19:57:47.445172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.441262 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" podStartSLOduration=2.44124733 podStartE2EDuration="2.44124733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:47.440578626 +0000 UTC m=+1587.135786584" watchObservedRunningTime="2026-10-30 19:57:47.44124733 +0000 UTC m=+1587.136455287" Oct 30 19:57:47.445172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.443930 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-779fa3\"/\"openshift-service-ca.crt\"" Oct 30 19:57:47.508304 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.508254 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-home\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.508946 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.508908 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-model-cache\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.509128 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.509109 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6794b918-54df-4217-b67a-26fec2eb5646-tls-certs\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.509305 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.509286 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-tmp-dir\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.509474 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.509457 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6ds7\" (UniqueName: \"kubernetes.io/projected/6794b918-54df-4217-b67a-26fec2eb5646-kube-api-access-f6ds7\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.509606 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.509594 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-dshm\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.510518 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.510495 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-model-cache\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.511266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.510876 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-tmp-dir\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.511771 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.511743 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-home\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.517612 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.516777 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-dshm\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.520669 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.519210 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6794b918-54df-4217-b67a-26fec2eb5646-tls-certs\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.539512 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.539467 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6ds7\" (UniqueName: \"kubernetes.io/projected/6794b918-54df-4217-b67a-26fec2eb5646-kube-api-access-f6ds7\") pod \"auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.568081 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.568028 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:47.752186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.742780 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:47.937741 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:47.936616 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:48.166450 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:48.161406 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:57:48.220680 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:48.199784 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:48.265224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:48.263968 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" event={"ID":"c222e9d2-821b-4c1c-92d2-88297be8f52d","Type":"ContainerStarted","Data":"8a34b82f37a51d39c907a01d4354f8fe58d3a160a7cf56f98e696787ace7a106"} Oct 30 19:57:48.310120 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:48.310080 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" event={"ID":"872504eb-20ef-411c-b526-61e05a15a9c4","Type":"ContainerStarted","Data":"b80337cf3ed50a192c528978b244cd2870ec86fa9a3c2958258414a5ad2cc0ef"} Oct 30 19:57:48.314118 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:48.312908 5091 generic.go:356] "Generic (PLEG): container finished" podID="4014c09e-3d4f-4a8d-afde-97c482579676" containerID="e39752ca634098647dd8be4bca1c86a9ab6e4f99d6b7e40ce96e05d3c6cd89d1" exitCode=0 Oct 30 19:57:48.314118 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:48.313880 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" event={"ID":"4014c09e-3d4f-4a8d-afde-97c482579676","Type":"ContainerDied","Data":"e39752ca634098647dd8be4bca1c86a9ab6e4f99d6b7e40ce96e05d3c6cd89d1"} Oct 30 19:57:48.419995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:48.419910 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 19:57:48.419995 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:48.419954 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" podUID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 19:57:48.427072 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:48.426965 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt"] Oct 30 19:57:48.789013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:48.788979 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:49.056381 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:49.056056 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:57:49.056381 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:49.056221 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 19:57:49.056381 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:49.056270 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" podUID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 19:57:49.143570 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:49.143530 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 19:57:49.143832 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:49.143802 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" podUID="872504eb-20ef-411c-b526-61e05a15a9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 19:57:49.146196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:49.144197 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:57:49.343989 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:49.343893 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" event={"ID":"6794b918-54df-4217-b67a-26fec2eb5646","Type":"ContainerStarted","Data":"9be74be70a9602855343bbfaf9dc95b9e04f7f3afb85e043c1c339048abbc4b0"} Oct 30 19:57:49.343989 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:49.343938 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" event={"ID":"6794b918-54df-4217-b67a-26fec2eb5646","Type":"ContainerStarted","Data":"390b2a773a489283705a0cf36a15dee406cb86d5773591afe820f115e10593e4"} Oct 30 19:57:49.431772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:49.431703 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 19:57:49.431772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:49.431755 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" podUID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 19:57:49.744953 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:49.744844 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:50.164225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:50.133460 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 19:57:50.197199 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:50.195330 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" podUID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 19:57:50.197199 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:50.146009 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:50.327185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:50.247186 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 19:57:50.327185 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:50.247277 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" podUID="872504eb-20ef-411c-b526-61e05a15a9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 19:57:50.424807 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:50.424725 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 19:57:50.475202 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:50.435270 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" podUID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 19:57:50.720523 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:50.720417 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:57:51.014808 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.014777 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" Oct 30 19:57:51.068166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.068110 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 19:57:51.068761 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.068536 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" podUID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 19:57:51.104963 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.104922 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" event={"ID":"4014c09e-3d4f-4a8d-afde-97c482579676","Type":"ContainerDied","Data":"7e9ef33b4ee81725c0654216555ef07f9f7a1056ffa8e291d1b8f8643b46776a"} Oct 30 19:57:51.109198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.105223 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e9ef33b4ee81725c0654216555ef07f9f7a1056ffa8e291d1b8f8643b46776a" Oct 30 19:57:51.140237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.124671 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" podStartSLOduration=4.124650153 podStartE2EDuration="4.124650153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:49.427458564 +0000 UTC m=+1589.122666524" watchObservedRunningTime="2026-10-30 19:57:51.124650153 +0000 UTC m=+1590.819858113" Oct 30 19:57:51.163176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.158954 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 19:57:51.163176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.159022 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" podUID="872504eb-20ef-411c-b526-61e05a15a9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 19:57:51.194170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.190349 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:57:51.194170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.192538 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4014c09e-3d4f-4a8d-afde-97c482579676-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4014c09e-3d4f-4a8d-afde-97c482579676-maas-api-openshift-ca\") pod \"4014c09e-3d4f-4a8d-afde-97c482579676\" (UID: \"4014c09e-3d4f-4a8d-afde-97c482579676\") " Oct 30 19:57:51.281935 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.281851 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4014c09e-3d4f-4a8d-afde-97c482579676-maas-api-openshift-ca" pod "4014c09e-3d4f-4a8d-afde-97c482579676" (UID: "4014c09e-3d4f-4a8d-afde-97c482579676"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:57:51.314171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.282754 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:51.348167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.323369 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4014c09e-3d4f-4a8d-afde-97c482579676-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4014c09e-3d4f-4a8d-afde-97c482579676-maas-api-xks-ca\") pod \"4014c09e-3d4f-4a8d-afde-97c482579676\" (UID: \"4014c09e-3d4f-4a8d-afde-97c482579676\") " Oct 30 19:57:51.356167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.350888 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4014c09e-3d4f-4a8d-afde-97c482579676-kube-api-access-nffmp\" (UniqueName: \"kubernetes.io/projected/4014c09e-3d4f-4a8d-afde-97c482579676-kube-api-access-nffmp\") pod \"4014c09e-3d4f-4a8d-afde-97c482579676\" (UID: \"4014c09e-3d4f-4a8d-afde-97c482579676\") " Oct 30 19:57:51.366685 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.366656 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4014c09e-3d4f-4a8d-afde-97c482579676-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:51.419048 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.411325 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4014c09e-3d4f-4a8d-afde-97c482579676-kube-api-access-nffmp" pod "4014c09e-3d4f-4a8d-afde-97c482579676" (UID: "4014c09e-3d4f-4a8d-afde-97c482579676"). InnerVolumeSpecName "kube-api-access-nffmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:57:51.421173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.420691 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 19:57:51.421173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.420748 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" podUID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 19:57:51.502928 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.502732 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nffmp\" (UniqueName: \"kubernetes.io/projected/4014c09e-3d4f-4a8d-afde-97c482579676-kube-api-access-nffmp\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:51.502928 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.502882 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4014c09e-3d4f-4a8d-afde-97c482579676-maas-api-xks-ca" pod "4014c09e-3d4f-4a8d-afde-97c482579676" (UID: "4014c09e-3d4f-4a8d-afde-97c482579676"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:57:51.615228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:51.607748 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4014c09e-3d4f-4a8d-afde-97c482579676-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:57:52.036509 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:52.036466 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx" Oct 30 19:57:52.050809 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:52.050778 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 19:57:52.051018 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:52.050991 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" podUID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 19:57:52.123636 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:52.123542 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 19:57:52.123890 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:52.123863 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" podUID="872504eb-20ef-411c-b526-61e05a15a9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 19:57:52.226167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:52.223168 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:52.329396 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:52.329360 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:57:52.330237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:52.330210 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:57:52.424226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:52.419568 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" start-of-body= Oct 30 19:57:52.424226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:52.419643 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" podUID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.216:15021/healthz/ready\": dial tcp 10.132.0.216:15021: connect: connection refused" Oct 30 19:57:53.077171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:53.075949 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 19:57:53.077171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:53.076014 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" podUID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 19:57:53.133192 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:53.130297 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 19:57:53.133192 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:53.130361 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" podUID="872504eb-20ef-411c-b526-61e05a15a9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 19:57:53.223230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:53.219778 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:53.431440 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:53.431319 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:57:53.434334 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:53.434311 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 19:57:54.069330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:54.068591 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" start-of-body= Oct 30 19:57:54.069330 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:54.068653 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" podUID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.217:15021/healthz/ready\": dial tcp 10.132.0.217:15021: connect: connection refused" Oct 30 19:57:54.122753 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:54.122724 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" start-of-body= Oct 30 19:57:54.123176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:54.122772 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" podUID="872504eb-20ef-411c-b526-61e05a15a9c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.218:15021/healthz/ready\": dial tcp 10.132.0.218:15021: connect: connection refused" Oct 30 19:57:54.290016 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:54.289976 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:55.056542 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:55.056504 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:57:55.057724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:55.057698 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 19:57:55.126750 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:55.126721 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:57:55.127781 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:55.127765 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 19:57:55.363057 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:55.359792 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:56.508174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:56.508127 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:56.773734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:56.773651 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:56.925936 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:56.925900 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:57.561846 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:57.561770 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:57.745785 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:57.745750 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:57.748626 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:57.748596 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:57.764666 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:57.764639 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:58.087575 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.087540 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn"] Oct 30 19:57:58.088536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.088512 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4014c09e-3d4f-4a8d-afde-97c482579676" containerName="revoke-keys" Oct 30 19:57:58.088536 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.088535 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4014c09e-3d4f-4a8d-afde-97c482579676" containerName="revoke-keys" Oct 30 19:57:58.088784 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.088768 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4014c09e-3d4f-4a8d-afde-97c482579676" containerName="revoke-keys" Oct 30 19:57:58.093867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.093847 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.101614 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.100703 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn"] Oct 30 19:57:58.113401 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.113376 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-f0e884-openshift-default-dockercfg-z5dhl\"" Oct 30 19:57:58.169839 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.169805 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:58.206065 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.206004 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-credential-socket\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.206237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.206079 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a8e988-93ce-494f-8555-b36d6597e4db-istio-token\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.206237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.206198 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a8e988-93ce-494f-8555-b36d6597e4db-istio-podinfo\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.206381 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.206294 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-istio-envoy\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.206381 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.206342 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-workload-certs\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.206487 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.206393 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-workload-socket\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.206487 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.206454 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djr4g\" (UniqueName: \"kubernetes.io/projected/66a8e988-93ce-494f-8555-b36d6597e4db-kube-api-access-djr4g\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.206487 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.206481 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a8e988-93ce-494f-8555-b36d6597e4db-istiod-ca-cert\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.206648 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.206510 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-istio-data\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.209814 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.209783 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:57:58.310073 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.310030 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a8e988-93ce-494f-8555-b36d6597e4db-istio-podinfo\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.322375 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.322334 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-istio-envoy\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.322659 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.322635 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-workload-certs\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.345830 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.345753 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-workload-socket\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.356369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.356338 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djr4g\" (UniqueName: \"kubernetes.io/projected/66a8e988-93ce-494f-8555-b36d6597e4db-kube-api-access-djr4g\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.357298 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.357273 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a8e988-93ce-494f-8555-b36d6597e4db-istiod-ca-cert\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.364277 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.364252 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-istio-data\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.364888 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.364869 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-credential-socket\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.365383 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.323698 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-workload-certs\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.371544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.358126 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a8e988-93ce-494f-8555-b36d6597e4db-istiod-ca-cert\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.371704 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.364707 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-istio-data\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.371801 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.346510 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-workload-socket\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.371879 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.345492 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:58.372006 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.365281 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-credential-socket\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.377833 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.377377 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a8e988-93ce-494f-8555-b36d6597e4db-istio-token\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.520205 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.520170 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a8e988-93ce-494f-8555-b36d6597e4db-istio-podinfo\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.521369 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.521319 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djr4g\" (UniqueName: \"kubernetes.io/projected/66a8e988-93ce-494f-8555-b36d6597e4db-kube-api-access-djr4g\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.523588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.523444 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a8e988-93ce-494f-8555-b36d6597e4db-istio-token\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.529506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.529460 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-istio-envoy\") pod \"e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.714736 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.714631 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:57:58.720247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:58.718117 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:59.049028 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:57:59.048992 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a8e988_93ce_494f_8555_b36d6597e4db.slice/crio-89397d714afe1ec1f9d96e60a79fa79d94e9f2c699fe8ba798cb6580f9848678 WatchSource:0}: Error finding container 89397d714afe1ec1f9d96e60a79fa79d94e9f2c699fe8ba798cb6580f9848678: Status 404 returned error can't find the container with id 89397d714afe1ec1f9d96e60a79fa79d94e9f2c699fe8ba798cb6580f9848678 Oct 30 19:57:59.050944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:59.050923 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn"] Oct 30 19:57:59.051929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:59.051894 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:59.052039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:59.051958 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:59.052039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:59.051987 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:57:59.205816 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:59.205775 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" event={"ID":"66a8e988-93ce-494f-8555-b36d6597e4db","Type":"ContainerStarted","Data":"aa5ccd61e0e19db09008e482125ba4530fc7ab0c1596e82144534b5c3fa5885c"} Oct 30 19:57:59.206025 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:59.205851 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" event={"ID":"66a8e988-93ce-494f-8555-b36d6597e4db","Type":"ContainerStarted","Data":"89397d714afe1ec1f9d96e60a79fa79d94e9f2c699fe8ba798cb6580f9848678"} Oct 30 19:57:59.243167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:59.243089 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" podStartSLOduration=1.243067271 podStartE2EDuration="1.243067271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:57:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:57:59.241468033 +0000 UTC m=+1598.936675993" watchObservedRunningTime="2026-10-30 19:57:59.243067271 +0000 UTC m=+1598.938275230" Oct 30 19:57:59.710669 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:59.710619 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:57:59.716812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:57:59.716766 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:58:00.636686 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:00.636644 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:00.779511 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:00.779479 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:58:00.784080 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:00.784046 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:01.071589 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.071553 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg"] Oct 30 19:58:01.077630 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.077083 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.096109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.095470 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-54d4c4\"/\"kube-root-ca.crt\"" Oct 30 19:58:01.097421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.097393 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-54d4c4\"/\"auth-test-model-54d4c4-kserve-self-signed-certs\"" Oct 30 19:58:01.097735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.097412 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-54d4c4\"/\"openshift-service-ca.crt\"" Oct 30 19:58:01.097735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.097484 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-54d4c4\"/\"default-dockercfg-rvhnx\"" Oct 30 19:58:01.101456 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.101436 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg"] Oct 30 19:58:01.226544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.226517 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:58:01.227787 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.227764 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:58:01.240463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.240432 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-dshm\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.240609 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.240540 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-home\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.240699 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.240632 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-tmp-dir\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.240769 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.240729 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmd8m\" (UniqueName: \"kubernetes.io/projected/51c139a8-fc30-4991-baa6-75295965d8de-kube-api-access-zmd8m\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.240769 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.240749 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-model-cache\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.240892 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.240859 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c139a8-fc30-4991-baa6-75295965d8de-tls-certs\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.265767 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.265726 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:01.342335 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.342262 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-model-cache\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.342534 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.342396 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c139a8-fc30-4991-baa6-75295965d8de-tls-certs\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.342534 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.342480 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-dshm\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.342656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.342563 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-home\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.342948 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.342714 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-tmp-dir\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.342948 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.342724 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-model-cache\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.342948 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.342815 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmd8m\" (UniqueName: \"kubernetes.io/projected/51c139a8-fc30-4991-baa6-75295965d8de-kube-api-access-zmd8m\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.342948 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.342913 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-home\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.343221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.342967 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-tmp-dir\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.345320 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.345205 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-dshm\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.345502 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.345459 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c139a8-fc30-4991-baa6-75295965d8de-tls-certs\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.357779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.357749 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmd8m\" (UniqueName: \"kubernetes.io/projected/51c139a8-fc30-4991-baa6-75295965d8de-kube-api-access-zmd8m\") pod \"auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.406430 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.406391 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:01.720679 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.720586 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:01.742057 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.742015 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff"] Oct 30 19:58:01.743225 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.742263 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" podUID="582eb068-27ce-4484-8aa0-02d5a409e103" containerName="istio-proxy" containerID="cri-o://e43fc0b8a7641c479ae06ac9de0200f914c1110e8e2edacfe21feae9918d0671" gracePeriod=30 Oct 30 19:58:01.986087 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:01.986000 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:02.007546 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:58:02.007516 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod51c139a8_fc30_4991_baa6_75295965d8de.slice/crio-58d0893ad004fff98d85436a474865e152d66ca484f0b5810ac7a65ec040ef9d WatchSource:0}: Error finding container 58d0893ad004fff98d85436a474865e152d66ca484f0b5810ac7a65ec040ef9d: Status 404 returned error can't find the container with id 58d0893ad004fff98d85436a474865e152d66ca484f0b5810ac7a65ec040ef9d Oct 30 19:58:02.008076 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:02.008052 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg"] Oct 30 19:58:02.248221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:02.246259 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" event={"ID":"51c139a8-fc30-4991-baa6-75295965d8de","Type":"ContainerStarted","Data":"6373822c312f3ea8baf0f6c24888b0d52cd85b288fdfedf84d448b392c744d66"} Oct 30 19:58:02.248221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:02.246309 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" event={"ID":"51c139a8-fc30-4991-baa6-75295965d8de","Type":"ContainerStarted","Data":"58d0893ad004fff98d85436a474865e152d66ca484f0b5810ac7a65ec040ef9d"} Oct 30 19:58:02.298930 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:02.298864 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" podStartSLOduration=1.2988437990000001 podStartE2EDuration="1.298843799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:02.290431199 +0000 UTC m=+1601.985639156" watchObservedRunningTime="2026-10-30 19:58:02.298843799 +0000 UTC m=+1601.994051758" Oct 30 19:58:02.308848 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:02.308790 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:02.407276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:02.407246 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:02.584818 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:02.584706 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:03.630048 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.629918 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:03.690023 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.685347 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw"] Oct 30 19:58:03.718320 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.691614 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" Oct 30 19:58:03.740309 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.703399 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x"] Oct 30 19:58:03.756614 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.754386 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw"] Oct 30 19:58:03.756614 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.754440 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x"] Oct 30 19:58:03.756614 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.754533 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" Oct 30 19:58:03.756614 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.755900 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-f0e884\"" Oct 30 19:58:03.762215 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.762141 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-f0e884-dockercfg-4kjdg\"" Oct 30 19:58:03.778761 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.778725 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw2ln\" (UniqueName: \"kubernetes.io/projected/d40d3ad5-0159-4c56-9358-5f34a251cfa4-kube-api-access-rw2ln\") pod \"payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x\" (UID: \"d40d3ad5-0159-4c56-9358-5f34a251cfa4\") " pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" Oct 30 19:58:03.778903 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.778864 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40d3ad5-0159-4c56-9358-5f34a251cfa4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x\" (UID: \"d40d3ad5-0159-4c56-9358-5f34a251cfa4\") " pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" Oct 30 19:58:03.972226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.940921 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw2ln\" (UniqueName: \"kubernetes.io/projected/d40d3ad5-0159-4c56-9358-5f34a251cfa4-kube-api-access-rw2ln\") pod \"payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x\" (UID: \"d40d3ad5-0159-4c56-9358-5f34a251cfa4\") " pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" Oct 30 19:58:03.972226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.941051 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40d3ad5-0159-4c56-9358-5f34a251cfa4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x\" (UID: \"d40d3ad5-0159-4c56-9358-5f34a251cfa4\") " pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" Oct 30 19:58:03.972226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.941167 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5px4\" (UniqueName: \"kubernetes.io/projected/d2b85134-5b96-4dba-8226-654ecbde00eb-kube-api-access-n5px4\") pod \"payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw\" (UID: \"d2b85134-5b96-4dba-8226-654ecbde00eb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" Oct 30 19:58:03.972226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.941281 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b85134-5b96-4dba-8226-654ecbde00eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw\" (UID: \"d2b85134-5b96-4dba-8226-654ecbde00eb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" Oct 30 19:58:03.972226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:03.942632 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40d3ad5-0159-4c56-9358-5f34a251cfa4-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x\" (UID: \"d40d3ad5-0159-4c56-9358-5f34a251cfa4\") " pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" Oct 30 19:58:04.013991 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:04.005715 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw2ln\" (UniqueName: \"kubernetes.io/projected/d40d3ad5-0159-4c56-9358-5f34a251cfa4-kube-api-access-rw2ln\") pod \"payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x\" (UID: \"d40d3ad5-0159-4c56-9358-5f34a251cfa4\") " pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" Oct 30 19:58:04.060170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:04.052754 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5px4\" (UniqueName: \"kubernetes.io/projected/d2b85134-5b96-4dba-8226-654ecbde00eb-kube-api-access-n5px4\") pod \"payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw\" (UID: \"d2b85134-5b96-4dba-8226-654ecbde00eb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" Oct 30 19:58:04.060170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:04.052891 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b85134-5b96-4dba-8226-654ecbde00eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw\" (UID: \"d2b85134-5b96-4dba-8226-654ecbde00eb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" Oct 30 19:58:04.060170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:04.054125 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b85134-5b96-4dba-8226-654ecbde00eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw\" (UID: \"d2b85134-5b96-4dba-8226-654ecbde00eb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" Oct 30 19:58:04.095173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:04.095003 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:04.120172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:04.114471 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" Oct 30 19:58:04.155170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:04.153115 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5px4\" (UniqueName: \"kubernetes.io/projected/d2b85134-5b96-4dba-8226-654ecbde00eb-kube-api-access-n5px4\") pod \"payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw\" (UID: \"d2b85134-5b96-4dba-8226-654ecbde00eb\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" Oct 30 19:58:04.392519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:04.392240 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" Oct 30 19:58:04.625170 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:58:04.623892 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40d3ad5_0159_4c56_9358_5f34a251cfa4.slice/crio-88b6dda5fb20e6c2aa9d7fdefb82bed11fbd318c885ed522d5cdde9df3ed87f6 WatchSource:0}: Error finding container 88b6dda5fb20e6c2aa9d7fdefb82bed11fbd318c885ed522d5cdde9df3ed87f6: Status 404 returned error can't find the container with id 88b6dda5fb20e6c2aa9d7fdefb82bed11fbd318c885ed522d5cdde9df3ed87f6 Oct 30 19:58:04.625439 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:04.625243 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x"] Oct 30 19:58:04.811333 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:04.810749 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw"] Oct 30 19:58:05.168688 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:05.168602 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:05.314747 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:05.314706 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" event={"ID":"d40d3ad5-0159-4c56-9358-5f34a251cfa4","Type":"ContainerStarted","Data":"55ef4af69a422f4d7b8e8819b3e8d91bff11c1efe00345fe1edd5021574d514a"} Oct 30 19:58:05.314964 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:05.314811 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" Oct 30 19:58:05.314964 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:05.314836 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" event={"ID":"d40d3ad5-0159-4c56-9358-5f34a251cfa4","Type":"ContainerStarted","Data":"88b6dda5fb20e6c2aa9d7fdefb82bed11fbd318c885ed522d5cdde9df3ed87f6"} Oct 30 19:58:05.316720 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:05.316689 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" event={"ID":"d2b85134-5b96-4dba-8226-654ecbde00eb","Type":"ContainerStarted","Data":"97d66f3609ed0d38ae5a78a7cb52777c492bb7648ebe197ea8b8e4cd0f9764f1"} Oct 30 19:58:05.316870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:05.316737 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" event={"ID":"d2b85134-5b96-4dba-8226-654ecbde00eb","Type":"ContainerStarted","Data":"62dd69dd4f4a06bd22978632e044a453d9f169e2272e38ef86bc671ae6ee5692"} Oct 30 19:58:05.316870 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:05.316776 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" Oct 30 19:58:05.344761 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:05.344710 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" podStartSLOduration=2.344695236 podStartE2EDuration="2.344695236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:05.339531591 +0000 UTC m=+1605.034739543" watchObservedRunningTime="2026-10-30 19:58:05.344695236 +0000 UTC m=+1605.039903196" Oct 30 19:58:05.371735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:05.371677 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" podStartSLOduration=2.371660832 podStartE2EDuration="2.371660832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:05.370976264 +0000 UTC m=+1605.066184222" watchObservedRunningTime="2026-10-30 19:58:05.371660832 +0000 UTC m=+1605.066868791" Oct 30 19:58:05.866850 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:05.866800 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:06.140217 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:06.138480 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:07.071112 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.071079 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:58:07.146125 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.146086 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:07.184067 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.184037 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/582eb068-27ce-4484-8aa0-02d5a409e103-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582eb068-27ce-4484-8aa0-02d5a409e103-istiod-ca-cert\") pod \"582eb068-27ce-4484-8aa0-02d5a409e103\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " Oct 30 19:58:07.184280 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.184115 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/582eb068-27ce-4484-8aa0-02d5a409e103-istio-token\" (UniqueName: \"kubernetes.io/projected/582eb068-27ce-4484-8aa0-02d5a409e103-istio-token\") pod \"582eb068-27ce-4484-8aa0-02d5a409e103\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " Oct 30 19:58:07.184280 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.184253 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-workload-socket\") pod \"582eb068-27ce-4484-8aa0-02d5a409e103\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " Oct 30 19:58:07.184420 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.184320 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-workload-certs\") pod \"582eb068-27ce-4484-8aa0-02d5a409e103\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " Oct 30 19:58:07.184420 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.184361 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/582eb068-27ce-4484-8aa0-02d5a409e103-kube-api-access-v45qc\" (UniqueName: \"kubernetes.io/projected/582eb068-27ce-4484-8aa0-02d5a409e103-kube-api-access-v45qc\") pod \"582eb068-27ce-4484-8aa0-02d5a409e103\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " Oct 30 19:58:07.184531 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.184473 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-istio-data\") pod \"582eb068-27ce-4484-8aa0-02d5a409e103\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " Oct 30 19:58:07.184531 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.184513 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/582eb068-27ce-4484-8aa0-02d5a409e103-istiod-ca-cert" pod "582eb068-27ce-4484-8aa0-02d5a409e103" (UID: "582eb068-27ce-4484-8aa0-02d5a409e103"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:58:07.184660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.184571 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-credential-socket\") pod \"582eb068-27ce-4484-8aa0-02d5a409e103\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " Oct 30 19:58:07.184660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.184623 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-workload-certs" pod "582eb068-27ce-4484-8aa0-02d5a409e103" (UID: "582eb068-27ce-4484-8aa0-02d5a409e103"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:07.184660 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.184638 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-workload-socket" pod "582eb068-27ce-4484-8aa0-02d5a409e103" (UID: "582eb068-27ce-4484-8aa0-02d5a409e103"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:07.184820 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.184667 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/582eb068-27ce-4484-8aa0-02d5a409e103-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582eb068-27ce-4484-8aa0-02d5a409e103-istio-podinfo\") pod \"582eb068-27ce-4484-8aa0-02d5a409e103\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " Oct 30 19:58:07.184820 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.184706 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-istio-envoy\") pod \"582eb068-27ce-4484-8aa0-02d5a409e103\" (UID: \"582eb068-27ce-4484-8aa0-02d5a409e103\") " Oct 30 19:58:07.184931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.184847 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-credential-socket" pod "582eb068-27ce-4484-8aa0-02d5a409e103" (UID: "582eb068-27ce-4484-8aa0-02d5a409e103"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:07.184931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.184864 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-istio-data" pod "582eb068-27ce-4484-8aa0-02d5a409e103" (UID: "582eb068-27ce-4484-8aa0-02d5a409e103"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:07.185479 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.185445 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/582eb068-27ce-4484-8aa0-02d5a409e103-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:07.185594 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.185487 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:07.185594 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.185503 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:07.185594 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.185531 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:07.185594 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.185547 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:07.187361 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.187333 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582eb068-27ce-4484-8aa0-02d5a409e103-kube-api-access-v45qc" pod "582eb068-27ce-4484-8aa0-02d5a409e103" (UID: "582eb068-27ce-4484-8aa0-02d5a409e103"). InnerVolumeSpecName "kube-api-access-v45qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:07.188280 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.188252 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-istio-envoy" pod "582eb068-27ce-4484-8aa0-02d5a409e103" (UID: "582eb068-27ce-4484-8aa0-02d5a409e103"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:58:07.188280 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.188257 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/582eb068-27ce-4484-8aa0-02d5a409e103-istio-podinfo" pod "582eb068-27ce-4484-8aa0-02d5a409e103" (UID: "582eb068-27ce-4484-8aa0-02d5a409e103"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:58:07.188462 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.188298 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/582eb068-27ce-4484-8aa0-02d5a409e103-istio-token" pod "582eb068-27ce-4484-8aa0-02d5a409e103" (UID: "582eb068-27ce-4484-8aa0-02d5a409e103"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:58:07.287540 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.287309 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v45qc\" (UniqueName: \"kubernetes.io/projected/582eb068-27ce-4484-8aa0-02d5a409e103-kube-api-access-v45qc\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:07.287540 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.287348 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/582eb068-27ce-4484-8aa0-02d5a409e103-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:07.287540 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.287365 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/582eb068-27ce-4484-8aa0-02d5a409e103-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:07.287540 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.287379 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/582eb068-27ce-4484-8aa0-02d5a409e103-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:58:07.337318 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.337280 5091 generic.go:356] "Generic (PLEG): container finished" podID="582eb068-27ce-4484-8aa0-02d5a409e103" containerID="e43fc0b8a7641c479ae06ac9de0200f914c1110e8e2edacfe21feae9918d0671" exitCode=0 Oct 30 19:58:07.337497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.337348 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" Oct 30 19:58:07.337497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.337355 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" event={"ID":"582eb068-27ce-4484-8aa0-02d5a409e103","Type":"ContainerDied","Data":"e43fc0b8a7641c479ae06ac9de0200f914c1110e8e2edacfe21feae9918d0671"} Oct 30 19:58:07.337497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.337408 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff" event={"ID":"582eb068-27ce-4484-8aa0-02d5a409e103","Type":"ContainerDied","Data":"c8d40908d29717778b6cdd094e98f7322503294a8b20a7938c7493bba35db6e3"} Oct 30 19:58:07.337497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.337429 5091 scope.go:122] "RemoveContainer" containerID="e43fc0b8a7641c479ae06ac9de0200f914c1110e8e2edacfe21feae9918d0671" Oct 30 19:58:07.350397 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.350371 5091 scope.go:122] "RemoveContainer" containerID="e43fc0b8a7641c479ae06ac9de0200f914c1110e8e2edacfe21feae9918d0671" Oct 30 19:58:07.350722 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:58:07.350697 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e43fc0b8a7641c479ae06ac9de0200f914c1110e8e2edacfe21feae9918d0671\": container with ID starting with e43fc0b8a7641c479ae06ac9de0200f914c1110e8e2edacfe21feae9918d0671 not found: ID does not exist" containerID="e43fc0b8a7641c479ae06ac9de0200f914c1110e8e2edacfe21feae9918d0671" Oct 30 19:58:07.350775 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.350734 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e43fc0b8a7641c479ae06ac9de0200f914c1110e8e2edacfe21feae9918d0671"} err="failed to get container status \"e43fc0b8a7641c479ae06ac9de0200f914c1110e8e2edacfe21feae9918d0671\": rpc error: code = NotFound desc = could not find container \"e43fc0b8a7641c479ae06ac9de0200f914c1110e8e2edacfe21feae9918d0671\": container with ID starting with e43fc0b8a7641c479ae06ac9de0200f914c1110e8e2edacfe21feae9918d0671 not found: ID does not exist" Oct 30 19:58:07.369197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.369160 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff"] Oct 30 19:58:07.375332 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:07.375302 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-4ghff"] Oct 30 19:58:08.120981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:08.115203 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:08.442206 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:08.442122 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="582eb068-27ce-4484-8aa0-02d5a409e103" path="/var/lib/kubelet/pods/582eb068-27ce-4484-8aa0-02d5a409e103/volumes" Oct 30 19:58:09.031591 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:09.021917 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:09.964168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:09.961631 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:10.889171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:10.884202 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:11.406799 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:11.406766 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:11.420023 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:11.419996 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:11.873333 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:11.873293 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:12.410321 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:12.410290 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:13.068612 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:13.059568 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:13.263824 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:13.263786 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:14.156277 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:14.156238 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:15.077075 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:15.076996 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:15.995165 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:58:15.994896 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:58:16.159067 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:16.158888 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:16.330265 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:16.330229 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" Oct 30 19:58:16.330570 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:16.330550 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" Oct 30 19:58:16.471196 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:16.471158 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:17.054464 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:17.054402 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb"] Oct 30 19:58:17.062448 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:17.062421 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f08a3c84-k9mqb"] Oct 30 19:58:17.520992 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:17.520951 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:18.443055 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:18.443022 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1897963b-68eb-4959-b391-16b595122405" path="/var/lib/kubelet/pods/1897963b-68eb-4959-b391-16b595122405/volumes" Oct 30 19:58:18.643005 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:18.642973 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:19.892597 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:19.892560 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:20.790967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:20.790920 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:21.831030 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:21.830993 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:22.188810 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.188722 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc"] Oct 30 19:58:22.189907 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.189878 5091 cpu_manager.go:414] "containerMap: removing container" podUID="582eb068-27ce-4484-8aa0-02d5a409e103" containerName="istio-proxy" Oct 30 19:58:22.190111 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.190085 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="582eb068-27ce-4484-8aa0-02d5a409e103" containerName="istio-proxy" Oct 30 19:58:22.190309 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.190292 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="582eb068-27ce-4484-8aa0-02d5a409e103" containerName="istio-proxy" Oct 30 19:58:22.195944 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.195919 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.204689 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.204664 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-7cbcbb\"/\"kube-root-ca.crt\"" Oct 30 19:58:22.206532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.206364 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-7cbcbb\"/\"openshift-service-ca.crt\"" Oct 30 19:58:22.206532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.206392 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-7cbcbb\"/\"default-dockercfg-ft2bn\"" Oct 30 19:58:22.206532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.206504 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-7cbcbb\"/\"auth-test-model-7cbcbb-kserve-self-signed-certs\"" Oct 30 19:58:22.207962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.207939 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc"] Oct 30 19:58:22.248121 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.248086 5091 scope.go:122] "RemoveContainer" containerID="8177dab00e2245eeb07f2d9fd020e64653d07c97dff98d42fbfc18ef679e08da" Oct 30 19:58:22.259549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.259529 5091 scope.go:122] "RemoveContainer" containerID="4c3db7eaa8f860ce6eebcc2bfea78bec262704c892ea34ebf36a30a2735a36ef" Oct 30 19:58:22.270696 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.270670 5091 scope.go:122] "RemoveContainer" containerID="6456ca098c6e82f3fc20fa6e5a845f1744d3fb6d139dcbabc8824ab4923c8075" Oct 30 19:58:22.283544 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.283437 5091 scope.go:122] "RemoveContainer" containerID="34fefb684f739b0da27fee7f02cb8ae87a240ddc08274806ea7fb1f2c217b445" Oct 30 19:58:22.298971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.298950 5091 scope.go:122] "RemoveContainer" containerID="23192f8ce42fb233ee559a275b4fb4f5818fba05bf41eea328fd76fd27fdcedf" Oct 30 19:58:22.301818 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.301795 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-tmp-dir\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.301949 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.301837 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480630c3-cfff-45e5-939d-caf7a5eaace6-tls-certs\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.302009 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.301984 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8c4z\" (UniqueName: \"kubernetes.io/projected/480630c3-cfff-45e5-939d-caf7a5eaace6-kube-api-access-x8c4z\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.302131 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.302114 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-model-cache\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.302294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.302276 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-home\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.302361 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.302337 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-dshm\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.310358 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.310336 5091 scope.go:122] "RemoveContainer" containerID="7eda6418d319208bcdf7b372b303f5aca4ce90f70a1bc3d2d2cd0e5ee428ca10" Oct 30 19:58:22.312039 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.311987 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:22.406290 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.406252 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-tmp-dir\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.406551 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.406530 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480630c3-cfff-45e5-939d-caf7a5eaace6-tls-certs\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.408624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.406683 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-tmp-dir\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.408624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.406699 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8c4z\" (UniqueName: \"kubernetes.io/projected/480630c3-cfff-45e5-939d-caf7a5eaace6-kube-api-access-x8c4z\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.408624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.406841 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-model-cache\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.408624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.406954 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-home\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.408624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.407022 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-dshm\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.408624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.407935 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-model-cache\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.408624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.408195 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-home\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.427801 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.427717 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-dshm\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.455354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.455278 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8c4z\" (UniqueName: \"kubernetes.io/projected/480630c3-cfff-45e5-939d-caf7a5eaace6-kube-api-access-x8c4z\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.498176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.497827 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480630c3-cfff-45e5-939d-caf7a5eaace6-tls-certs\") pod \"auth-test-model-7cbcbb-kserve-5498db68df-hcsrc\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.521176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.518835 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:22.941390 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:58:22.937312 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod480630c3_cfff_45e5_939d_caf7a5eaace6.slice/crio-2e0458354479178082018d37a237192d98d54fedc65cfaf6238d37b8d80d1108 WatchSource:0}: Error finding container 2e0458354479178082018d37a237192d98d54fedc65cfaf6238d37b8d80d1108: Status 404 returned error can't find the container with id 2e0458354479178082018d37a237192d98d54fedc65cfaf6238d37b8d80d1108 Oct 30 19:58:22.941390 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.937809 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:22.943349 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:22.943233 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc"] Oct 30 19:58:23.579967 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:23.579927 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" event={"ID":"480630c3-cfff-45e5-939d-caf7a5eaace6","Type":"ContainerStarted","Data":"d9bb6bd17b2e838c7bfa6699f956bef0015c007f81c84d6e70277911d056cbff"} Oct 30 19:58:23.580195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:23.579980 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" event={"ID":"480630c3-cfff-45e5-939d-caf7a5eaace6","Type":"ContainerStarted","Data":"2e0458354479178082018d37a237192d98d54fedc65cfaf6238d37b8d80d1108"} Oct 30 19:58:23.611441 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:23.611387 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" podStartSLOduration=1.6113731310000001 podStartE2EDuration="1.611373131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:23.610985673 +0000 UTC m=+1623.306193640" watchObservedRunningTime="2026-10-30 19:58:23.611373131 +0000 UTC m=+1623.306581088" Oct 30 19:58:23.986983 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:23.986892 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:24.519758 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:24.519732 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:24.915743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:24.915652 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:25.888168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:25.887690 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:26.908169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:26.907310 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:27.920800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:27.919129 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:28.565184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.565131 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb"] Oct 30 19:58:28.571370 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.571342 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.595056 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.595026 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb"] Oct 30 19:58:28.600573 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.600550 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm"] Oct 30 19:58:28.605273 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.605254 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.626017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.625988 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm"] Oct 30 19:58:28.647721 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.647692 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.647880 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.647831 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.647956 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.647936 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chsdr\" (UniqueName: \"kubernetes.io/projected/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-kube-api-access-chsdr\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.648080 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.647967 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.648080 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.648001 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.648080 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.648049 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.648276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.648080 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhnv5\" (UniqueName: \"kubernetes.io/projected/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-kube-api-access-fhnv5\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.648276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.648181 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.648276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.648228 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.648276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.648262 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.648495 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.648295 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.648495 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.648353 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.648495 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.648396 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.648495 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.648436 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.648495 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.648483 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.648762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.648507 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.648762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.648552 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.648762 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.648575 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.750383 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.750351 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.750542 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.750455 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chsdr\" (UniqueName: \"kubernetes.io/projected/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-kube-api-access-chsdr\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.750542 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.750484 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.750542 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.750512 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.750735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.750556 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.750735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.750585 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhnv5\" (UniqueName: \"kubernetes.io/projected/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-kube-api-access-fhnv5\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.750735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.750664 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.750735 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.750711 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.750981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.750746 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.750981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.750777 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.750981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.750780 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.750981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.750845 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.750981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.750859 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.751340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.751117 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.751340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.751175 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.751340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.751229 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-workload-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.751340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.751255 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.751340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.751285 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.751340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.751311 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.751340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.751337 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.751779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.751370 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.751779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.751396 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.751779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.751473 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istiod-ca-cert\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.751779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.751497 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.751779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.751541 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.751779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.751736 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-credential-socket\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.752101 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.751851 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-data\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.752101 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.751855 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-workload-certs\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.753888 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.753863 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.754008 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.753960 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-envoy\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.754639 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.754615 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.754639 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.754628 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-podinfo\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.771429 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.771402 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhnv5\" (UniqueName: \"kubernetes.io/projected/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-kube-api-access-fhnv5\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.775115 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.775093 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chsdr\" (UniqueName: \"kubernetes.io/projected/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-kube-api-access-chsdr\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.775754 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.775729 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.776874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.776852 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-token\") pod \"e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:28.896557 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.896472 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:28.935644 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:28.935455 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:29.114769 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.114732 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:29.259784 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.259746 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl"] Oct 30 19:58:29.296090 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.296056 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.305248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.305224 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-0eeba5-openshift-default-dockercfg-j457m\"" Oct 30 19:58:29.305652 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.305625 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:58:29.305788 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.305752 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:58:29.305862 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.305823 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:58:29.314952 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.312103 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl"] Oct 30 19:58:29.319083 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.318057 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb"] Oct 30 19:58:29.388425 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:58:29.388395 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff97dc89_e1e0_457c_8ef2_8a4684f36e31.slice/crio-1bf5863cee8c84c3f77d5853b64ade8ce45325264db383447f4e4ce2b6e4fb4e WatchSource:0}: Error finding container 1bf5863cee8c84c3f77d5853b64ade8ce45325264db383447f4e4ce2b6e4fb4e: Status 404 returned error can't find the container with id 1bf5863cee8c84c3f77d5853b64ade8ce45325264db383447f4e4ce2b6e4fb4e Oct 30 19:58:29.388779 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.388751 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm"] Oct 30 19:58:29.391803 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.391770 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:58:29.391900 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.391838 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:58:29.391900 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.391893 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:58:29.406745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.403452 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-data\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.406745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.403518 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istiod-ca-cert\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.406745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.403584 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-credential-socket\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.406745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.403653 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-workload-certs\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.406745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.403749 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-workload-socket\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.406745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.403786 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lgrx\" (UniqueName: \"kubernetes.io/projected/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-kube-api-access-7lgrx\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.406745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.403821 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-token\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.406745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.403917 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-envoy\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.406745 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.403942 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-podinfo\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.510610 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.509314 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-credential-socket\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.522724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.514316 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-workload-certs\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.522724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.514459 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-workload-socket\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.522724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.514503 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lgrx\" (UniqueName: \"kubernetes.io/projected/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-kube-api-access-7lgrx\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.522724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.514547 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-token\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.522724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.514673 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-envoy\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.522724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.514701 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-podinfo\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.522724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.514892 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-data\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.522724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.514957 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istiod-ca-cert\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.522724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.516192 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istiod-ca-cert\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.522724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.516522 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-workload-certs\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.522724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.516792 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-workload-socket\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.522724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.521247 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-envoy\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.522724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.521846 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-credential-socket\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.522724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.522646 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-data\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.536922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.526729 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-podinfo\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.552581 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.551801 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-token\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.575166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.565919 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lgrx\" (UniqueName: \"kubernetes.io/projected/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-kube-api-access-7lgrx\") pod \"e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.624918 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.624676 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:29.751804 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.751767 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" event={"ID":"ff97dc89-e1e0-457c-8ef2-8a4684f36e31","Type":"ContainerStarted","Data":"68832aa66e8f93e216fc70eaba91df2186e8add251452beab56420b434911714"} Oct 30 19:58:29.774254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.774220 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" event={"ID":"ff97dc89-e1e0-457c-8ef2-8a4684f36e31","Type":"ContainerStarted","Data":"1bf5863cee8c84c3f77d5853b64ade8ce45325264db383447f4e4ce2b6e4fb4e"} Oct 30 19:58:29.774518 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.774501 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" event={"ID":"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8","Type":"ContainerStarted","Data":"ab47c816ecf73eaca11a1d6a055b56ebc135ebbbb19b71ea81407d71998ff790"} Oct 30 19:58:29.784495 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.784435 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" event={"ID":"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8","Type":"ContainerStarted","Data":"4888f04372e69b28a165c87c94f4f320346b8e9067f425196d39c4f5fcd119d2"} Oct 30 19:58:29.823082 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.823024 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" podStartSLOduration=1.82300397 podStartE2EDuration="1.82300397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:29.808861778 +0000 UTC m=+1629.504069738" watchObservedRunningTime="2026-10-30 19:58:29.82300397 +0000 UTC m=+1629.518211928" Oct 30 19:58:29.874331 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.874256 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" podStartSLOduration=1.8742380669999998 podStartE2EDuration="1.874238067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:29.865912744 +0000 UTC m=+1629.561120704" watchObservedRunningTime="2026-10-30 19:58:29.874238067 +0000 UTC m=+1629.569446025" Oct 30 19:58:29.897427 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.897314 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:29.945526 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:29.945439 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:30.175237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:30.175203 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl"] Oct 30 19:58:30.186588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:30.186357 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:58:30.186588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:30.186445 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:58:30.186588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:30.186490 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:58:30.271713 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:30.271670 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:30.810172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:30.809252 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" event={"ID":"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4","Type":"ContainerStarted","Data":"3a44fe7d5fe6acc03144dca101f3f5928132b56d35fc7d5a8cc308327bd6ab82"} Oct 30 19:58:30.810172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:30.809309 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" event={"ID":"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4","Type":"ContainerStarted","Data":"1a1803844b3dd6cd2a9ce3f33fa30f1cb175f635ba6da2f51944416888c6e291"} Oct 30 19:58:30.903171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:30.902396 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:58:30.903171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:30.902467 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:58:30.938825 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:30.938790 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 19:58:30.939029 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:30.938842 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 19:58:31.638171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:31.637217 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:31.652588 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:31.651392 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:58:31.653408 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:31.652960 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" podUID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:58:31.904031 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:31.903556 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:58:31.904031 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:31.903621 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:58:31.931085 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:31.930986 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:31.944601 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:31.944566 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 19:58:31.944871 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:31.944846 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 19:58:32.222344 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:32.222239 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:32.403169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:32.402243 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" podUID="51c139a8-fc30-4991-baa6-75295965d8de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:32.429169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:32.413334 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" podUID="51c139a8-fc30-4991-baa6-75295965d8de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.227:8000/health\": context deadline exceeded" Oct 30 19:58:32.530169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:32.529430 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:32.636734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:32.629685 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:58:32.636734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:32.629765 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" podUID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:58:32.734173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:32.733264 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:32.877284 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:32.876236 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:32.912182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:32.911739 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:58:32.912182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:32.911799 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:58:32.951166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:32.947855 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 19:58:32.951166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:32.947937 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 19:58:33.141224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.128992 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": context deadline exceeded" Oct 30 19:58:33.144198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.144117 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" podStartSLOduration=4.1440954 podStartE2EDuration="4.1440954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:30.862385849 +0000 UTC m=+1630.557593807" watchObservedRunningTime="2026-10-30 19:58:33.1440954 +0000 UTC m=+1632.839303360" Oct 30 19:58:33.185166 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.177212 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx"] Oct 30 19:58:33.203256 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.203212 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" podUID="6794b918-54df-4217-b67a-26fec2eb5646" containerName="main" probeResult="failure" output="Get \"https://10.132.0.225:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:33.205746 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.205712 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:33.237457 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.237401 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 19:58:33.238915 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.238883 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 19:58:33.240331 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.239564 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:58:33.249955 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.242795 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e6786e6d-6zznx"] Oct 30 19:58:33.421348 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.420743 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" podUID="51c139a8-fc30-4991-baa6-75295965d8de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.227:8000/ready\": context deadline exceeded" Oct 30 19:58:33.422497 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.421830 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:33.592210 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.577513 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:33.592210 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.578012 5091 prober.go:122] "Probe failed" probeType="Startup" pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" podUID="480630c3-cfff-45e5-939d-caf7a5eaace6" containerName="main" probeResult="failure" output="Get \"https://10.132.0.230:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:33.676696 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.676606 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:58:33.676696 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.676667 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" podUID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:58:33.738962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.736676 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:33.738962 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:33.737193 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:58:34.095171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.094394 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 19:58:34.095171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.094461 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 19:58:34.126170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.116687 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:58:34.126170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.116743 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:58:34.215184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.211075 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" podUID="6794b918-54df-4217-b67a-26fec2eb5646" containerName="main" probeResult="failure" output="Get \"https://10.132.0.225:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:34.249170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.247617 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:34.288943 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.288916 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:58:34.634234 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.589304 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" podUID="51c139a8-fc30-4991-baa6-75295965d8de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:34.656553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.656508 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:58:34.678491 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.678447 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" podUID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:58:34.743171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.742230 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:34.834170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.833400 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90f72cc0-a860-47ae-90ae-2b15bbcb4135" path="/var/lib/kubelet/pods/90f72cc0-a860-47ae-90ae-2b15bbcb4135/volumes" Oct 30 19:58:34.920435 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.920340 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:58:34.920702 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.920672 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:58:34.936768 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.936733 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 19:58:34.937061 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:34.937034 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 19:58:35.149839 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:35.149775 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:35.254197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:35.253814 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 19:58:35.258220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:35.255057 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:35.292187 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:35.291930 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" podUID="6794b918-54df-4217-b67a-26fec2eb5646" containerName="main" probeResult="failure" output="Get \"https://10.132.0.225:8000/ready\": context deadline exceeded" Oct 30 19:58:35.302687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:35.300946 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:58:35.409861 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:35.409817 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:35.596411 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:35.596309 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" podUID="51c139a8-fc30-4991-baa6-75295965d8de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:35.638171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:35.636807 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:35.638171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:35.636880 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:58:35.661562 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:35.661120 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:58:35.661562 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:35.661196 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" podUID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:58:35.990228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:35.936435 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:58:35.990228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:35.936523 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:58:36.115356 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:36.021010 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 19:58:36.115356 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:36.021088 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 19:58:36.115356 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:36.052390 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:58:36.619272 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:36.619225 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" podUID="51c139a8-fc30-4991-baa6-75295965d8de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:36.827688 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:36.827652 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:58:36.828438 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:36.828403 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" podUID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:58:37.028367 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.028315 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:58:37.075818 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.075771 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:58:37.076506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.030572 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 19:58:37.076722 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.076698 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 19:58:37.141191 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.137978 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:37.281340 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.281237 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:37.334803 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.333286 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 19:58:37.363172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.335632 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:37.544459 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.423327 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" podUID="6794b918-54df-4217-b67a-26fec2eb5646" containerName="main" probeResult="failure" output="Get \"https://10.132.0.225:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:37.547386 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.518820 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/health\": context deadline exceeded" Oct 30 19:58:37.639984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.519009 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:37.639984 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.627352 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" podUID="51c139a8-fc30-4991-baa6-75295965d8de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:38.006834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.921608 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:58:38.006834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.921669 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" podUID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:58:38.006834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.921756 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:58:38.006834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.921779 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:58:38.006834 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:58:37.935709 5091 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.441s" Oct 30 19:58:38.006834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.938968 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 19:58:38.006834 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:37.939019 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 19:58:38.031173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:38.028873 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:38.142515 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:38.142473 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:38.336574 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:38.336471 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:38.358172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:38.346775 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:58:38.401498 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:38.401455 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:38.576167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:38.547293 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" podUID="6794b918-54df-4217-b67a-26fec2eb5646" containerName="main" probeResult="failure" output="Get \"https://10.132.0.225:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:38.579731 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:38.576448 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:38.627466 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:38.615219 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:58:38.652068 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:38.652041 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:58:38.672216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:38.650858 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" podUID="51c139a8-fc30-4991-baa6-75295965d8de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.227:8000/ready\": context deadline exceeded" Oct 30 19:58:38.696022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:38.651109 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:58:38.712681 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:38.712637 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" podUID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:58:38.998229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:38.997897 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" podUID="6794b918-54df-4217-b67a-26fec2eb5646" containerName="main" probeResult="failure" output="Get \"https://10.132.0.225:8000/health\": context deadline exceeded" Oct 30 19:58:39.034170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.030593 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:39.034170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.030738 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:58:39.034170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.030786 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:58:39.034170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.033582 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 19:58:39.034170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.033628 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 19:58:39.192195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.191733 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:39.294228 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.285822 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:39.464171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.457551 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:39.559224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.558282 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" podUID="6794b918-54df-4217-b67a-26fec2eb5646" containerName="main" probeResult="failure" output="Get \"https://10.132.0.225:8000/ready\": context deadline exceeded" Oct 30 19:58:39.630420 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.630312 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:39.636790 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.636295 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:58:39.636790 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.636358 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" podUID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:58:39.637975 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.637166 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:39.708287 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.708240 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" podUID="51c139a8-fc30-4991-baa6-75295965d8de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:39.740266 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.740219 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 19:58:39.953222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.937173 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 19:58:39.953222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.937231 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 19:58:40.029178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:39.955567 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:58:40.029519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:40.029481 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:58:40.289114 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:40.289072 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 19:58:40.290247 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:40.289721 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 19:58:40.290793 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:40.290776 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:58:40.638344 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:40.638236 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:40.725302 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:40.725248 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" podUID="51c139a8-fc30-4991-baa6-75295965d8de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.227:8000/ready\": context deadline exceeded" Oct 30 19:58:40.874230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:40.869716 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" start-of-body= Oct 30 19:58:40.913233 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:40.874498 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" podUID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.233:15021/healthz/ready\": dial tcp 10.132.0.233:15021: connect: connection refused" Oct 30 19:58:40.916634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:40.915709 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:40.916634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:40.916344 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" start-of-body= Oct 30 19:58:40.916634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:40.916392 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": dial tcp 10.132.0.231:15021: connect: connection refused" Oct 30 19:58:40.960323 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:40.958902 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" start-of-body= Oct 30 19:58:40.960323 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:40.958960 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": dial tcp 10.132.0.232:15021: connect: connection refused" Oct 30 19:58:40.996170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:40.993702 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 19:58:41.124125 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:41.124037 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:41.125596 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:41.125574 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 19:58:41.128582 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:41.128558 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 19:58:41.982922 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:41.982751 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:42.111219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:42.110238 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:42.111219 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:42.111219 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:42.112704 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:42.112678 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 19:58:42.536253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:42.536218 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:42.555332 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:42.555302 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 19:58:42.873052 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:42.872123 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:42.975171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:42.967276 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:58:42.975171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:42.967356 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.231:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:42.997648 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:42.997237 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 19:58:42.997648 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:42.997296 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.232:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 19:58:43.717989 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:43.717957 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:43.719362 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:43.719340 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:43.720424 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:43.720401 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 19:58:43.799970 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:43.798533 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:43.800507 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:43.800488 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:43.802904 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:43.802853 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 19:58:43.989397 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:43.989281 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:44.891167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:44.884690 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:45.784224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:45.784186 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:45.978051 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:58:45.977984 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:58:46.034763 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:46.034644 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z"] Oct 30 19:58:46.040456 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:46.040428 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8e1e28af-dmg8z"] Oct 30 19:58:46.447404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:46.447324 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea59a6da-e105-4dd4-9e37-2041ca75cffc" path="/var/lib/kubelet/pods/ea59a6da-e105-4dd4-9e37-2041ca75cffc/volumes" Oct 30 19:58:46.598494 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:46.597421 5091 patch_prober.go:29] interesting pod/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": read tcp 10.132.0.2:60764->10.132.0.226:15021: read: connection reset by peer" start-of-body= Oct 30 19:58:46.598494 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:46.597491 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" podUID="66a8e988-93ce-494f-8555-b36d6597e4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": read tcp 10.132.0.2:60764->10.132.0.226:15021: read: connection reset by peer" Oct 30 19:58:46.689678 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:46.689613 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:46.702109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:46.702009 5091 generic.go:356] "Generic (PLEG): container finished" podID="66a8e988-93ce-494f-8555-b36d6597e4db" containerID="aa5ccd61e0e19db09008e482125ba4530fc7ab0c1596e82144534b5c3fa5885c" exitCode=0 Oct 30 19:58:46.705415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:46.705356 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" event={"ID":"66a8e988-93ce-494f-8555-b36d6597e4db","Type":"ContainerDied","Data":"aa5ccd61e0e19db09008e482125ba4530fc7ab0c1596e82144534b5c3fa5885c"} Oct 30 19:58:46.706670 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:46.706651 5091 scope.go:122] "RemoveContainer" containerID="aa5ccd61e0e19db09008e482125ba4530fc7ab0c1596e82144534b5c3fa5885c" Oct 30 19:58:46.717171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:46.716238 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:58:46.717171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:46.716332 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:58:46.717171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:46.716403 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:58:47.664173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:47.663706 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:47.732480 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:47.732447 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" event={"ID":"66a8e988-93ce-494f-8555-b36d6597e4db","Type":"ContainerStarted","Data":"4f742f7e501f7b3995a3da05359e98d1b1011c06426294b7dc3eec54d7031fe6"} Oct 30 19:58:48.482260 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.482221 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc"] Oct 30 19:58:48.488681 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.488650 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.490276 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.490249 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc"] Oct 30 19:58:48.495927 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.495902 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f0e884\"/\"explicit-model-f0e884-kserve-self-signed-certs\"" Oct 30 19:58:48.496045 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.495989 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f0e884\"/\"kube-root-ca.crt\"" Oct 30 19:58:48.496109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.496050 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-f0e884\"/\"openshift-service-ca.crt\"" Oct 30 19:58:48.496186 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.496114 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-f0e884\"/\"default-dockercfg-7g99m\"" Oct 30 19:58:48.504687 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.504664 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:48.580042 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.580006 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-tls-certs\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.580236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.580185 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpqws\" (UniqueName: \"kubernetes.io/projected/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-kube-api-access-xpqws\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.580289 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.580277 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-tmp-dir\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.580342 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.580298 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-model-cache\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.580404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.580362 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-home\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.580404 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.580387 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-dshm\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.681668 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.681634 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-tmp-dir\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.681668 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.681673 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-model-cache\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.682262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.681735 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-home\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.682262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.681760 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-dshm\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.682262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.681907 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-tls-certs\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.682262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.682013 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpqws\" (UniqueName: \"kubernetes.io/projected/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-kube-api-access-xpqws\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.682262 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.682175 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-home\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.682568 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.682413 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-tmp-dir\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.682823 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.682799 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-model-cache\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.684644 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.684622 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-dshm\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.684925 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.684908 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-tls-certs\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.696009 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.695987 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpqws\" (UniqueName: \"kubernetes.io/projected/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-kube-api-access-xpqws\") pod \"explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:48.720168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.717563 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:58:48.720168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.717687 5091 patch_prober.go:29] interesting pod/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 19:58:48.720168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.717733 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" podUID="66a8e988-93ce-494f-8555-b36d6597e4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 19:58:48.732172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.729214 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:48.807168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:48.806289 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:49.277023 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:49.276989 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc"] Oct 30 19:58:49.278438 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:58:49.278405 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ee3f3b0_e4a0_44c8_aa3a_1a527bc80127.slice/crio-6ea3299fc2092edeb6aa24efbed247d7c11a28d54a7bb0ec7fe7790f62ca503d WatchSource:0}: Error finding container 6ea3299fc2092edeb6aa24efbed247d7c11a28d54a7bb0ec7fe7790f62ca503d: Status 404 returned error can't find the container with id 6ea3299fc2092edeb6aa24efbed247d7c11a28d54a7bb0ec7fe7790f62ca503d Oct 30 19:58:49.716216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:49.716124 5091 patch_prober.go:29] interesting pod/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 19:58:49.716216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:49.716190 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" podUID="66a8e988-93ce-494f-8555-b36d6597e4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 19:58:49.819170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:49.813925 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:49.940197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:49.940123 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" event={"ID":"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127","Type":"ContainerStarted","Data":"2d6430012351c3c0157ed8f7054899d53b8ccc35457a11fe45153e657364270d"} Oct 30 19:58:49.940197 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:49.940183 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" event={"ID":"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127","Type":"ContainerStarted","Data":"6ea3299fc2092edeb6aa24efbed247d7c11a28d54a7bb0ec7fe7790f62ca503d"} Oct 30 19:58:49.982301 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:49.982166 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" podStartSLOduration=1.982131579 podStartE2EDuration="1.982131579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:49.977630912 +0000 UTC m=+1649.672838870" watchObservedRunningTime="2026-10-30 19:58:49.982131579 +0000 UTC m=+1649.677339539" Oct 30 19:58:50.716553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:50.716514 5091 patch_prober.go:29] interesting pod/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 19:58:50.716989 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:50.716584 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" podUID="66a8e988-93ce-494f-8555-b36d6597e4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 19:58:50.807424 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:50.807388 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:58:51.030413 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:51.030379 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:51.716442 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:51.716407 5091 patch_prober.go:29] interesting pod/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 19:58:51.716634 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:51.716461 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" podUID="66a8e988-93ce-494f-8555-b36d6597e4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 19:58:51.923182 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:51.918858 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:52.716953 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:52.716917 5091 patch_prober.go:29] interesting pod/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Oct 30 19:58:52.717443 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:52.716979 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" podUID="66a8e988-93ce-494f-8555-b36d6597e4db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Oct 30 19:58:52.921171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:52.909774 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:53.721464 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:53.721428 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:58:53.721970 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:53.721785 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:58:53.722600 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:53.722579 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:58:53.859730 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:53.855486 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:55.023176 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:55.021717 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:56.008746 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.008707 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:56.344509 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.344437 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc"] Oct 30 19:58:56.473190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.469438 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:56.598302 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.598208 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq"] Oct 30 19:58:56.625291 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.625250 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" Oct 30 19:58:56.632727 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.632699 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 19:58:56.635903 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.635870 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq"] Oct 30 19:58:56.678355 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.678329 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 19:58:56.697553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.697519 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05823dc4-7f08-4f66-a116-134b83bc2be5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq\" (UID: \"05823dc4-7f08-4f66-a116-134b83bc2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" Oct 30 19:58:56.697695 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.697680 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z2p6\" (UniqueName: \"kubernetes.io/projected/05823dc4-7f08-4f66-a116-134b83bc2be5-kube-api-access-4z2p6\") pod \"maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq\" (UID: \"05823dc4-7f08-4f66-a116-134b83bc2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" Oct 30 19:58:56.697753 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.697729 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05823dc4-7f08-4f66-a116-134b83bc2be5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq\" (UID: \"05823dc4-7f08-4f66-a116-134b83bc2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" Oct 30 19:58:56.798873 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.798841 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z2p6\" (UniqueName: \"kubernetes.io/projected/05823dc4-7f08-4f66-a116-134b83bc2be5-kube-api-access-4z2p6\") pod \"maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq\" (UID: \"05823dc4-7f08-4f66-a116-134b83bc2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" Oct 30 19:58:56.799054 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.798927 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05823dc4-7f08-4f66-a116-134b83bc2be5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq\" (UID: \"05823dc4-7f08-4f66-a116-134b83bc2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" Oct 30 19:58:56.799133 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.799050 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05823dc4-7f08-4f66-a116-134b83bc2be5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq\" (UID: \"05823dc4-7f08-4f66-a116-134b83bc2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" Oct 30 19:58:56.799778 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.799754 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05823dc4-7f08-4f66-a116-134b83bc2be5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq\" (UID: \"05823dc4-7f08-4f66-a116-134b83bc2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" Oct 30 19:58:56.801464 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.801436 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05823dc4-7f08-4f66-a116-134b83bc2be5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq\" (UID: \"05823dc4-7f08-4f66-a116-134b83bc2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" Oct 30 19:58:56.813447 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.813417 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z2p6\" (UniqueName: \"kubernetes.io/projected/05823dc4-7f08-4f66-a116-134b83bc2be5-kube-api-access-4z2p6\") pod \"maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq\" (UID: \"05823dc4-7f08-4f66-a116-134b83bc2be5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" Oct 30 19:58:56.950211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:56.950089 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" Oct 30 19:58:57.239480 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:58:57.239450 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05823dc4_7f08_4f66_a116_134b83bc2be5.slice/crio-46cdf618ee7fea209bb05d23b0166f297ece7a204ac8ab7a8365f8bd952c08b0 WatchSource:0}: Error finding container 46cdf618ee7fea209bb05d23b0166f297ece7a204ac8ab7a8365f8bd952c08b0: Status 404 returned error can't find the container with id 46cdf618ee7fea209bb05d23b0166f297ece7a204ac8ab7a8365f8bd952c08b0 Oct 30 19:58:57.241641 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:57.241618 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq"] Oct 30 19:58:57.324925 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:57.322618 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:58.089326 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:58.089284 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" event={"ID":"05823dc4-7f08-4f66-a116-134b83bc2be5","Type":"ContainerStarted","Data":"45676de1b02051f1774b6e4c9f5ebfcd02713d209b4760b6c72f516ecbe623cc"} Oct 30 19:58:58.089751 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:58.089353 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" event={"ID":"05823dc4-7f08-4f66-a116-134b83bc2be5","Type":"ContainerStarted","Data":"46cdf618ee7fea209bb05d23b0166f297ece7a204ac8ab7a8365f8bd952c08b0"} Oct 30 19:58:58.124769 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:58.124711 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" podStartSLOduration=2.12469308 podStartE2EDuration="2.12469308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:58:58.118281787 +0000 UTC m=+1657.813489745" watchObservedRunningTime="2026-10-30 19:58:58.12469308 +0000 UTC m=+1657.819901042" Oct 30 19:58:58.290945 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:58:58.290914 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05823dc4_7f08_4f66_a116_134b83bc2be5.slice/crio-45676de1b02051f1774b6e4c9f5ebfcd02713d209b4760b6c72f516ecbe623cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05823dc4_7f08_4f66_a116_134b83bc2be5.slice/crio-conmon-45676de1b02051f1774b6e4c9f5ebfcd02713d209b4760b6c72f516ecbe623cc.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:58:58.408218 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:58.404251 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:58:59.028422 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:59.028390 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs"] Oct 30 19:58:59.033822 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:59.033798 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6cdeda8-9cvrs"] Oct 30 19:58:59.101672 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:59.101633 5091 generic.go:356] "Generic (PLEG): container finished" podID="05823dc4-7f08-4f66-a116-134b83bc2be5" containerID="45676de1b02051f1774b6e4c9f5ebfcd02713d209b4760b6c72f516ecbe623cc" exitCode=0 Oct 30 19:58:59.102109 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:59.101712 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" event={"ID":"05823dc4-7f08-4f66-a116-134b83bc2be5","Type":"ContainerDied","Data":"45676de1b02051f1774b6e4c9f5ebfcd02713d209b4760b6c72f516ecbe623cc"} Oct 30 19:58:59.280229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:58:59.278834 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:00.303395 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:00.303370 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" Oct 30 19:59:00.375483 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:00.375371 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05823dc4-7f08-4f66-a116-134b83bc2be5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05823dc4-7f08-4f66-a116-134b83bc2be5-maas-api-openshift-ca\") pod \"05823dc4-7f08-4f66-a116-134b83bc2be5\" (UID: \"05823dc4-7f08-4f66-a116-134b83bc2be5\") " Oct 30 19:59:00.375707 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:00.375689 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05823dc4-7f08-4f66-a116-134b83bc2be5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05823dc4-7f08-4f66-a116-134b83bc2be5-maas-api-xks-ca\") pod \"05823dc4-7f08-4f66-a116-134b83bc2be5\" (UID: \"05823dc4-7f08-4f66-a116-134b83bc2be5\") " Oct 30 19:59:00.375790 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:00.375732 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05823dc4-7f08-4f66-a116-134b83bc2be5-kube-api-access-4z2p6\" (UniqueName: \"kubernetes.io/projected/05823dc4-7f08-4f66-a116-134b83bc2be5-kube-api-access-4z2p6\") pod \"05823dc4-7f08-4f66-a116-134b83bc2be5\" (UID: \"05823dc4-7f08-4f66-a116-134b83bc2be5\") " Oct 30 19:59:00.381410 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:00.381367 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05823dc4-7f08-4f66-a116-134b83bc2be5-maas-api-openshift-ca" pod "05823dc4-7f08-4f66-a116-134b83bc2be5" (UID: "05823dc4-7f08-4f66-a116-134b83bc2be5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:00.392263 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:00.392208 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05823dc4-7f08-4f66-a116-134b83bc2be5-kube-api-access-4z2p6" pod "05823dc4-7f08-4f66-a116-134b83bc2be5" (UID: "05823dc4-7f08-4f66-a116-134b83bc2be5"). InnerVolumeSpecName "kube-api-access-4z2p6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:00.396523 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:00.396481 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05823dc4-7f08-4f66-a116-134b83bc2be5-maas-api-xks-ca" pod "05823dc4-7f08-4f66-a116-134b83bc2be5" (UID: "05823dc4-7f08-4f66-a116-134b83bc2be5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:00.443553 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:00.443520 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf" path="/var/lib/kubelet/pods/a8f83d61-a4f4-4a99-a021-2ca09dcb1dcf/volumes" Oct 30 19:59:00.501696 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:00.501410 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:00.508224 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:00.507954 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05823dc4-7f08-4f66-a116-134b83bc2be5-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:00.508488 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:00.508464 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z2p6\" (UniqueName: \"kubernetes.io/projected/05823dc4-7f08-4f66-a116-134b83bc2be5-kube-api-access-4z2p6\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:00.508617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:00.508604 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05823dc4-7f08-4f66-a116-134b83bc2be5-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:01.120867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:01.120787 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" Oct 30 19:59:01.120867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:01.120824 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq" event={"ID":"05823dc4-7f08-4f66-a116-134b83bc2be5","Type":"ContainerDied","Data":"46cdf618ee7fea209bb05d23b0166f297ece7a204ac8ab7a8365f8bd952c08b0"} Oct 30 19:59:01.121114 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:01.120884 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46cdf618ee7fea209bb05d23b0166f297ece7a204ac8ab7a8365f8bd952c08b0" Oct 30 19:59:01.381227 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:01.381079 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:02.233838 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:02.233106 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:03.093184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:03.093124 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:04.112075 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:04.112035 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:05.119170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:05.114879 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:06.102177 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:06.099128 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:06.854293 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:06.853846 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x"] Oct 30 19:59:06.977730 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:06.977676 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw"] Oct 30 19:59:07.033156 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:07.033104 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:07.318459 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:07.318302 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:08.391251 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:08.391210 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:09.092501 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:09.092461 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s"] Oct 30 19:59:09.106487 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:09.106417 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1d20fe46-f6f5s"] Oct 30 19:59:09.716178 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:09.709922 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:10.445931 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:10.445898 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b24b839-150f-467e-9114-a09bacf549d9" path="/var/lib/kubelet/pods/2b24b839-150f-467e-9114-a09bacf549d9/volumes" Oct 30 19:59:10.855169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:10.849798 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:11.375183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:11.371490 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" podUID="5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127" containerName="main" containerID="cri-o://2d6430012351c3c0157ed8f7054899d53b8ccc35457a11fe45153e657364270d" gracePeriod=45 Oct 30 19:59:11.524295 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:11.524264 5091 generic.go:356] "Generic (PLEG): container finished" podID="5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127" containerID="2d6430012351c3c0157ed8f7054899d53b8ccc35457a11fe45153e657364270d" exitCode=0 Oct 30 19:59:11.524582 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:11.524335 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" event={"ID":"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127","Type":"ContainerDied","Data":"2d6430012351c3c0157ed8f7054899d53b8ccc35457a11fe45153e657364270d"} Oct 30 19:59:11.928237 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:11.923959 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" podUID="d40d3ad5-0159-4c56-9358-5f34a251cfa4" containerName="payload-processing" containerID="cri-o://55ef4af69a422f4d7b8e8819b3e8d91bff11c1efe00345fe1edd5021574d514a" gracePeriod=25 Oct 30 19:59:11.935531 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:11.935490 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:11.992198 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:11.991524 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" podUID="d2b85134-5b96-4dba-8226-654ecbde00eb" containerName="payload-pre-processing" containerID="cri-o://97d66f3609ed0d38ae5a78a7cb52777c492bb7648ebe197ea8b8e4cd0f9764f1" gracePeriod=25 Oct 30 19:59:12.143725 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.143334 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:59:12.231767 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:59:12.231617 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40d3ad5_0159_4c56_9358_5f34a251cfa4.slice/crio-55ef4af69a422f4d7b8e8819b3e8d91bff11c1efe00345fe1edd5021574d514a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40d3ad5_0159_4c56_9358_5f34a251cfa4.slice/crio-conmon-55ef4af69a422f4d7b8e8819b3e8d91bff11c1efe00345fe1edd5021574d514a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b85134_5b96_4dba_8226_654ecbde00eb.slice/crio-97d66f3609ed0d38ae5a78a7cb52777c492bb7648ebe197ea8b8e4cd0f9764f1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b85134_5b96_4dba_8226_654ecbde00eb.slice/crio-conmon-97d66f3609ed0d38ae5a78a7cb52777c492bb7648ebe197ea8b8e4cd0f9764f1.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:59:12.231767 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:59:12.231717 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b85134_5b96_4dba_8226_654ecbde00eb.slice/crio-97d66f3609ed0d38ae5a78a7cb52777c492bb7648ebe197ea8b8e4cd0f9764f1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40d3ad5_0159_4c56_9358_5f34a251cfa4.slice/crio-55ef4af69a422f4d7b8e8819b3e8d91bff11c1efe00345fe1edd5021574d514a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b85134_5b96_4dba_8226_654ecbde00eb.slice/crio-conmon-97d66f3609ed0d38ae5a78a7cb52777c492bb7648ebe197ea8b8e4cd0f9764f1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40d3ad5_0159_4c56_9358_5f34a251cfa4.slice/crio-conmon-55ef4af69a422f4d7b8e8819b3e8d91bff11c1efe00345fe1edd5021574d514a.scope\": RecentStats: unable to find data in memory cache]" Oct 30 19:59:12.317706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.317664 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-tls-certs\") pod \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " Oct 30 19:59:12.317909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.317744 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-tmp-dir\") pod \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " Oct 30 19:59:12.317909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.317804 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-model-cache\") pod \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " Oct 30 19:59:12.317909 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.317855 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-home\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-home\") pod \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " Oct 30 19:59:12.318160 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.318123 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-dshm\") pod \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " Oct 30 19:59:12.318255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.318178 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-kube-api-access-xpqws\" (UniqueName: \"kubernetes.io/projected/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-kube-api-access-xpqws\") pod \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\" (UID: \"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127\") " Oct 30 19:59:12.319981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.319381 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-model-cache" pod "5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127" (UID: "5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:12.319981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.319585 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-tmp-dir" pod "5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127" (UID: "5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:12.319981 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.319758 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-home" pod "5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127" (UID: "5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:12.333659 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.333610 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-kube-api-access-xpqws" pod "5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127" (UID: "5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127"). InnerVolumeSpecName "kube-api-access-xpqws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:12.334030 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.333950 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-tls-certs" pod "5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127" (UID: "5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:59:12.335549 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.335516 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-dshm" pod "5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127" (UID: "5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:12.420949 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.420897 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-dshm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:12.420949 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.420932 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpqws\" (UniqueName: \"kubernetes.io/projected/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-kube-api-access-xpqws\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:12.420949 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.420946 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-tls-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:12.421241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.420959 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-tmp-dir\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:12.421241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.420977 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-model-cache\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:12.421241 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.420989 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127-home\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:12.818671 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.811725 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" event={"ID":"5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127","Type":"ContainerDied","Data":"6ea3299fc2092edeb6aa24efbed247d7c11a28d54a7bb0ec7fe7790f62ca503d"} Oct 30 19:59:12.818671 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.811801 5091 scope.go:122] "RemoveContainer" containerID="2d6430012351c3c0157ed8f7054899d53b8ccc35457a11fe45153e657364270d" Oct 30 19:59:12.818671 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.812090 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc" Oct 30 19:59:12.865171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.855497 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" Oct 30 19:59:12.943828 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.942495 5091 generic.go:356] "Generic (PLEG): container finished" podID="d40d3ad5-0159-4c56-9358-5f34a251cfa4" containerID="55ef4af69a422f4d7b8e8819b3e8d91bff11c1efe00345fe1edd5021574d514a" exitCode=0 Oct 30 19:59:12.943828 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.942621 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" event={"ID":"d40d3ad5-0159-4c56-9358-5f34a251cfa4","Type":"ContainerDied","Data":"55ef4af69a422f4d7b8e8819b3e8d91bff11c1efe00345fe1edd5021574d514a"} Oct 30 19:59:12.944505 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:12.944264 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:13.019609 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.019577 5091 scope.go:122] "RemoveContainer" containerID="55ef4af69a422f4d7b8e8819b3e8d91bff11c1efe00345fe1edd5021574d514a" Oct 30 19:59:13.024022 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.023998 5091 generic.go:356] "Generic (PLEG): container finished" podID="d2b85134-5b96-4dba-8226-654ecbde00eb" containerID="97d66f3609ed0d38ae5a78a7cb52777c492bb7648ebe197ea8b8e4cd0f9764f1" exitCode=0 Oct 30 19:59:13.024282 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.024260 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" event={"ID":"d2b85134-5b96-4dba-8226-654ecbde00eb","Type":"ContainerDied","Data":"97d66f3609ed0d38ae5a78a7cb52777c492bb7648ebe197ea8b8e4cd0f9764f1"} Oct 30 19:59:13.043790 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.043768 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" Oct 30 19:59:13.054013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.052903 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d40d3ad5-0159-4c56-9358-5f34a251cfa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40d3ad5-0159-4c56-9358-5f34a251cfa4-plugins-config-volume\") pod \"d40d3ad5-0159-4c56-9358-5f34a251cfa4\" (UID: \"d40d3ad5-0159-4c56-9358-5f34a251cfa4\") " Oct 30 19:59:13.054013 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.053118 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d40d3ad5-0159-4c56-9358-5f34a251cfa4-kube-api-access-rw2ln\" (UniqueName: \"kubernetes.io/projected/d40d3ad5-0159-4c56-9358-5f34a251cfa4-kube-api-access-rw2ln\") pod \"d40d3ad5-0159-4c56-9358-5f34a251cfa4\" (UID: \"d40d3ad5-0159-4c56-9358-5f34a251cfa4\") " Oct 30 19:59:13.068167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.066247 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40d3ad5-0159-4c56-9358-5f34a251cfa4-plugins-config-volume" pod "d40d3ad5-0159-4c56-9358-5f34a251cfa4" (UID: "d40d3ad5-0159-4c56-9358-5f34a251cfa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:13.068167 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.066739 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc"] Oct 30 19:59:13.097170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.094481 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40d3ad5-0159-4c56-9358-5f34a251cfa4-kube-api-access-rw2ln" pod "d40d3ad5-0159-4c56-9358-5f34a251cfa4" (UID: "d40d3ad5-0159-4c56-9358-5f34a251cfa4"). InnerVolumeSpecName "kube-api-access-rw2ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:13.104216 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.102565 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-f0e884/explicit-model-f0e884-kserve-74d5fdbd4f-4jqbc"] Oct 30 19:59:13.157175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.154701 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b85134-5b96-4dba-8226-654ecbde00eb-kube-api-access-n5px4\" (UniqueName: \"kubernetes.io/projected/d2b85134-5b96-4dba-8226-654ecbde00eb-kube-api-access-n5px4\") pod \"d2b85134-5b96-4dba-8226-654ecbde00eb\" (UID: \"d2b85134-5b96-4dba-8226-654ecbde00eb\") " Oct 30 19:59:13.157175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.154967 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b85134-5b96-4dba-8226-654ecbde00eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b85134-5b96-4dba-8226-654ecbde00eb-plugins-config-volume\") pod \"d2b85134-5b96-4dba-8226-654ecbde00eb\" (UID: \"d2b85134-5b96-4dba-8226-654ecbde00eb\") " Oct 30 19:59:13.157175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.155662 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40d3ad5-0159-4c56-9358-5f34a251cfa4-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:13.157175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.155686 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw2ln\" (UniqueName: \"kubernetes.io/projected/d40d3ad5-0159-4c56-9358-5f34a251cfa4-kube-api-access-rw2ln\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:13.157175 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.156401 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b85134-5b96-4dba-8226-654ecbde00eb-plugins-config-volume" pod "d2b85134-5b96-4dba-8226-654ecbde00eb" (UID: "d2b85134-5b96-4dba-8226-654ecbde00eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:13.178323 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.178292 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b85134-5b96-4dba-8226-654ecbde00eb-kube-api-access-n5px4" pod "d2b85134-5b96-4dba-8226-654ecbde00eb" (UID: "d2b85134-5b96-4dba-8226-654ecbde00eb"). InnerVolumeSpecName "kube-api-access-n5px4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:13.264232 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.259651 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5px4\" (UniqueName: \"kubernetes.io/projected/d2b85134-5b96-4dba-8226-654ecbde00eb-kube-api-access-n5px4\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:13.264232 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.259683 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2b85134-5b96-4dba-8226-654ecbde00eb-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:13.317816 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.317415 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:13.898835 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:13.898783 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:14.037352 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:14.037320 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" Oct 30 19:59:14.037352 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:14.037343 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x" event={"ID":"d40d3ad5-0159-4c56-9358-5f34a251cfa4","Type":"ContainerDied","Data":"88b6dda5fb20e6c2aa9d7fdefb82bed11fbd318c885ed522d5cdde9df3ed87f6"} Oct 30 19:59:14.039433 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:14.039273 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" event={"ID":"d2b85134-5b96-4dba-8226-654ecbde00eb","Type":"ContainerDied","Data":"62dd69dd4f4a06bd22978632e044a453d9f169e2272e38ef86bc671ae6ee5692"} Oct 30 19:59:14.039433 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:14.039321 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw" Oct 30 19:59:14.039433 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:14.039347 5091 scope.go:122] "RemoveContainer" containerID="97d66f3609ed0d38ae5a78a7cb52777c492bb7648ebe197ea8b8e4cd0f9764f1" Oct 30 19:59:14.098706 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:14.098672 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x"] Oct 30 19:59:14.106372 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:14.106340 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f0e884-b8df455b8-wb58x"] Oct 30 19:59:14.123622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:14.123590 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw"] Oct 30 19:59:14.128422 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:14.128398 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f0e884-9f9499899-rlbsw"] Oct 30 19:59:14.447715 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:14.447679 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127" path="/var/lib/kubelet/pods/5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127/volumes" Oct 30 19:59:14.448615 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:14.448590 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b85134-5b96-4dba-8226-654ecbde00eb" path="/var/lib/kubelet/pods/d2b85134-5b96-4dba-8226-654ecbde00eb/volumes" Oct 30 19:59:14.449221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:14.449167 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d40d3ad5-0159-4c56-9358-5f34a251cfa4" path="/var/lib/kubelet/pods/d40d3ad5-0159-4c56-9358-5f34a251cfa4/volumes" Oct 30 19:59:15.016717 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:15.016678 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:15.976256 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:15.975103 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:16.066788 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:59:16.066745 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:59:16.560168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:16.558490 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:16.730528 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:16.730489 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:16.917250 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:16.917114 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:17.832643 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:17.832554 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:18.114893 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:18.114809 5091 generic.go:356] "Generic (PLEG): container finished" podID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" containerID="03b46fa6422e9f1fa0bfffa7e75b2f5fb1197c945e6d84456ea2933b605dae84" exitCode=0 Oct 30 19:59:18.114893 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:18.114881 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" event={"ID":"9d750d29-4fd8-4bd5-b879-ff33ba319afc","Type":"ContainerDied","Data":"03b46fa6422e9f1fa0bfffa7e75b2f5fb1197c945e6d84456ea2933b605dae84"} Oct 30 19:59:18.115426 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:18.115407 5091 scope.go:122] "RemoveContainer" containerID="03b46fa6422e9f1fa0bfffa7e75b2f5fb1197c945e6d84456ea2933b605dae84" Oct 30 19:59:18.116887 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:18.116862 5091 generic.go:356] "Generic (PLEG): container finished" podID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerID="4ca9093b95260f9f6ff042b6e39d3dac8c32c65ef715112cebd541e805140ce5" exitCode=0 Oct 30 19:59:18.116983 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:18.116935 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" event={"ID":"3d5e6444-233c-4c4e-b82f-559b46e46a38","Type":"ContainerDied","Data":"4ca9093b95260f9f6ff042b6e39d3dac8c32c65ef715112cebd541e805140ce5"} Oct 30 19:59:18.117118 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:18.117089 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:59:18.117238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:18.117172 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:59:18.117238 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:18.117206 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:59:18.117332 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:18.117317 5091 scope.go:122] "RemoveContainer" containerID="4ca9093b95260f9f6ff042b6e39d3dac8c32c65ef715112cebd541e805140ce5" Oct 30 19:59:18.118784 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:18.118758 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:59:18.118866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:18.118820 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:59:18.118866 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:18.118858 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 19:59:18.905066 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:18.905034 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:19.128743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:19.128704 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" event={"ID":"9d750d29-4fd8-4bd5-b879-ff33ba319afc","Type":"ContainerStarted","Data":"daaec18d4a1bc97f4f8a355474073ef66e2787785128308e8465aaebe009697e"} Oct 30 19:59:19.130954 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:19.130926 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" event={"ID":"3d5e6444-233c-4c4e-b82f-559b46e46a38","Type":"ContainerStarted","Data":"06d5f5db822feef54d4220bc708d3a9e052d9680ac974faa5775cca2b51a8c2a"} Oct 30 19:59:19.432331 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:19.432301 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:59:19.433306 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:19.433281 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 19:59:19.433407 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:19.433325 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" podUID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 19:59:19.772165 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:19.770795 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:19.966169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:19.950510 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:59:19.966169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:19.950646 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 19:59:19.966169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:19.950687 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" podUID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 19:59:20.432487 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:20.432450 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 19:59:20.432681 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:20.432504 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" podUID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 19:59:20.714252 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:20.714168 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:20.960171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:20.955601 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 19:59:20.960171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:20.955663 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" podUID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 19:59:21.438092 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:21.438058 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 19:59:21.438620 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:21.438130 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" podUID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 19:59:21.531253 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:21.531169 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:21.723222 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:21.722718 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:21.897170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:21.892213 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq"] Oct 30 19:59:21.904173 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:21.902714 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:21.930229 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:21.928913 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-f0e884-mkbhq"] Oct 30 19:59:21.957211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:21.956752 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 19:59:21.957211 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:21.956812 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" podUID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 19:59:22.200317 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:22.196273 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:22.343532 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:22.343500 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:22.431936 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:22.431893 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 19:59:22.432181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:22.431954 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" podUID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 19:59:22.442865 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:22.442839 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05823dc4-7f08-4f66-a116-134b83bc2be5" path="/var/lib/kubelet/pods/05823dc4-7f08-4f66-a116-134b83bc2be5/volumes" Oct 30 19:59:22.615174 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:22.600550 5091 scope.go:122] "RemoveContainer" containerID="873c58bd568ed4f4c113c8d45a4592a0a256723e1c31bb80e56af442d36cfc80" Oct 30 19:59:22.629595 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:22.629571 5091 scope.go:122] "RemoveContainer" containerID="bcdc8b20752c0a4838b6dd7f8cf85fa48cd70fd6a11b2551e9081b39a4496a1b" Oct 30 19:59:22.903220 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:22.896596 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:22.960171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:22.951562 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 19:59:22.960171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:22.951636 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" podUID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 19:59:23.090171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:23.089254 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp"] Oct 30 19:59:23.112169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:23.108463 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d1e132d-8csbp"] Oct 30 19:59:23.432023 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:23.431988 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 19:59:23.432230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:23.432050 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" podUID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 19:59:23.568183 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:23.566684 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:59:23.880396 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:23.879496 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:23.952169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:23.951883 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 19:59:23.952169 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:23.951955 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" podUID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 19:59:24.432230 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:24.432199 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 19:59:24.432415 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:24.432263 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" podUID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 19:59:24.442853 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:24.442825 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8" path="/var/lib/kubelet/pods/fd7ff1e6-af60-41f6-9cac-d90fa7ce3ce8/volumes" Oct 30 19:59:24.717254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:24.717164 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:24.953195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:24.950499 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" start-of-body= Oct 30 19:59:24.953195 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:24.950580 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" podUID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.220:15021/healthz/ready\": dial tcp 10.132.0.220:15021: connect: connection refused" Oct 30 19:59:25.432734 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:25.432689 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Oct 30 19:59:25.432933 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:25.432792 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" podUID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Oct 30 19:59:25.566945 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:25.566908 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:25.953853 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:25.953818 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:59:25.954411 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:25.954349 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:59:25.955294 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:25.955273 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 19:59:26.443754 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:26.443725 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:59:26.444772 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:26.444752 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 19:59:26.600578 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:26.600545 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:27.585839 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:27.585803 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:28.516181 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:28.516096 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:29.426081 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:29.426037 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:29.730119 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:29.730028 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:29.972651 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:29.972613 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:30.234602 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:30.165715 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn"] Oct 30 19:59:30.300179 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:30.283432 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" podUID="66a8e988-93ce-494f-8555-b36d6597e4db" containerName="istio-proxy" containerID="cri-o://4f742f7e501f7b3995a3da05359e98d1b1011c06426294b7dc3eec54d7031fe6" gracePeriod=30 Oct 30 19:59:30.576917 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:30.576720 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:30.942988 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:30.942894 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:31.296324 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:31.296281 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:31.555702 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:31.555616 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:31.731170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:31.727998 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:32.092172 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:32.091572 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:33.089251 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:33.087097 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:33.262743 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:33.262708 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:33.525622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:33.525379 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:34.051856 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:34.051817 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x"] Oct 30 19:59:34.056157 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:34.056116 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40fed590-xff7x"] Oct 30 19:59:34.289351 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:34.289317 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:34.445248 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:34.445176 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55ee3356-2860-4401-b287-40d98706cc1f" path="/var/lib/kubelet/pods/55ee3356-2860-4401-b287-40d98706cc1f/volumes" Oct 30 19:59:35.032938 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.032902 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c"] Oct 30 19:59:35.037505 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.037477 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea26fd20-bdh6c"] Oct 30 19:59:35.205168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.200271 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:35.494874 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.494782 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq"] Oct 30 19:59:35.495506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.495487 5091 cpu_manager.go:414] "containerMap: removing container" podUID="05823dc4-7f08-4f66-a116-134b83bc2be5" containerName="revoke-keys" Oct 30 19:59:35.495506 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.495506 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05823dc4-7f08-4f66-a116-134b83bc2be5" containerName="revoke-keys" Oct 30 19:59:35.495656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.495519 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d2b85134-5b96-4dba-8226-654ecbde00eb" containerName="payload-pre-processing" Oct 30 19:59:35.495656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.495524 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b85134-5b96-4dba-8226-654ecbde00eb" containerName="payload-pre-processing" Oct 30 19:59:35.495656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.495550 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d40d3ad5-0159-4c56-9358-5f34a251cfa4" containerName="payload-processing" Oct 30 19:59:35.495656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.495555 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d40d3ad5-0159-4c56-9358-5f34a251cfa4" containerName="payload-processing" Oct 30 19:59:35.495656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.495574 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127" containerName="main" Oct 30 19:59:35.495656 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.495579 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127" containerName="main" Oct 30 19:59:35.495978 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.495692 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d40d3ad5-0159-4c56-9358-5f34a251cfa4" containerName="payload-processing" Oct 30 19:59:35.495978 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.495702 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ee3f3b0-e4a0-44c8-aa3a-1a527bc80127" containerName="main" Oct 30 19:59:35.495978 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.495713 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="05823dc4-7f08-4f66-a116-134b83bc2be5" containerName="revoke-keys" Oct 30 19:59:35.495978 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.495721 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b85134-5b96-4dba-8226-654ecbde00eb" containerName="payload-pre-processing" Oct 30 19:59:35.501975 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.501950 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.508303 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.508281 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-8fdea8\"/\"openshift-service-ca.crt\"" Oct 30 19:59:35.510190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.509016 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq"] Oct 30 19:59:35.510190 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.510163 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-8fdea8\"/\"kube-root-ca.crt\"" Oct 30 19:59:35.512345 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.512323 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-8fdea8\"/\"default-dockercfg-qt48n\"" Oct 30 19:59:35.512463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.512424 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-8fdea8\"/\"noten-model-8fdea8-kserve-self-signed-certs\"" Oct 30 19:59:35.594844 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.594809 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:35.641624 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.641590 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-model-cache\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.641802 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.641672 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-tmp-dir\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.641867 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.641829 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-home\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.641925 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.641876 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-dshm\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.642078 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.642059 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw6wn\" (UniqueName: \"kubernetes.io/projected/124d537d-04bc-45bf-866e-d0bf638a8cd2-kube-api-access-fw6wn\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.642236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.642219 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124d537d-04bc-45bf-866e-d0bf638a8cd2-tls-certs\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.754226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.750103 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-model-cache\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.754226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.750180 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-tmp-dir\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.754226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.750286 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-home\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.754226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.750323 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-dshm\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.754226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.750471 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw6wn\" (UniqueName: \"kubernetes.io/projected/124d537d-04bc-45bf-866e-d0bf638a8cd2-kube-api-access-fw6wn\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.754226 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.750575 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124d537d-04bc-45bf-866e-d0bf638a8cd2-tls-certs\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.758168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.757576 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-home\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.758168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.757832 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-model-cache\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.758168 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.758062 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-tmp-dir\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.778180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.768802 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124d537d-04bc-45bf-866e-d0bf638a8cd2-tls-certs\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.819826 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.819405 5091 generic.go:356] "Generic (PLEG): container finished" podID="66a8e988-93ce-494f-8555-b36d6597e4db" containerID="4f742f7e501f7b3995a3da05359e98d1b1011c06426294b7dc3eec54d7031fe6" exitCode=0 Oct 30 19:59:35.819826 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.819577 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" event={"ID":"66a8e988-93ce-494f-8555-b36d6597e4db","Type":"ContainerDied","Data":"4f742f7e501f7b3995a3da05359e98d1b1011c06426294b7dc3eec54d7031fe6"} Oct 30 19:59:35.819826 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.819639 5091 scope.go:122] "RemoveContainer" containerID="aa5ccd61e0e19db09008e482125ba4530fc7ab0c1596e82144534b5c3fa5885c" Oct 30 19:59:35.838668 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.838622 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-dshm\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:35.841917 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:35.841891 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw6wn\" (UniqueName: \"kubernetes.io/projected/124d537d-04bc-45bf-866e-d0bf638a8cd2-kube-api-access-fw6wn\") pod \"noten-model-8fdea8-kserve-64b8b59db9-hs7sq\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:36.119657 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.119630 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:36.284292 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.283594 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:36.343488 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.343194 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:59:36.445338 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.445305 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="819637f6-ac70-4a33-a78f-fa3a4b4712bd" path="/var/lib/kubelet/pods/819637f6-ac70-4a33-a78f-fa3a4b4712bd/volumes" Oct 30 19:59:36.473119 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.473087 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66a8e988-93ce-494f-8555-b36d6597e4db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a8e988-93ce-494f-8555-b36d6597e4db-istiod-ca-cert\") pod \"66a8e988-93ce-494f-8555-b36d6597e4db\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " Oct 30 19:59:36.473329 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.473192 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-istio-data\") pod \"66a8e988-93ce-494f-8555-b36d6597e4db\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " Oct 30 19:59:36.473329 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.473251 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a8e988-93ce-494f-8555-b36d6597e4db-kube-api-access-djr4g\" (UniqueName: \"kubernetes.io/projected/66a8e988-93ce-494f-8555-b36d6597e4db-kube-api-access-djr4g\") pod \"66a8e988-93ce-494f-8555-b36d6597e4db\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " Oct 30 19:59:36.473468 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.473357 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66a8e988-93ce-494f-8555-b36d6597e4db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a8e988-93ce-494f-8555-b36d6597e4db-istio-podinfo\") pod \"66a8e988-93ce-494f-8555-b36d6597e4db\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " Oct 30 19:59:36.473468 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.473386 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-workload-socket\") pod \"66a8e988-93ce-494f-8555-b36d6597e4db\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " Oct 30 19:59:36.473593 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.473479 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66a8e988-93ce-494f-8555-b36d6597e4db-istiod-ca-cert" pod "66a8e988-93ce-494f-8555-b36d6597e4db" (UID: "66a8e988-93ce-494f-8555-b36d6597e4db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:36.473593 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.473501 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66a8e988-93ce-494f-8555-b36d6597e4db-istio-token\" (UniqueName: \"kubernetes.io/projected/66a8e988-93ce-494f-8555-b36d6597e4db-istio-token\") pod \"66a8e988-93ce-494f-8555-b36d6597e4db\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " Oct 30 19:59:36.473593 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.473549 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-istio-data" pod "66a8e988-93ce-494f-8555-b36d6597e4db" (UID: "66a8e988-93ce-494f-8555-b36d6597e4db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:36.473593 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.473561 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-istio-envoy\") pod \"66a8e988-93ce-494f-8555-b36d6597e4db\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " Oct 30 19:59:36.473812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.473598 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-credential-socket\") pod \"66a8e988-93ce-494f-8555-b36d6597e4db\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " Oct 30 19:59:36.473812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.473672 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-workload-certs\") pod \"66a8e988-93ce-494f-8555-b36d6597e4db\" (UID: \"66a8e988-93ce-494f-8555-b36d6597e4db\") " Oct 30 19:59:36.473812 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.473765 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-workload-socket" pod "66a8e988-93ce-494f-8555-b36d6597e4db" (UID: "66a8e988-93ce-494f-8555-b36d6597e4db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:36.473993 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.473943 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-credential-socket" pod "66a8e988-93ce-494f-8555-b36d6597e4db" (UID: "66a8e988-93ce-494f-8555-b36d6597e4db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:36.474054 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.474041 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-workload-certs" pod "66a8e988-93ce-494f-8555-b36d6597e4db" (UID: "66a8e988-93ce-494f-8555-b36d6597e4db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:36.474423 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.474396 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66a8e988-93ce-494f-8555-b36d6597e4db-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:36.474423 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.474421 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:36.474622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.474437 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:36.474622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.474453 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:36.474622 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.474468 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:36.475833 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.475803 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66a8e988-93ce-494f-8555-b36d6597e4db-istio-podinfo" pod "66a8e988-93ce-494f-8555-b36d6597e4db" (UID: "66a8e988-93ce-494f-8555-b36d6597e4db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:59:36.475939 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.475917 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a8e988-93ce-494f-8555-b36d6597e4db-istio-token" pod "66a8e988-93ce-494f-8555-b36d6597e4db" (UID: "66a8e988-93ce-494f-8555-b36d6597e4db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:36.476117 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.476095 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66a8e988-93ce-494f-8555-b36d6597e4db-kube-api-access-djr4g" pod "66a8e988-93ce-494f-8555-b36d6597e4db" (UID: "66a8e988-93ce-494f-8555-b36d6597e4db"). InnerVolumeSpecName "kube-api-access-djr4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:36.476117 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.476105 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-istio-envoy" pod "66a8e988-93ce-494f-8555-b36d6597e4db" (UID: "66a8e988-93ce-494f-8555-b36d6597e4db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:36.556441 ip-10-0-132-214 kubenswrapper[5091]: W1030 19:59:36.556407 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod124d537d_04bc_45bf_866e_d0bf638a8cd2.slice/crio-d3383080e4bd0d4753f6e62bb8d47af8e2996d694713dfcea8dd29b05c50846d WatchSource:0}: Error finding container d3383080e4bd0d4753f6e62bb8d47af8e2996d694713dfcea8dd29b05c50846d: Status 404 returned error can't find the container with id d3383080e4bd0d4753f6e62bb8d47af8e2996d694713dfcea8dd29b05c50846d Oct 30 19:59:36.556801 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.556570 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq"] Oct 30 19:59:36.576567 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.576543 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djr4g\" (UniqueName: \"kubernetes.io/projected/66a8e988-93ce-494f-8555-b36d6597e4db-kube-api-access-djr4g\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:36.576649 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.576581 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66a8e988-93ce-494f-8555-b36d6597e4db-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:36.576649 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.576598 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66a8e988-93ce-494f-8555-b36d6597e4db-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:36.576649 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.576618 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66a8e988-93ce-494f-8555-b36d6597e4db-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:36.830605 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.830560 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" event={"ID":"124d537d-04bc-45bf-866e-d0bf638a8cd2","Type":"ContainerStarted","Data":"d0f27bc2b02766cfeb9208d7b0a2d98901a4982e5acc83c3bb09b974129ad9c3"} Oct 30 19:59:36.830804 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.830620 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" event={"ID":"124d537d-04bc-45bf-866e-d0bf638a8cd2","Type":"ContainerStarted","Data":"d3383080e4bd0d4753f6e62bb8d47af8e2996d694713dfcea8dd29b05c50846d"} Oct 30 19:59:36.832380 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.832356 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" Oct 30 19:59:36.832534 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.832415 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn" event={"ID":"66a8e988-93ce-494f-8555-b36d6597e4db","Type":"ContainerDied","Data":"89397d714afe1ec1f9d96e60a79fa79d94e9f2c699fe8ba798cb6580f9848678"} Oct 30 19:59:36.832534 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.832472 5091 scope.go:122] "RemoveContainer" containerID="4f742f7e501f7b3995a3da05359e98d1b1011c06426294b7dc3eec54d7031fe6" Oct 30 19:59:36.856994 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.856944 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" podStartSLOduration=1.856931372 podStartE2EDuration="1.856931372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:59:36.854317668 +0000 UTC m=+1696.549525646" watchObservedRunningTime="2026-10-30 19:59:36.856931372 +0000 UTC m=+1696.552139383" Oct 30 19:59:36.877803 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.877651 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn"] Oct 30 19:59:36.882463 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:36.882439 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-f0e884-openshift-default-6474bdb5df-4hzcn"] Oct 30 19:59:37.119816 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:37.119782 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 19:59:37.229604 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:37.229488 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:38.097025 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:38.096982 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:38.443446 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:38.443370 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66a8e988-93ce-494f-8555-b36d6597e4db" path="/var/lib/kubelet/pods/66a8e988-93ce-494f-8555-b36d6597e4db/volumes" Oct 30 19:59:38.997170 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:38.994139 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:39.927180 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:39.917215 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:40.779613 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:40.779542 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:40.813724 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:40.804938 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw"] Oct 30 19:59:40.845917 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:40.845875 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" containerID="cri-o://19f7d3dd2b17ccc22d8f28aa42604da8cc0469b9989e1e2778f9458d23974969" gracePeriod=30 Oct 30 19:59:41.622481 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:41.622437 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:42.565254 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:42.562232 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:43.580902 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:43.580680 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:44.516959 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:44.516735 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:45.398521 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:45.392873 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:45.984913 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:59:45.984878 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 19:59:46.056597 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.056566 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv"] Oct 30 19:59:46.061513 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.061478 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11cab416-lx2zv"] Oct 30 19:59:46.094221 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.092514 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq"] Oct 30 19:59:46.285863 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.285532 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:46.496313 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.481059 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:46.695223 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.669816 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:59:46.821081 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.821044 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="611c9ca9-d00a-47e3-9c77-caf4204b3b1c" path="/var/lib/kubelet/pods/611c9ca9-d00a-47e3-9c77-caf4204b3b1c/volumes" Oct 30 19:59:46.824519 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.824483 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-data\") pod \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " Oct 30 19:59:46.824684 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.824558 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-kube-api-access-mr7p8\" (UniqueName: \"kubernetes.io/projected/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-kube-api-access-mr7p8\") pod \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " Oct 30 19:59:46.824684 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.824626 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-workload-socket\") pod \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " Oct 30 19:59:46.824821 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.824712 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-workload-certs\") pod \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " Oct 30 19:59:46.824821 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.824781 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-podinfo\") pod \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " Oct 30 19:59:46.824929 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.824914 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-token\" (UniqueName: \"kubernetes.io/projected/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-token\") pod \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " Oct 30 19:59:46.824996 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.824941 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-credential-socket\") pod \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " Oct 30 19:59:46.825250 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.825228 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istiod-ca-cert\") pod \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " Oct 30 19:59:46.825617 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.825596 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-envoy\") pod \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\" (UID: \"56e73eeb-fd82-4fd3-86ab-ca818b7961a8\") " Oct 30 19:59:46.826383 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.826353 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istiod-ca-cert" pod "56e73eeb-fd82-4fd3-86ab-ca818b7961a8" (UID: "56e73eeb-fd82-4fd3-86ab-ca818b7961a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:59:46.826683 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.826659 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-data" pod "56e73eeb-fd82-4fd3-86ab-ca818b7961a8" (UID: "56e73eeb-fd82-4fd3-86ab-ca818b7961a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:46.827363 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.827336 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-credential-socket" pod "56e73eeb-fd82-4fd3-86ab-ca818b7961a8" (UID: "56e73eeb-fd82-4fd3-86ab-ca818b7961a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:46.827468 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.827422 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-workload-certs" pod "56e73eeb-fd82-4fd3-86ab-ca818b7961a8" (UID: "56e73eeb-fd82-4fd3-86ab-ca818b7961a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:46.827642 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.827618 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-workload-socket" pod "56e73eeb-fd82-4fd3-86ab-ca818b7961a8" (UID: "56e73eeb-fd82-4fd3-86ab-ca818b7961a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:46.829255 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.829234 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-podinfo" pod "56e73eeb-fd82-4fd3-86ab-ca818b7961a8" (UID: "56e73eeb-fd82-4fd3-86ab-ca818b7961a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 19:59:46.829354 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.829292 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-kube-api-access-mr7p8" pod "56e73eeb-fd82-4fd3-86ab-ca818b7961a8" (UID: "56e73eeb-fd82-4fd3-86ab-ca818b7961a8"). InnerVolumeSpecName "kube-api-access-mr7p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:46.829425 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.829359 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-envoy" pod "56e73eeb-fd82-4fd3-86ab-ca818b7961a8" (UID: "56e73eeb-fd82-4fd3-86ab-ca818b7961a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:59:46.829996 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.829973 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-token" pod "56e73eeb-fd82-4fd3-86ab-ca818b7961a8" (UID: "56e73eeb-fd82-4fd3-86ab-ca818b7961a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:59:46.898851 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.898793 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:46.928236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.928209 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:46.928236 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.928233 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:46.928421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.928243 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:46.928421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.928253 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:46.928421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.928261 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:46.928421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.928269 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:46.928421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.928288 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:46.928421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.928297 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr7p8\" (UniqueName: \"kubernetes.io/projected/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-kube-api-access-mr7p8\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:46.928421 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.928307 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e73eeb-fd82-4fd3-86ab-ca818b7961a8-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 19:59:46.950971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.950894 5091 generic.go:356] "Generic (PLEG): container finished" podID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerID="19f7d3dd2b17ccc22d8f28aa42604da8cc0469b9989e1e2778f9458d23974969" exitCode=0 Oct 30 19:59:46.950971 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.950957 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" Oct 30 19:59:46.951141 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.950966 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" event={"ID":"56e73eeb-fd82-4fd3-86ab-ca818b7961a8","Type":"ContainerDied","Data":"19f7d3dd2b17ccc22d8f28aa42604da8cc0469b9989e1e2778f9458d23974969"} Oct 30 19:59:46.951141 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.951030 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw" event={"ID":"56e73eeb-fd82-4fd3-86ab-ca818b7961a8","Type":"ContainerDied","Data":"d5386632bc57e5f838cee39a9f4e2dd7096ede616080f05da477f86cac629751"} Oct 30 19:59:46.951141 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.951057 5091 scope.go:122] "RemoveContainer" containerID="19f7d3dd2b17ccc22d8f28aa42604da8cc0469b9989e1e2778f9458d23974969" Oct 30 19:59:46.969533 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.965453 5091 scope.go:122] "RemoveContainer" containerID="19f7d3dd2b17ccc22d8f28aa42604da8cc0469b9989e1e2778f9458d23974969" Oct 30 19:59:46.970377 ip-10-0-132-214 kubenswrapper[5091]: E1030 19:59:46.970352 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19f7d3dd2b17ccc22d8f28aa42604da8cc0469b9989e1e2778f9458d23974969\": container with ID starting with 19f7d3dd2b17ccc22d8f28aa42604da8cc0469b9989e1e2778f9458d23974969 not found: ID does not exist" containerID="19f7d3dd2b17ccc22d8f28aa42604da8cc0469b9989e1e2778f9458d23974969" Oct 30 19:59:46.970492 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.970398 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19f7d3dd2b17ccc22d8f28aa42604da8cc0469b9989e1e2778f9458d23974969"} err="failed to get container status \"19f7d3dd2b17ccc22d8f28aa42604da8cc0469b9989e1e2778f9458d23974969\": rpc error: code = NotFound desc = could not find container \"19f7d3dd2b17ccc22d8f28aa42604da8cc0469b9989e1e2778f9458d23974969\": container with ID starting with 19f7d3dd2b17ccc22d8f28aa42604da8cc0469b9989e1e2778f9458d23974969 not found: ID does not exist" Oct 30 19:59:46.996729 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:46.996702 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw"] Oct 30 19:59:47.000877 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:47.000856 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2a3e1b-openshift-default-679cc6f7f7-b6qfw"] Oct 30 19:59:47.133374 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:47.133340 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:48.007038 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:48.007001 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:48.443267 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:48.443197 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" path="/var/lib/kubelet/pods/56e73eeb-fd82-4fd3-86ab-ca818b7961a8/volumes" Oct 30 19:59:48.869942 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:48.869529 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:49.035608 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:49.035575 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8"] Oct 30 19:59:49.039692 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:49.039667 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dafe6264-7k7p8"] Oct 30 19:59:49.896017 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:49.893205 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:50.443865 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:50.443834 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09c58301-710b-4000-a116-20e76db1292b" path="/var/lib/kubelet/pods/09c58301-710b-4000-a116-20e76db1292b/volumes" Oct 30 19:59:50.822171 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:50.809376 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:51.970800 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:51.970714 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:52.149223 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:52.143020 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:53.034534 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:53.034501 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc"] Oct 30 19:59:53.038842 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:53.038814 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-91ea7035-vf8nc"] Oct 30 19:59:53.210442 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:53.203248 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:54.163348 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:54.163305 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:54.442705 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:54.442613 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3427d142-5236-4930-b38e-911de853d9af" path="/var/lib/kubelet/pods/3427d142-5236-4930-b38e-911de853d9af/volumes" Oct 30 19:59:55.148092 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:55.148055 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:55.994824 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:55.991208 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:56.933184 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:56.933117 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:57.775742 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:57.775704 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:58.733999 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:58.733966 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 19:59:59.649142 ip-10-0-132-214 kubenswrapper[5091]: I1030 19:59:59.649090 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:00.157802 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.157769 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m"] Oct 30 20:00:00.158462 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.158446 5091 cpu_manager.go:414] "containerMap: removing container" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" Oct 30 20:00:00.158518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.158463 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" Oct 30 20:00:00.158518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.158478 5091 cpu_manager.go:414] "containerMap: removing container" podUID="66a8e988-93ce-494f-8555-b36d6597e4db" containerName="istio-proxy" Oct 30 20:00:00.158518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.158483 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a8e988-93ce-494f-8555-b36d6597e4db" containerName="istio-proxy" Oct 30 20:00:00.158635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.158625 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e73eeb-fd82-4fd3-86ab-ca818b7961a8" containerName="istio-proxy" Oct 30 20:00:00.158681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.158638 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a8e988-93ce-494f-8555-b36d6597e4db" containerName="istio-proxy" Oct 30 20:00:00.158681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.158645 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="66a8e988-93ce-494f-8555-b36d6597e4db" containerName="istio-proxy" Oct 30 20:00:00.164817 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.164789 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" Oct 30 20:00:00.165001 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.164825 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr"] Oct 30 20:00:00.165797 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.165774 5091 cpu_manager.go:414] "containerMap: removing container" podUID="66a8e988-93ce-494f-8555-b36d6597e4db" containerName="istio-proxy" Oct 30 20:00:00.165797 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.165796 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66a8e988-93ce-494f-8555-b36d6597e4db" containerName="istio-proxy" Oct 30 20:00:00.171201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.171173 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7"] Oct 30 20:00:00.171342 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.171249 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" Oct 30 20:00:00.176934 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.176912 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" Oct 30 20:00:00.190782 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.177533 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz"] Oct 30 20:00:00.190782 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.178662 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:00:00.190782 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.183765 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" Oct 30 20:00:00.201940 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.197516 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m"] Oct 30 20:00:00.205413 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.205382 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr"] Oct 30 20:00:00.214276 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.214230 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7"] Oct 30 20:00:00.224975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.223707 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/005e2b9f-1283-48d3-9b3d-ddd316161e8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-4stn7\" (UID: \"005e2b9f-1283-48d3-9b3d-ddd316161e8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" Oct 30 20:00:00.224975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.223798 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f62ee1-1e91-4ed1-a82b-c5371d65af02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m\" (UID: \"b6f62ee1-1e91-4ed1-a82b-c5371d65af02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" Oct 30 20:00:00.224975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.223868 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p22lp\" (UniqueName: \"kubernetes.io/projected/0f3161e3-ac92-40a9-bd2f-18384362eeb8-kube-api-access-p22lp\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr\" (UID: \"0f3161e3-ac92-40a9-bd2f-18384362eeb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" Oct 30 20:00:00.224975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.223920 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5888g\" (UniqueName: \"kubernetes.io/projected/b6f62ee1-1e91-4ed1-a82b-c5371d65af02-kube-api-access-5888g\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m\" (UID: \"b6f62ee1-1e91-4ed1-a82b-c5371d65af02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" Oct 30 20:00:00.224975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.223980 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t9ss\" (UniqueName: \"kubernetes.io/projected/254c3d23-0de3-46c3-bb52-92502e789102-kube-api-access-6t9ss\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz\" (UID: \"254c3d23-0de3-46c3-bb52-92502e789102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" Oct 30 20:00:00.224975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.224107 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3161e3-ac92-40a9-bd2f-18384362eeb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr\" (UID: \"0f3161e3-ac92-40a9-bd2f-18384362eeb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" Oct 30 20:00:00.224975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.224179 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254c3d23-0de3-46c3-bb52-92502e789102-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz\" (UID: \"254c3d23-0de3-46c3-bb52-92502e789102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" Oct 30 20:00:00.224975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.224277 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g2ht\" (UniqueName: \"kubernetes.io/projected/005e2b9f-1283-48d3-9b3d-ddd316161e8c-kube-api-access-2g2ht\") pod \"maas-api-key-cleanup-29889840-4stn7\" (UID: \"005e2b9f-1283-48d3-9b3d-ddd316161e8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" Oct 30 20:00:00.230048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.230021 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz"] Oct 30 20:00:00.258606 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.258566 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r"] Oct 30 20:00:00.304117 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.303356 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds"] Oct 30 20:00:00.305988 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.305316 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" Oct 30 20:00:00.311405 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.311364 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j"] Oct 30 20:00:00.311544 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.311469 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" Oct 30 20:00:00.316236 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.316206 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r"] Oct 30 20:00:00.316373 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.316244 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds"] Oct 30 20:00:00.316373 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.316308 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" Oct 30 20:00:00.321659 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.321628 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff"] Oct 30 20:00:00.326242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.325521 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3161e3-ac92-40a9-bd2f-18384362eeb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr\" (UID: \"0f3161e3-ac92-40a9-bd2f-18384362eeb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" Oct 30 20:00:00.326242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.325582 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254c3d23-0de3-46c3-bb52-92502e789102-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz\" (UID: \"254c3d23-0de3-46c3-bb52-92502e789102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" Oct 30 20:00:00.326242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.325676 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g2ht\" (UniqueName: \"kubernetes.io/projected/005e2b9f-1283-48d3-9b3d-ddd316161e8c-kube-api-access-2g2ht\") pod \"maas-api-key-cleanup-29889840-4stn7\" (UID: \"005e2b9f-1283-48d3-9b3d-ddd316161e8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" Oct 30 20:00:00.326242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.325783 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/005e2b9f-1283-48d3-9b3d-ddd316161e8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-4stn7\" (UID: \"005e2b9f-1283-48d3-9b3d-ddd316161e8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" Oct 30 20:00:00.326242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.325980 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f62ee1-1e91-4ed1-a82b-c5371d65af02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m\" (UID: \"b6f62ee1-1e91-4ed1-a82b-c5371d65af02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" Oct 30 20:00:00.326242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.326047 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p22lp\" (UniqueName: \"kubernetes.io/projected/0f3161e3-ac92-40a9-bd2f-18384362eeb8-kube-api-access-p22lp\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr\" (UID: \"0f3161e3-ac92-40a9-bd2f-18384362eeb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" Oct 30 20:00:00.326242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.326097 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5888g\" (UniqueName: \"kubernetes.io/projected/b6f62ee1-1e91-4ed1-a82b-c5371d65af02-kube-api-access-5888g\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m\" (UID: \"b6f62ee1-1e91-4ed1-a82b-c5371d65af02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" Oct 30 20:00:00.326242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.326179 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t9ss\" (UniqueName: \"kubernetes.io/projected/254c3d23-0de3-46c3-bb52-92502e789102-kube-api-access-6t9ss\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz\" (UID: \"254c3d23-0de3-46c3-bb52-92502e789102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" Oct 30 20:00:00.326742 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.326470 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254c3d23-0de3-46c3-bb52-92502e789102-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz\" (UID: \"254c3d23-0de3-46c3-bb52-92502e789102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" Oct 30 20:00:00.328195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.326803 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3161e3-ac92-40a9-bd2f-18384362eeb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr\" (UID: \"0f3161e3-ac92-40a9-bd2f-18384362eeb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" Oct 30 20:00:00.328195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.327350 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f62ee1-1e91-4ed1-a82b-c5371d65af02-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m\" (UID: \"b6f62ee1-1e91-4ed1-a82b-c5371d65af02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" Oct 30 20:00:00.328195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.327555 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/005e2b9f-1283-48d3-9b3d-ddd316161e8c-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889840-4stn7\" (UID: \"005e2b9f-1283-48d3-9b3d-ddd316161e8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" Oct 30 20:00:00.329973 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.329950 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j"] Oct 30 20:00:00.330107 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.330058 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" Oct 30 20:00:00.332195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.332176 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff"] Oct 30 20:00:00.341603 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.341582 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t9ss\" (UniqueName: \"kubernetes.io/projected/254c3d23-0de3-46c3-bb52-92502e789102-kube-api-access-6t9ss\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz\" (UID: \"254c3d23-0de3-46c3-bb52-92502e789102\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" Oct 30 20:00:00.343714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.343694 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p22lp\" (UniqueName: \"kubernetes.io/projected/0f3161e3-ac92-40a9-bd2f-18384362eeb8-kube-api-access-p22lp\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr\" (UID: \"0f3161e3-ac92-40a9-bd2f-18384362eeb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" Oct 30 20:00:00.345744 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.345717 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5888g\" (UniqueName: \"kubernetes.io/projected/b6f62ee1-1e91-4ed1-a82b-c5371d65af02-kube-api-access-5888g\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m\" (UID: \"b6f62ee1-1e91-4ed1-a82b-c5371d65af02\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" Oct 30 20:00:00.345870 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.345846 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g2ht\" (UniqueName: \"kubernetes.io/projected/005e2b9f-1283-48d3-9b3d-ddd316161e8c-kube-api-access-2g2ht\") pod \"maas-api-key-cleanup-29889840-4stn7\" (UID: \"005e2b9f-1283-48d3-9b3d-ddd316161e8c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" Oct 30 20:00:00.427995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.427912 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a64db5-dcf8-436d-b44a-3a4891da6bde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j\" (UID: \"65a64db5-dcf8-436d-b44a-3a4891da6bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" Oct 30 20:00:00.427995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.427960 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/125a6e67-31de-4110-8bb8-cdc0af811e46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff\" (UID: \"125a6e67-31de-4110-8bb8-cdc0af811e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" Oct 30 20:00:00.428280 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.428087 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca322c7-6812-4c40-af1b-62a23ebb4ffe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r\" (UID: \"7ca322c7-6812-4c40-af1b-62a23ebb4ffe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" Oct 30 20:00:00.428280 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.428205 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28f4p\" (UniqueName: \"kubernetes.io/projected/65a64db5-dcf8-436d-b44a-3a4891da6bde-kube-api-access-28f4p\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j\" (UID: \"65a64db5-dcf8-436d-b44a-3a4891da6bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" Oct 30 20:00:00.428374 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.428293 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd8wm\" (UniqueName: \"kubernetes.io/projected/7ca322c7-6812-4c40-af1b-62a23ebb4ffe-kube-api-access-wd8wm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r\" (UID: \"7ca322c7-6812-4c40-af1b-62a23ebb4ffe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" Oct 30 20:00:00.428374 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.428353 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6gsq\" (UniqueName: \"kubernetes.io/projected/96e701fd-f225-4347-8d3c-5be59458f0fb-kube-api-access-m6gsq\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds\" (UID: \"96e701fd-f225-4347-8d3c-5be59458f0fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" Oct 30 20:00:00.428438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.428410 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e701fd-f225-4347-8d3c-5be59458f0fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds\" (UID: \"96e701fd-f225-4347-8d3c-5be59458f0fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" Oct 30 20:00:00.428481 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.428467 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7x89\" (UniqueName: \"kubernetes.io/projected/125a6e67-31de-4110-8bb8-cdc0af811e46-kube-api-access-s7x89\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff\" (UID: \"125a6e67-31de-4110-8bb8-cdc0af811e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" Oct 30 20:00:00.499972 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.499939 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" Oct 30 20:00:00.506094 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.506072 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" Oct 30 20:00:00.529957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.529924 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a64db5-dcf8-436d-b44a-3a4891da6bde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j\" (UID: \"65a64db5-dcf8-436d-b44a-3a4891da6bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" Oct 30 20:00:00.530086 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.529985 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/125a6e67-31de-4110-8bb8-cdc0af811e46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff\" (UID: \"125a6e67-31de-4110-8bb8-cdc0af811e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" Oct 30 20:00:00.530163 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.530103 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca322c7-6812-4c40-af1b-62a23ebb4ffe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r\" (UID: \"7ca322c7-6812-4c40-af1b-62a23ebb4ffe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" Oct 30 20:00:00.530230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.530219 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28f4p\" (UniqueName: \"kubernetes.io/projected/65a64db5-dcf8-436d-b44a-3a4891da6bde-kube-api-access-28f4p\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j\" (UID: \"65a64db5-dcf8-436d-b44a-3a4891da6bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" Oct 30 20:00:00.530339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.530316 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd8wm\" (UniqueName: \"kubernetes.io/projected/7ca322c7-6812-4c40-af1b-62a23ebb4ffe-kube-api-access-wd8wm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r\" (UID: \"7ca322c7-6812-4c40-af1b-62a23ebb4ffe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" Oct 30 20:00:00.530470 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.530421 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6gsq\" (UniqueName: \"kubernetes.io/projected/96e701fd-f225-4347-8d3c-5be59458f0fb-kube-api-access-m6gsq\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds\" (UID: \"96e701fd-f225-4347-8d3c-5be59458f0fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" Oct 30 20:00:00.530549 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.530467 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e701fd-f225-4347-8d3c-5be59458f0fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds\" (UID: \"96e701fd-f225-4347-8d3c-5be59458f0fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" Oct 30 20:00:00.530549 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.530495 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7x89\" (UniqueName: \"kubernetes.io/projected/125a6e67-31de-4110-8bb8-cdc0af811e46-kube-api-access-s7x89\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff\" (UID: \"125a6e67-31de-4110-8bb8-cdc0af811e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" Oct 30 20:00:00.530710 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.530682 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a64db5-dcf8-436d-b44a-3a4891da6bde-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j\" (UID: \"65a64db5-dcf8-436d-b44a-3a4891da6bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" Oct 30 20:00:00.531517 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.531497 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca322c7-6812-4c40-af1b-62a23ebb4ffe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r\" (UID: \"7ca322c7-6812-4c40-af1b-62a23ebb4ffe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" Oct 30 20:00:00.531985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.531961 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e701fd-f225-4347-8d3c-5be59458f0fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds\" (UID: \"96e701fd-f225-4347-8d3c-5be59458f0fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" Oct 30 20:00:00.532735 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.532713 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/125a6e67-31de-4110-8bb8-cdc0af811e46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff\" (UID: \"125a6e67-31de-4110-8bb8-cdc0af811e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" Oct 30 20:00:00.533331 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.533305 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" Oct 30 20:00:00.543366 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.543138 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" Oct 30 20:00:00.548779 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.548749 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd8wm\" (UniqueName: \"kubernetes.io/projected/7ca322c7-6812-4c40-af1b-62a23ebb4ffe-kube-api-access-wd8wm\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r\" (UID: \"7ca322c7-6812-4c40-af1b-62a23ebb4ffe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" Oct 30 20:00:00.555491 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.551640 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7x89\" (UniqueName: \"kubernetes.io/projected/125a6e67-31de-4110-8bb8-cdc0af811e46-kube-api-access-s7x89\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff\" (UID: \"125a6e67-31de-4110-8bb8-cdc0af811e46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" Oct 30 20:00:00.558280 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.558228 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6gsq\" (UniqueName: \"kubernetes.io/projected/96e701fd-f225-4347-8d3c-5be59458f0fb-kube-api-access-m6gsq\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds\" (UID: \"96e701fd-f225-4347-8d3c-5be59458f0fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" Oct 30 20:00:00.574430 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.574399 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28f4p\" (UniqueName: \"kubernetes.io/projected/65a64db5-dcf8-436d-b44a-3a4891da6bde-kube-api-access-28f4p\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j\" (UID: \"65a64db5-dcf8-436d-b44a-3a4891da6bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" Oct 30 20:00:00.638856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.635357 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" Oct 30 20:00:00.642225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.641430 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" Oct 30 20:00:00.716196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.693159 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" Oct 30 20:00:00.716196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.697097 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:00.716196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.699501 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" Oct 30 20:00:00.936726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:00.936228 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz"] Oct 30 20:00:01.035025 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.034996 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7"] Oct 30 20:00:01.046844 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:00:01.046384 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod005e2b9f_1283_48d3_9b3d_ddd316161e8c.slice/crio-1c856bd88bd75a71f0ef0dc213db4a2da8dcce798b40b2e09c65f1ffe3b5f9fa WatchSource:0}: Error finding container 1c856bd88bd75a71f0ef0dc213db4a2da8dcce798b40b2e09c65f1ffe3b5f9fa: Status 404 returned error can't find the container with id 1c856bd88bd75a71f0ef0dc213db4a2da8dcce798b40b2e09c65f1ffe3b5f9fa Oct 30 20:00:01.050891 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.050871 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m"] Oct 30 20:00:01.059310 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.059281 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr"] Oct 30 20:00:01.126442 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.126407 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" podUID="124d537d-04bc-45bf-866e-d0bf638a8cd2" containerName="main" containerID="cri-o://d0f27bc2b02766cfeb9208d7b0a2d98901a4982e5acc83c3bb09b974129ad9c3" gracePeriod=45 Oct 30 20:00:01.157762 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.157727 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r"] Oct 30 20:00:01.209779 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.209745 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff"] Oct 30 20:00:01.211726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.211695 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" event={"ID":"005e2b9f-1283-48d3-9b3d-ddd316161e8c","Type":"ContainerStarted","Data":"1c856bd88bd75a71f0ef0dc213db4a2da8dcce798b40b2e09c65f1ffe3b5f9fa"} Oct 30 20:00:01.214399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.214372 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" event={"ID":"0f3161e3-ac92-40a9-bd2f-18384362eeb8","Type":"ContainerStarted","Data":"6a746fb5ac769e46b75610dabe5df8384a56dc67ad0103f81e64a6b3172a045d"} Oct 30 20:00:01.214997 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.214968 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j"] Oct 30 20:00:01.215757 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.215738 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" event={"ID":"254c3d23-0de3-46c3-bb52-92502e789102","Type":"ContainerStarted","Data":"52373f0523a530ee18015b730aec2ccec45c80194f98adbd8aac4f301d02a313"} Oct 30 20:00:01.216884 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.216864 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" event={"ID":"b6f62ee1-1e91-4ed1-a82b-c5371d65af02","Type":"ContainerStarted","Data":"a437c2a0d0e051a24abe818f5341c2f43c88fa7c73971f8314e1bf42af65208c"} Oct 30 20:00:01.229959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.229930 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds"] Oct 30 20:00:01.255447 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:00:01.254437 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a64db5_dcf8_436d_b44a_3a4891da6bde.slice/crio-a2187ad519dd029f3fb3ec56241fa0c31db61b26084a211498f74e81ba26e9f9 WatchSource:0}: Error finding container a2187ad519dd029f3fb3ec56241fa0c31db61b26084a211498f74e81ba26e9f9: Status 404 returned error can't find the container with id a2187ad519dd029f3fb3ec56241fa0c31db61b26084a211498f74e81ba26e9f9 Oct 30 20:00:01.698169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.693122 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:01.818712 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.818443 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 20:00:01.860118 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.858643 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/124d537d-04bc-45bf-866e-d0bf638a8cd2-kube-api-access-fw6wn\" (UniqueName: \"kubernetes.io/projected/124d537d-04bc-45bf-866e-d0bf638a8cd2-kube-api-access-fw6wn\") pod \"124d537d-04bc-45bf-866e-d0bf638a8cd2\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " Oct 30 20:00:01.860118 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.858742 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-dshm\") pod \"124d537d-04bc-45bf-866e-d0bf638a8cd2\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " Oct 30 20:00:01.860118 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.858791 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-model-cache\") pod \"124d537d-04bc-45bf-866e-d0bf638a8cd2\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " Oct 30 20:00:01.860118 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.858831 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-home\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-home\") pod \"124d537d-04bc-45bf-866e-d0bf638a8cd2\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " Oct 30 20:00:01.860118 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.858928 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/124d537d-04bc-45bf-866e-d0bf638a8cd2-tls-certs\" (UniqueName: \"kubernetes.io/secret/124d537d-04bc-45bf-866e-d0bf638a8cd2-tls-certs\") pod \"124d537d-04bc-45bf-866e-d0bf638a8cd2\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " Oct 30 20:00:01.860118 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.859026 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-tmp-dir\") pod \"124d537d-04bc-45bf-866e-d0bf638a8cd2\" (UID: \"124d537d-04bc-45bf-866e-d0bf638a8cd2\") " Oct 30 20:00:01.863301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.862945 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-model-cache" pod "124d537d-04bc-45bf-866e-d0bf638a8cd2" (UID: "124d537d-04bc-45bf-866e-d0bf638a8cd2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:01.863773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.863738 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-tmp-dir" pod "124d537d-04bc-45bf-866e-d0bf638a8cd2" (UID: "124d537d-04bc-45bf-866e-d0bf638a8cd2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:01.864113 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.863965 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-home" pod "124d537d-04bc-45bf-866e-d0bf638a8cd2" (UID: "124d537d-04bc-45bf-866e-d0bf638a8cd2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:01.867462 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.867423 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-dshm" pod "124d537d-04bc-45bf-866e-d0bf638a8cd2" (UID: "124d537d-04bc-45bf-866e-d0bf638a8cd2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:00:01.870453 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.870419 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/124d537d-04bc-45bf-866e-d0bf638a8cd2-tls-certs" pod "124d537d-04bc-45bf-866e-d0bf638a8cd2" (UID: "124d537d-04bc-45bf-866e-d0bf638a8cd2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:01.898849 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.898804 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/124d537d-04bc-45bf-866e-d0bf638a8cd2-kube-api-access-fw6wn" pod "124d537d-04bc-45bf-866e-d0bf638a8cd2" (UID: "124d537d-04bc-45bf-866e-d0bf638a8cd2"). InnerVolumeSpecName "kube-api-access-fw6wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:01.962054 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.962021 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-tmp-dir\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:01.962194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.962059 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw6wn\" (UniqueName: \"kubernetes.io/projected/124d537d-04bc-45bf-866e-d0bf638a8cd2-kube-api-access-fw6wn\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:01.962194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.962075 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-dshm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:01.962194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.962089 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-model-cache\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:01.962194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.962102 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/124d537d-04bc-45bf-866e-d0bf638a8cd2-home\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:01.962194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:01.962116 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/124d537d-04bc-45bf-866e-d0bf638a8cd2-tls-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:02.227199 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.227097 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" event={"ID":"b6f62ee1-1e91-4ed1-a82b-c5371d65af02","Type":"ContainerStarted","Data":"c9b4f2ed2fda859307691bba7c510557d091ce20f806478ed8e2e731506dc68e"} Oct 30 20:00:02.228850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.228820 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" event={"ID":"65a64db5-dcf8-436d-b44a-3a4891da6bde","Type":"ContainerStarted","Data":"f334939ccf66eee8b857e23ddfcc7f3cbe8c9213d5a5dd2be59c4996f041188e"} Oct 30 20:00:02.229002 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.228866 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" event={"ID":"65a64db5-dcf8-436d-b44a-3a4891da6bde","Type":"ContainerStarted","Data":"a2187ad519dd029f3fb3ec56241fa0c31db61b26084a211498f74e81ba26e9f9"} Oct 30 20:00:02.230417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.230391 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" event={"ID":"005e2b9f-1283-48d3-9b3d-ddd316161e8c","Type":"ContainerStarted","Data":"2aee71f1baf25a3b49187378fdb676302a0d31cc0af4021123c2364dc48614a4"} Oct 30 20:00:02.231709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.231685 5091 generic.go:356] "Generic (PLEG): container finished" podID="124d537d-04bc-45bf-866e-d0bf638a8cd2" containerID="d0f27bc2b02766cfeb9208d7b0a2d98901a4982e5acc83c3bb09b974129ad9c3" exitCode=0 Oct 30 20:00:02.231822 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.231716 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" Oct 30 20:00:02.231822 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.231766 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" event={"ID":"124d537d-04bc-45bf-866e-d0bf638a8cd2","Type":"ContainerDied","Data":"d0f27bc2b02766cfeb9208d7b0a2d98901a4982e5acc83c3bb09b974129ad9c3"} Oct 30 20:00:02.231822 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.231804 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq" event={"ID":"124d537d-04bc-45bf-866e-d0bf638a8cd2","Type":"ContainerDied","Data":"d3383080e4bd0d4753f6e62bb8d47af8e2996d694713dfcea8dd29b05c50846d"} Oct 30 20:00:02.231965 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.231827 5091 scope.go:122] "RemoveContainer" containerID="d0f27bc2b02766cfeb9208d7b0a2d98901a4982e5acc83c3bb09b974129ad9c3" Oct 30 20:00:02.234029 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.234007 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" event={"ID":"96e701fd-f225-4347-8d3c-5be59458f0fb","Type":"ContainerStarted","Data":"f02651958d5544fdf41b4f2bd6186eff14c17796c27dfea726fbafeccf650a7b"} Oct 30 20:00:02.234207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.234063 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" event={"ID":"96e701fd-f225-4347-8d3c-5be59458f0fb","Type":"ContainerStarted","Data":"fd5b6ab69e72586dae640e76a34595ad4a39c22b497c105c9ccb96aa6509578d"} Oct 30 20:00:02.235781 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.235755 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" event={"ID":"0f3161e3-ac92-40a9-bd2f-18384362eeb8","Type":"ContainerStarted","Data":"9391ea7b490f948b3ceab574529d62d0df06f25e07b3aefaac96b972cf18a857"} Oct 30 20:00:02.238750 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.238724 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" event={"ID":"125a6e67-31de-4110-8bb8-cdc0af811e46","Type":"ContainerStarted","Data":"c2d368b5f83d2294dd5e21a35a603c96dd39c796d17f03b9e4e4d03b4a9b2075"} Oct 30 20:00:02.238861 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.238770 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" event={"ID":"125a6e67-31de-4110-8bb8-cdc0af811e46","Type":"ContainerStarted","Data":"ca44f22c51d5bb636a4baedab6bc3808e1e94ce7acb93e2a04fc4c55b779efb6"} Oct 30 20:00:02.240764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.240738 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" event={"ID":"254c3d23-0de3-46c3-bb52-92502e789102","Type":"ContainerStarted","Data":"d7d2c878874448a6af6bc9d3b9ca56f5d6d009d06662a827003df6e2fb4c0f47"} Oct 30 20:00:02.264198 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.264173 5091 scope.go:122] "RemoveContainer" containerID="d0f27bc2b02766cfeb9208d7b0a2d98901a4982e5acc83c3bb09b974129ad9c3" Oct 30 20:00:02.264980 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:02.264949 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f27bc2b02766cfeb9208d7b0a2d98901a4982e5acc83c3bb09b974129ad9c3\": container with ID starting with d0f27bc2b02766cfeb9208d7b0a2d98901a4982e5acc83c3bb09b974129ad9c3 not found: ID does not exist" containerID="d0f27bc2b02766cfeb9208d7b0a2d98901a4982e5acc83c3bb09b974129ad9c3" Oct 30 20:00:02.265099 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.264991 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f27bc2b02766cfeb9208d7b0a2d98901a4982e5acc83c3bb09b974129ad9c3"} err="failed to get container status \"d0f27bc2b02766cfeb9208d7b0a2d98901a4982e5acc83c3bb09b974129ad9c3\": rpc error: code = NotFound desc = could not find container \"d0f27bc2b02766cfeb9208d7b0a2d98901a4982e5acc83c3bb09b974129ad9c3\": container with ID starting with d0f27bc2b02766cfeb9208d7b0a2d98901a4982e5acc83c3bb09b974129ad9c3 not found: ID does not exist" Oct 30 20:00:02.265802 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.265752 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" podStartSLOduration=2.265738512 podStartE2EDuration="2.265738512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.265729161 +0000 UTC m=+1721.960937122" watchObservedRunningTime="2026-10-30 20:00:02.265738512 +0000 UTC m=+1721.960946471" Oct 30 20:00:02.267615 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.266089 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" event={"ID":"7ca322c7-6812-4c40-af1b-62a23ebb4ffe","Type":"ContainerStarted","Data":"9e2c90e60e60cd4a73b63c83bcf6ae207e107c54d9a9d8884ac46b69aa68adeb"} Oct 30 20:00:02.267615 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.266184 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" event={"ID":"7ca322c7-6812-4c40-af1b-62a23ebb4ffe","Type":"ContainerStarted","Data":"0a6679aae2688d1ef68e488971131f3f96ecd096ce5e9195d5f511028e3b3f37"} Oct 30 20:00:02.287886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.287831 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" podStartSLOduration=2.287818243 podStartE2EDuration="2.287818243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.286501475 +0000 UTC m=+1721.981709446" watchObservedRunningTime="2026-10-30 20:00:02.287818243 +0000 UTC m=+1721.983026201" Oct 30 20:00:02.310375 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.310328 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" podStartSLOduration=2.310315454 podStartE2EDuration="2.310315454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.308994718 +0000 UTC m=+1722.004202677" watchObservedRunningTime="2026-10-30 20:00:02.310315454 +0000 UTC m=+1722.005523437" Oct 30 20:00:02.335084 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.335041 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" podStartSLOduration=2.335029028 podStartE2EDuration="2.335029028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.331801859 +0000 UTC m=+1722.027009821" watchObservedRunningTime="2026-10-30 20:00:02.335029028 +0000 UTC m=+1722.030236980" Oct 30 20:00:02.377726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.377668 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" podStartSLOduration=2.377650358 podStartE2EDuration="2.377650358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.355294959 +0000 UTC m=+1722.050502918" watchObservedRunningTime="2026-10-30 20:00:02.377650358 +0000 UTC m=+1722.072858316" Oct 30 20:00:02.388472 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.388414 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" podStartSLOduration=2.388396532 podStartE2EDuration="2.388396532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.379164173 +0000 UTC m=+1722.074372129" watchObservedRunningTime="2026-10-30 20:00:02.388396532 +0000 UTC m=+1722.083604491" Oct 30 20:00:02.403546 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.403505 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" podStartSLOduration=2.403489676 podStartE2EDuration="2.403489676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.401303482 +0000 UTC m=+1722.096511440" watchObservedRunningTime="2026-10-30 20:00:02.403489676 +0000 UTC m=+1722.098697638" Oct 30 20:00:02.434868 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.434822 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" podStartSLOduration=2.434807497 podStartE2EDuration="2.434807497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:02.428370709 +0000 UTC m=+1722.123578666" watchObservedRunningTime="2026-10-30 20:00:02.434807497 +0000 UTC m=+1722.130015473" Oct 30 20:00:02.457623 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.457590 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq"] Oct 30 20:00:02.468707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.468674 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-8fdea8/noten-model-8fdea8-kserve-64b8b59db9-hs7sq"] Oct 30 20:00:02.614190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:02.612695 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:03.282261 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.282223 5091 generic.go:356] "Generic (PLEG): container finished" podID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" containerID="c9b4f2ed2fda859307691bba7c510557d091ce20f806478ed8e2e731506dc68e" exitCode=60 Oct 30 20:00:03.282737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.282314 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" event={"ID":"b6f62ee1-1e91-4ed1-a82b-c5371d65af02","Type":"ContainerDied","Data":"c9b4f2ed2fda859307691bba7c510557d091ce20f806478ed8e2e731506dc68e"} Oct 30 20:00:03.282737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.282678 5091 scope.go:122] "RemoveContainer" containerID="c9b4f2ed2fda859307691bba7c510557d091ce20f806478ed8e2e731506dc68e" Oct 30 20:00:03.284501 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.284472 5091 generic.go:356] "Generic (PLEG): container finished" podID="65a64db5-dcf8-436d-b44a-3a4891da6bde" containerID="f334939ccf66eee8b857e23ddfcc7f3cbe8c9213d5a5dd2be59c4996f041188e" exitCode=60 Oct 30 20:00:03.284678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.284507 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" event={"ID":"65a64db5-dcf8-436d-b44a-3a4891da6bde","Type":"ContainerDied","Data":"f334939ccf66eee8b857e23ddfcc7f3cbe8c9213d5a5dd2be59c4996f041188e"} Oct 30 20:00:03.284851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.284827 5091 scope.go:122] "RemoveContainer" containerID="f334939ccf66eee8b857e23ddfcc7f3cbe8c9213d5a5dd2be59c4996f041188e" Oct 30 20:00:03.286877 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.286800 5091 generic.go:356] "Generic (PLEG): container finished" podID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" containerID="2aee71f1baf25a3b49187378fdb676302a0d31cc0af4021123c2364dc48614a4" exitCode=60 Oct 30 20:00:03.286930 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.286898 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" event={"ID":"005e2b9f-1283-48d3-9b3d-ddd316161e8c","Type":"ContainerDied","Data":"2aee71f1baf25a3b49187378fdb676302a0d31cc0af4021123c2364dc48614a4"} Oct 30 20:00:03.287315 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.287294 5091 scope.go:122] "RemoveContainer" containerID="2aee71f1baf25a3b49187378fdb676302a0d31cc0af4021123c2364dc48614a4" Oct 30 20:00:03.290469 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.290447 5091 generic.go:356] "Generic (PLEG): container finished" podID="96e701fd-f225-4347-8d3c-5be59458f0fb" containerID="f02651958d5544fdf41b4f2bd6186eff14c17796c27dfea726fbafeccf650a7b" exitCode=60 Oct 30 20:00:03.290595 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.290526 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" event={"ID":"96e701fd-f225-4347-8d3c-5be59458f0fb","Type":"ContainerDied","Data":"f02651958d5544fdf41b4f2bd6186eff14c17796c27dfea726fbafeccf650a7b"} Oct 30 20:00:03.290826 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.290807 5091 scope.go:122] "RemoveContainer" containerID="f02651958d5544fdf41b4f2bd6186eff14c17796c27dfea726fbafeccf650a7b" Oct 30 20:00:03.292568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.292546 5091 generic.go:356] "Generic (PLEG): container finished" podID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" containerID="9391ea7b490f948b3ceab574529d62d0df06f25e07b3aefaac96b972cf18a857" exitCode=60 Oct 30 20:00:03.292690 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.292649 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" event={"ID":"0f3161e3-ac92-40a9-bd2f-18384362eeb8","Type":"ContainerDied","Data":"9391ea7b490f948b3ceab574529d62d0df06f25e07b3aefaac96b972cf18a857"} Oct 30 20:00:03.292875 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.292854 5091 scope.go:122] "RemoveContainer" containerID="9391ea7b490f948b3ceab574529d62d0df06f25e07b3aefaac96b972cf18a857" Oct 30 20:00:03.295595 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.295567 5091 generic.go:356] "Generic (PLEG): container finished" podID="125a6e67-31de-4110-8bb8-cdc0af811e46" containerID="c2d368b5f83d2294dd5e21a35a603c96dd39c796d17f03b9e4e4d03b4a9b2075" exitCode=60 Oct 30 20:00:03.295719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.295702 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" event={"ID":"125a6e67-31de-4110-8bb8-cdc0af811e46","Type":"ContainerDied","Data":"c2d368b5f83d2294dd5e21a35a603c96dd39c796d17f03b9e4e4d03b4a9b2075"} Oct 30 20:00:03.296313 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.296289 5091 scope.go:122] "RemoveContainer" containerID="c2d368b5f83d2294dd5e21a35a603c96dd39c796d17f03b9e4e4d03b4a9b2075" Oct 30 20:00:03.319459 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.319432 5091 generic.go:356] "Generic (PLEG): container finished" podID="254c3d23-0de3-46c3-bb52-92502e789102" containerID="d7d2c878874448a6af6bc9d3b9ca56f5d6d009d06662a827003df6e2fb4c0f47" exitCode=60 Oct 30 20:00:03.319562 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.319516 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" event={"ID":"254c3d23-0de3-46c3-bb52-92502e789102","Type":"ContainerDied","Data":"d7d2c878874448a6af6bc9d3b9ca56f5d6d009d06662a827003df6e2fb4c0f47"} Oct 30 20:00:03.319866 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.319847 5091 scope.go:122] "RemoveContainer" containerID="d7d2c878874448a6af6bc9d3b9ca56f5d6d009d06662a827003df6e2fb4c0f47" Oct 30 20:00:03.323888 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.323862 5091 generic.go:356] "Generic (PLEG): container finished" podID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" containerID="9e2c90e60e60cd4a73b63c83bcf6ae207e107c54d9a9d8884ac46b69aa68adeb" exitCode=60 Oct 30 20:00:03.323976 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.323941 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" event={"ID":"7ca322c7-6812-4c40-af1b-62a23ebb4ffe","Type":"ContainerDied","Data":"9e2c90e60e60cd4a73b63c83bcf6ae207e107c54d9a9d8884ac46b69aa68adeb"} Oct 30 20:00:03.324196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.324131 5091 scope.go:122] "RemoveContainer" containerID="9e2c90e60e60cd4a73b63c83bcf6ae207e107c54d9a9d8884ac46b69aa68adeb" Oct 30 20:00:03.694022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:03.693982 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:04.338425 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:04.338389 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" event={"ID":"96e701fd-f225-4347-8d3c-5be59458f0fb","Type":"ContainerStarted","Data":"0d960c21617c33fde29372afe87cb013543c2668c98459de6b3b5b36503e5a03"} Oct 30 20:00:04.340554 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:04.340526 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" event={"ID":"0f3161e3-ac92-40a9-bd2f-18384362eeb8","Type":"ContainerStarted","Data":"54e98f41770a04f7fdd6841ba87fcbb3768f5670f38460a0285b1a25689bb0e3"} Oct 30 20:00:04.342547 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:04.342525 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" event={"ID":"125a6e67-31de-4110-8bb8-cdc0af811e46","Type":"ContainerStarted","Data":"7576a9d86c409bd486a12e74c1692952d36e7dceca28df12cf034689ad0165d5"} Oct 30 20:00:04.344744 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:04.344722 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" event={"ID":"254c3d23-0de3-46c3-bb52-92502e789102","Type":"ContainerStarted","Data":"7e694f9f03783f2d9181753ee18ae40a303472e93cc0387e5435815194f3a659"} Oct 30 20:00:04.347943 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:04.347302 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" event={"ID":"7ca322c7-6812-4c40-af1b-62a23ebb4ffe","Type":"ContainerStarted","Data":"e39d2821ec416e00180ca92c40bbd45c9bb629657a52461623e8da2bbb210647"} Oct 30 20:00:04.354714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:04.353526 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" event={"ID":"b6f62ee1-1e91-4ed1-a82b-c5371d65af02","Type":"ContainerStarted","Data":"5edffccb15039e7e6a70c7e838aabaf7f6df90d70b414c94f017967f1fb34906"} Oct 30 20:00:04.356937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:04.356913 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" event={"ID":"65a64db5-dcf8-436d-b44a-3a4891da6bde","Type":"ContainerStarted","Data":"c7ba3dc6e068520e37fcbc96fb64c82ec40e61fabe48256109e70980bf5e6a18"} Oct 30 20:00:04.359426 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:04.359381 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" event={"ID":"005e2b9f-1283-48d3-9b3d-ddd316161e8c","Type":"ContainerStarted","Data":"dc3b62e50bb1b0d11612996deb5b6b1a1c9e66a7f72631c15e259fe6d6b94a27"} Oct 30 20:00:04.450552 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:04.446078 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="124d537d-04bc-45bf-866e-d0bf638a8cd2" path="/var/lib/kubelet/pods/124d537d-04bc-45bf-866e-d0bf638a8cd2/volumes" Oct 30 20:00:04.616365 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:04.615736 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:04.753003 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:04.752973 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125a6e67_31de_4110_8bb8_cdc0af811e46.slice/crio-7576a9d86c409bd486a12e74c1692952d36e7dceca28df12cf034689ad0165d5.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:00:05.371501 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.371466 5091 generic.go:356] "Generic (PLEG): container finished" podID="65a64db5-dcf8-436d-b44a-3a4891da6bde" containerID="c7ba3dc6e068520e37fcbc96fb64c82ec40e61fabe48256109e70980bf5e6a18" exitCode=60 Oct 30 20:00:05.372003 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.371553 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" event={"ID":"65a64db5-dcf8-436d-b44a-3a4891da6bde","Type":"ContainerDied","Data":"c7ba3dc6e068520e37fcbc96fb64c82ec40e61fabe48256109e70980bf5e6a18"} Oct 30 20:00:05.372003 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.371703 5091 scope.go:122] "RemoveContainer" containerID="f334939ccf66eee8b857e23ddfcc7f3cbe8c9213d5a5dd2be59c4996f041188e" Oct 30 20:00:05.372003 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.371858 5091 scope.go:122] "RemoveContainer" containerID="c7ba3dc6e068520e37fcbc96fb64c82ec40e61fabe48256109e70980bf5e6a18" Oct 30 20:00:05.372258 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:05.372229 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j_odh-ai-gateway-infra(65a64db5-dcf8-436d-b44a-3a4891da6bde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" podUID="65a64db5-dcf8-436d-b44a-3a4891da6bde" Oct 30 20:00:05.373922 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.373901 5091 generic.go:356] "Generic (PLEG): container finished" podID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" containerID="dc3b62e50bb1b0d11612996deb5b6b1a1c9e66a7f72631c15e259fe6d6b94a27" exitCode=60 Oct 30 20:00:05.374029 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.373991 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" event={"ID":"005e2b9f-1283-48d3-9b3d-ddd316161e8c","Type":"ContainerDied","Data":"dc3b62e50bb1b0d11612996deb5b6b1a1c9e66a7f72631c15e259fe6d6b94a27"} Oct 30 20:00:05.374279 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.374264 5091 scope.go:122] "RemoveContainer" containerID="dc3b62e50bb1b0d11612996deb5b6b1a1c9e66a7f72631c15e259fe6d6b94a27" Oct 30 20:00:05.374614 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:05.374593 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-4stn7_odh-ai-gateway-infra(005e2b9f-1283-48d3-9b3d-ddd316161e8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" podUID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" Oct 30 20:00:05.376040 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.376009 5091 generic.go:356] "Generic (PLEG): container finished" podID="96e701fd-f225-4347-8d3c-5be59458f0fb" containerID="0d960c21617c33fde29372afe87cb013543c2668c98459de6b3b5b36503e5a03" exitCode=60 Oct 30 20:00:05.376137 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.376089 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" event={"ID":"96e701fd-f225-4347-8d3c-5be59458f0fb","Type":"ContainerDied","Data":"0d960c21617c33fde29372afe87cb013543c2668c98459de6b3b5b36503e5a03"} Oct 30 20:00:05.376357 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.376341 5091 scope.go:122] "RemoveContainer" containerID="0d960c21617c33fde29372afe87cb013543c2668c98459de6b3b5b36503e5a03" Oct 30 20:00:05.376679 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:05.376652 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds_odh-ai-gateway-infra(96e701fd-f225-4347-8d3c-5be59458f0fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" podUID="96e701fd-f225-4347-8d3c-5be59458f0fb" Oct 30 20:00:05.378400 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.378379 5091 generic.go:356] "Generic (PLEG): container finished" podID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" containerID="54e98f41770a04f7fdd6841ba87fcbb3768f5670f38460a0285b1a25689bb0e3" exitCode=60 Oct 30 20:00:05.378511 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.378459 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" event={"ID":"0f3161e3-ac92-40a9-bd2f-18384362eeb8","Type":"ContainerDied","Data":"54e98f41770a04f7fdd6841ba87fcbb3768f5670f38460a0285b1a25689bb0e3"} Oct 30 20:00:05.378689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.378671 5091 scope.go:122] "RemoveContainer" containerID="54e98f41770a04f7fdd6841ba87fcbb3768f5670f38460a0285b1a25689bb0e3" Oct 30 20:00:05.378954 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:05.378932 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr_odh-ai-gateway-infra(0f3161e3-ac92-40a9-bd2f-18384362eeb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" podUID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" Oct 30 20:00:05.380529 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.380502 5091 generic.go:356] "Generic (PLEG): container finished" podID="125a6e67-31de-4110-8bb8-cdc0af811e46" containerID="7576a9d86c409bd486a12e74c1692952d36e7dceca28df12cf034689ad0165d5" exitCode=60 Oct 30 20:00:05.380625 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.380604 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" event={"ID":"125a6e67-31de-4110-8bb8-cdc0af811e46","Type":"ContainerDied","Data":"7576a9d86c409bd486a12e74c1692952d36e7dceca28df12cf034689ad0165d5"} Oct 30 20:00:05.380779 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.380762 5091 scope.go:122] "RemoveContainer" containerID="7576a9d86c409bd486a12e74c1692952d36e7dceca28df12cf034689ad0165d5" Oct 30 20:00:05.380990 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:05.380971 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff_odh-ai-gateway-infra(125a6e67-31de-4110-8bb8-cdc0af811e46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" podUID="125a6e67-31de-4110-8bb8-cdc0af811e46" Oct 30 20:00:05.382712 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.382691 5091 generic.go:356] "Generic (PLEG): container finished" podID="254c3d23-0de3-46c3-bb52-92502e789102" containerID="7e694f9f03783f2d9181753ee18ae40a303472e93cc0387e5435815194f3a659" exitCode=60 Oct 30 20:00:05.382799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.382770 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" event={"ID":"254c3d23-0de3-46c3-bb52-92502e789102","Type":"ContainerDied","Data":"7e694f9f03783f2d9181753ee18ae40a303472e93cc0387e5435815194f3a659"} Oct 30 20:00:05.382993 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.382972 5091 scope.go:122] "RemoveContainer" containerID="7e694f9f03783f2d9181753ee18ae40a303472e93cc0387e5435815194f3a659" Oct 30 20:00:05.383367 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:05.383334 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz_odh-ai-gateway-infra(254c3d23-0de3-46c3-bb52-92502e789102)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" podUID="254c3d23-0de3-46c3-bb52-92502e789102" Oct 30 20:00:05.384864 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.384837 5091 generic.go:356] "Generic (PLEG): container finished" podID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" containerID="e39d2821ec416e00180ca92c40bbd45c9bb629657a52461623e8da2bbb210647" exitCode=60 Oct 30 20:00:05.384995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.384916 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" event={"ID":"7ca322c7-6812-4c40-af1b-62a23ebb4ffe","Type":"ContainerDied","Data":"e39d2821ec416e00180ca92c40bbd45c9bb629657a52461623e8da2bbb210647"} Oct 30 20:00:05.385101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.385087 5091 scope.go:122] "RemoveContainer" containerID="e39d2821ec416e00180ca92c40bbd45c9bb629657a52461623e8da2bbb210647" Oct 30 20:00:05.385404 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:05.385379 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r_odh-ai-gateway-infra(7ca322c7-6812-4c40-af1b-62a23ebb4ffe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" podUID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" Oct 30 20:00:05.387275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.387254 5091 generic.go:356] "Generic (PLEG): container finished" podID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" containerID="5edffccb15039e7e6a70c7e838aabaf7f6df90d70b414c94f017967f1fb34906" exitCode=60 Oct 30 20:00:05.387373 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.387318 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" event={"ID":"b6f62ee1-1e91-4ed1-a82b-c5371d65af02","Type":"ContainerDied","Data":"5edffccb15039e7e6a70c7e838aabaf7f6df90d70b414c94f017967f1fb34906"} Oct 30 20:00:05.387510 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.387499 5091 scope.go:122] "RemoveContainer" containerID="5edffccb15039e7e6a70c7e838aabaf7f6df90d70b414c94f017967f1fb34906" Oct 30 20:00:05.387693 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:05.387676 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m_odh-ai-gateway-infra(b6f62ee1-1e91-4ed1-a82b-c5371d65af02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" podUID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" Oct 30 20:00:05.388288 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.388270 5091 scope.go:122] "RemoveContainer" containerID="2aee71f1baf25a3b49187378fdb676302a0d31cc0af4021123c2364dc48614a4" Oct 30 20:00:05.408608 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.408585 5091 scope.go:122] "RemoveContainer" containerID="f02651958d5544fdf41b4f2bd6186eff14c17796c27dfea726fbafeccf650a7b" Oct 30 20:00:05.423231 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.423201 5091 scope.go:122] "RemoveContainer" containerID="9391ea7b490f948b3ceab574529d62d0df06f25e07b3aefaac96b972cf18a857" Oct 30 20:00:05.438292 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.438267 5091 scope.go:122] "RemoveContainer" containerID="c2d368b5f83d2294dd5e21a35a603c96dd39c796d17f03b9e4e4d03b4a9b2075" Oct 30 20:00:05.454383 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.454364 5091 scope.go:122] "RemoveContainer" containerID="d7d2c878874448a6af6bc9d3b9ca56f5d6d009d06662a827003df6e2fb4c0f47" Oct 30 20:00:05.469722 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.469695 5091 scope.go:122] "RemoveContainer" containerID="9e2c90e60e60cd4a73b63c83bcf6ae207e107c54d9a9d8884ac46b69aa68adeb" Oct 30 20:00:05.483821 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.483801 5091 scope.go:122] "RemoveContainer" containerID="c9b4f2ed2fda859307691bba7c510557d091ce20f806478ed8e2e731506dc68e" Oct 30 20:00:05.613643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:05.613612 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:06.397834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:06.397804 5091 scope.go:122] "RemoveContainer" containerID="0d960c21617c33fde29372afe87cb013543c2668c98459de6b3b5b36503e5a03" Oct 30 20:00:06.398327 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:06.398258 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds_odh-ai-gateway-infra(96e701fd-f225-4347-8d3c-5be59458f0fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" podUID="96e701fd-f225-4347-8d3c-5be59458f0fb" Oct 30 20:00:06.399739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:06.399707 5091 scope.go:122] "RemoveContainer" containerID="54e98f41770a04f7fdd6841ba87fcbb3768f5670f38460a0285b1a25689bb0e3" Oct 30 20:00:06.400084 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:06.400057 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr_odh-ai-gateway-infra(0f3161e3-ac92-40a9-bd2f-18384362eeb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" podUID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" Oct 30 20:00:06.401334 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:06.401308 5091 scope.go:122] "RemoveContainer" containerID="7576a9d86c409bd486a12e74c1692952d36e7dceca28df12cf034689ad0165d5" Oct 30 20:00:06.401652 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:06.401631 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff_odh-ai-gateway-infra(125a6e67-31de-4110-8bb8-cdc0af811e46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" podUID="125a6e67-31de-4110-8bb8-cdc0af811e46" Oct 30 20:00:06.403306 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:06.403284 5091 scope.go:122] "RemoveContainer" containerID="7e694f9f03783f2d9181753ee18ae40a303472e93cc0387e5435815194f3a659" Oct 30 20:00:06.403587 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:06.403567 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz_odh-ai-gateway-infra(254c3d23-0de3-46c3-bb52-92502e789102)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" podUID="254c3d23-0de3-46c3-bb52-92502e789102" Oct 30 20:00:06.405011 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:06.404993 5091 scope.go:122] "RemoveContainer" containerID="e39d2821ec416e00180ca92c40bbd45c9bb629657a52461623e8da2bbb210647" Oct 30 20:00:06.405247 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:06.405228 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r_odh-ai-gateway-infra(7ca322c7-6812-4c40-af1b-62a23ebb4ffe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" podUID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" Oct 30 20:00:06.406750 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:06.406731 5091 scope.go:122] "RemoveContainer" containerID="5edffccb15039e7e6a70c7e838aabaf7f6df90d70b414c94f017967f1fb34906" Oct 30 20:00:06.406985 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:06.406964 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m_odh-ai-gateway-infra(b6f62ee1-1e91-4ed1-a82b-c5371d65af02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" podUID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" Oct 30 20:00:06.408488 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:06.408470 5091 scope.go:122] "RemoveContainer" containerID="c7ba3dc6e068520e37fcbc96fb64c82ec40e61fabe48256109e70980bf5e6a18" Oct 30 20:00:06.408705 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:06.408684 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j_odh-ai-gateway-infra(65a64db5-dcf8-436d-b44a-3a4891da6bde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" podUID="65a64db5-dcf8-436d-b44a-3a4891da6bde" Oct 30 20:00:06.410066 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:06.410048 5091 scope.go:122] "RemoveContainer" containerID="dc3b62e50bb1b0d11612996deb5b6b1a1c9e66a7f72631c15e259fe6d6b94a27" Oct 30 20:00:06.410272 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:06.410256 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-4stn7_odh-ai-gateway-infra(005e2b9f-1283-48d3-9b3d-ddd316161e8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" podUID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" Oct 30 20:00:06.619720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:06.619683 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:07.688609 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:07.688567 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:08.048995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:08.048956 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76"] Oct 30 20:00:08.055247 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:08.055216 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bd469540-28p76"] Oct 30 20:00:08.443934 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:08.443847 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="359bcdc9-3351-4285-a1a9-9c45c75d542d" path="/var/lib/kubelet/pods/359bcdc9-3351-4285-a1a9-9c45c75d542d/volumes" Oct 30 20:00:08.586166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:08.585847 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:09.435167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:09.428643 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:10.288169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:10.278570 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:11.206176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:11.200958 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:12.115167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:12.110421 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:13.076529 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:13.076488 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:13.090126 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:13.089229 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv"] Oct 30 20:00:13.093802 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:13.093773 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6257ee75-vgkvv"] Oct 30 20:00:13.343196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:13.343077 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:14.094912 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:14.094864 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:14.443405 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:14.443333 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9017622-5b22-40d9-9eb3-052cc0dbb14f" path="/var/lib/kubelet/pods/d9017622-5b22-40d9-9eb3-052cc0dbb14f/volumes" Oct 30 20:00:14.992172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:14.986399 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:15.434339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:15.434307 5091 scope.go:122] "RemoveContainer" containerID="c7ba3dc6e068520e37fcbc96fb64c82ec40e61fabe48256109e70980bf5e6a18" Oct 30 20:00:15.434817 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:15.434423 5091 scope.go:122] "RemoveContainer" containerID="0d960c21617c33fde29372afe87cb013543c2668c98459de6b3b5b36503e5a03" Oct 30 20:00:15.434817 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:15.434521 5091 scope.go:122] "RemoveContainer" containerID="e39d2821ec416e00180ca92c40bbd45c9bb629657a52461623e8da2bbb210647" Oct 30 20:00:15.936173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:15.929769 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:16.003368 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:16.003312 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:00:16.434116 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:16.434031 5091 scope.go:122] "RemoveContainer" containerID="7e694f9f03783f2d9181753ee18ae40a303472e93cc0387e5435815194f3a659" Oct 30 20:00:16.434308 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:16.434162 5091 scope.go:122] "RemoveContainer" containerID="54e98f41770a04f7fdd6841ba87fcbb3768f5670f38460a0285b1a25689bb0e3" Oct 30 20:00:16.554644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:16.554605 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" event={"ID":"96e701fd-f225-4347-8d3c-5be59458f0fb","Type":"ContainerStarted","Data":"5a0f92682a9641fa6b729191ac6b91cbc284e4af79055c5492ee41d3e5592004"} Oct 30 20:00:16.556631 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:16.556588 5091 generic.go:356] "Generic (PLEG): container finished" podID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" containerID="85043189c094ddc24da7dc32ed4ef6603197b6344dc49fc2d136bbd7b0e28265" exitCode=60 Oct 30 20:00:16.556798 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:16.556770 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" event={"ID":"7ca322c7-6812-4c40-af1b-62a23ebb4ffe","Type":"ContainerDied","Data":"85043189c094ddc24da7dc32ed4ef6603197b6344dc49fc2d136bbd7b0e28265"} Oct 30 20:00:16.556932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:16.556823 5091 scope.go:122] "RemoveContainer" containerID="e39d2821ec416e00180ca92c40bbd45c9bb629657a52461623e8da2bbb210647" Oct 30 20:00:16.557257 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:16.557237 5091 scope.go:122] "RemoveContainer" containerID="85043189c094ddc24da7dc32ed4ef6603197b6344dc49fc2d136bbd7b0e28265" Oct 30 20:00:16.557538 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:16.557512 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r_odh-ai-gateway-infra(7ca322c7-6812-4c40-af1b-62a23ebb4ffe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" podUID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" Oct 30 20:00:16.559279 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:16.559254 5091 generic.go:356] "Generic (PLEG): container finished" podID="65a64db5-dcf8-436d-b44a-3a4891da6bde" containerID="29f9a64e0a781982079ebbb41b058034a55e7127af54b63fa338e257b2b93087" exitCode=60 Oct 30 20:00:16.559405 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:16.559333 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" event={"ID":"65a64db5-dcf8-436d-b44a-3a4891da6bde","Type":"ContainerDied","Data":"29f9a64e0a781982079ebbb41b058034a55e7127af54b63fa338e257b2b93087"} Oct 30 20:00:16.559712 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:16.559694 5091 scope.go:122] "RemoveContainer" containerID="29f9a64e0a781982079ebbb41b058034a55e7127af54b63fa338e257b2b93087" Oct 30 20:00:16.559995 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:16.559967 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j_odh-ai-gateway-infra(65a64db5-dcf8-436d-b44a-3a4891da6bde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" podUID="65a64db5-dcf8-436d-b44a-3a4891da6bde" Oct 30 20:00:16.588190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:16.588124 5091 scope.go:122] "RemoveContainer" containerID="c7ba3dc6e068520e37fcbc96fb64c82ec40e61fabe48256109e70980bf5e6a18" Oct 30 20:00:16.756170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:16.753118 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:17.433493 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.433418 5091 scope.go:122] "RemoveContainer" containerID="dc3b62e50bb1b0d11612996deb5b6b1a1c9e66a7f72631c15e259fe6d6b94a27" Oct 30 20:00:17.572783 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.572748 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" event={"ID":"254c3d23-0de3-46c3-bb52-92502e789102","Type":"ContainerStarted","Data":"864c372fc47e976fca37b7c64772e62038ad36ca12a7aee77a9a7e3488b6945c"} Oct 30 20:00:17.579735 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.579705 5091 generic.go:356] "Generic (PLEG): container finished" podID="96e701fd-f225-4347-8d3c-5be59458f0fb" containerID="5a0f92682a9641fa6b729191ac6b91cbc284e4af79055c5492ee41d3e5592004" exitCode=60 Oct 30 20:00:17.579867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.579779 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" event={"ID":"96e701fd-f225-4347-8d3c-5be59458f0fb","Type":"ContainerDied","Data":"5a0f92682a9641fa6b729191ac6b91cbc284e4af79055c5492ee41d3e5592004"} Oct 30 20:00:17.579867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.579844 5091 scope.go:122] "RemoveContainer" containerID="0d960c21617c33fde29372afe87cb013543c2668c98459de6b3b5b36503e5a03" Oct 30 20:00:17.580218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.580189 5091 scope.go:122] "RemoveContainer" containerID="5a0f92682a9641fa6b729191ac6b91cbc284e4af79055c5492ee41d3e5592004" Oct 30 20:00:17.580505 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:17.580484 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds_odh-ai-gateway-infra(96e701fd-f225-4347-8d3c-5be59458f0fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" podUID="96e701fd-f225-4347-8d3c-5be59458f0fb" Oct 30 20:00:17.582097 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.582075 5091 generic.go:356] "Generic (PLEG): container finished" podID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" containerID="33432fe1627c42bf50f75309ab356446cda223c4430f1cc072361e6e751cd2f7" exitCode=60 Oct 30 20:00:17.582229 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.582109 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" event={"ID":"0f3161e3-ac92-40a9-bd2f-18384362eeb8","Type":"ContainerDied","Data":"33432fe1627c42bf50f75309ab356446cda223c4430f1cc072361e6e751cd2f7"} Oct 30 20:00:17.582453 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.582438 5091 scope.go:122] "RemoveContainer" containerID="33432fe1627c42bf50f75309ab356446cda223c4430f1cc072361e6e751cd2f7" Oct 30 20:00:17.582650 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:17.582633 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr_odh-ai-gateway-infra(0f3161e3-ac92-40a9-bd2f-18384362eeb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" podUID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" Oct 30 20:00:17.596897 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.596876 5091 scope.go:122] "RemoveContainer" containerID="54e98f41770a04f7fdd6841ba87fcbb3768f5670f38460a0285b1a25689bb0e3" Oct 30 20:00:17.608639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.608616 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds"] Oct 30 20:00:17.620773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.620747 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j"] Oct 30 20:00:17.665078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.664969 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r"] Oct 30 20:00:17.779706 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.764254 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:17.921964 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.921942 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" Oct 30 20:00:17.996704 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.996663 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a64db5-dcf8-436d-b44a-3a4891da6bde-kube-api-access-28f4p\" (UniqueName: \"kubernetes.io/projected/65a64db5-dcf8-436d-b44a-3a4891da6bde-kube-api-access-28f4p\") pod \"65a64db5-dcf8-436d-b44a-3a4891da6bde\" (UID: \"65a64db5-dcf8-436d-b44a-3a4891da6bde\") " Oct 30 20:00:17.996943 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.996924 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65a64db5-dcf8-436d-b44a-3a4891da6bde-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a64db5-dcf8-436d-b44a-3a4891da6bde-maas-api-service-ca\") pod \"65a64db5-dcf8-436d-b44a-3a4891da6bde\" (UID: \"65a64db5-dcf8-436d-b44a-3a4891da6bde\") " Oct 30 20:00:17.997852 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:17.997825 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a64db5-dcf8-436d-b44a-3a4891da6bde-maas-api-service-ca" pod "65a64db5-dcf8-436d-b44a-3a4891da6bde" (UID: "65a64db5-dcf8-436d-b44a-3a4891da6bde"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:18.000567 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.000539 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a64db5-dcf8-436d-b44a-3a4891da6bde-kube-api-access-28f4p" pod "65a64db5-dcf8-436d-b44a-3a4891da6bde" (UID: "65a64db5-dcf8-436d-b44a-3a4891da6bde"). InnerVolumeSpecName "kube-api-access-28f4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:18.028394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.028374 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" Oct 30 20:00:18.098501 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.098470 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca322c7-6812-4c40-af1b-62a23ebb4ffe-kube-api-access-wd8wm\" (UniqueName: \"kubernetes.io/projected/7ca322c7-6812-4c40-af1b-62a23ebb4ffe-kube-api-access-wd8wm\") pod \"7ca322c7-6812-4c40-af1b-62a23ebb4ffe\" (UID: \"7ca322c7-6812-4c40-af1b-62a23ebb4ffe\") " Oct 30 20:00:18.098649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.098575 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca322c7-6812-4c40-af1b-62a23ebb4ffe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca322c7-6812-4c40-af1b-62a23ebb4ffe-maas-api-service-ca\") pod \"7ca322c7-6812-4c40-af1b-62a23ebb4ffe\" (UID: \"7ca322c7-6812-4c40-af1b-62a23ebb4ffe\") " Oct 30 20:00:18.098941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.098918 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca322c7-6812-4c40-af1b-62a23ebb4ffe-maas-api-service-ca" pod "7ca322c7-6812-4c40-af1b-62a23ebb4ffe" (UID: "7ca322c7-6812-4c40-af1b-62a23ebb4ffe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:18.099136 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.099117 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/65a64db5-dcf8-436d-b44a-3a4891da6bde-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.099248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.099141 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28f4p\" (UniqueName: \"kubernetes.io/projected/65a64db5-dcf8-436d-b44a-3a4891da6bde-kube-api-access-28f4p\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.099248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.099172 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ca322c7-6812-4c40-af1b-62a23ebb4ffe-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.101116 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.101088 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca322c7-6812-4c40-af1b-62a23ebb4ffe-kube-api-access-wd8wm" pod "7ca322c7-6812-4c40-af1b-62a23ebb4ffe" (UID: "7ca322c7-6812-4c40-af1b-62a23ebb4ffe"). InnerVolumeSpecName "kube-api-access-wd8wm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:18.200709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.200677 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wd8wm\" (UniqueName: \"kubernetes.io/projected/7ca322c7-6812-4c40-af1b-62a23ebb4ffe-kube-api-access-wd8wm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:18.434284 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.434211 5091 scope.go:122] "RemoveContainer" containerID="7576a9d86c409bd486a12e74c1692952d36e7dceca28df12cf034689ad0165d5" Oct 30 20:00:18.434462 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.434369 5091 scope.go:122] "RemoveContainer" containerID="5edffccb15039e7e6a70c7e838aabaf7f6df90d70b414c94f017967f1fb34906" Oct 30 20:00:18.593999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.593968 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" Oct 30 20:00:18.594506 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.594007 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j" event={"ID":"65a64db5-dcf8-436d-b44a-3a4891da6bde","Type":"ContainerDied","Data":"a2187ad519dd029f3fb3ec56241fa0c31db61b26084a211498f74e81ba26e9f9"} Oct 30 20:00:18.594506 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.594086 5091 scope.go:122] "RemoveContainer" containerID="29f9a64e0a781982079ebbb41b058034a55e7127af54b63fa338e257b2b93087" Oct 30 20:00:18.596562 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.596539 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" event={"ID":"005e2b9f-1283-48d3-9b3d-ddd316161e8c","Type":"ContainerStarted","Data":"177cfc475f9f2be2715b35fbc455943eebd054389f1d429d188efa9865833cc2"} Oct 30 20:00:18.608361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.608332 5091 generic.go:356] "Generic (PLEG): container finished" podID="254c3d23-0de3-46c3-bb52-92502e789102" containerID="864c372fc47e976fca37b7c64772e62038ad36ca12a7aee77a9a7e3488b6945c" exitCode=60 Oct 30 20:00:18.608484 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.608436 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" event={"ID":"254c3d23-0de3-46c3-bb52-92502e789102","Type":"ContainerDied","Data":"864c372fc47e976fca37b7c64772e62038ad36ca12a7aee77a9a7e3488b6945c"} Oct 30 20:00:18.608936 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.608915 5091 scope.go:122] "RemoveContainer" containerID="864c372fc47e976fca37b7c64772e62038ad36ca12a7aee77a9a7e3488b6945c" Oct 30 20:00:18.609123 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.609099 5091 scope.go:122] "RemoveContainer" containerID="7e694f9f03783f2d9181753ee18ae40a303472e93cc0387e5435815194f3a659" Oct 30 20:00:18.609419 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:18.609391 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz_odh-ai-gateway-infra(254c3d23-0de3-46c3-bb52-92502e789102)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" podUID="254c3d23-0de3-46c3-bb52-92502e789102" Oct 30 20:00:18.611336 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.611308 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" event={"ID":"7ca322c7-6812-4c40-af1b-62a23ebb4ffe","Type":"ContainerDied","Data":"0a6679aae2688d1ef68e488971131f3f96ecd096ce5e9195d5f511028e3b3f37"} Oct 30 20:00:18.611464 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.611449 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r" Oct 30 20:00:18.628637 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.628615 5091 scope.go:122] "RemoveContainer" containerID="85043189c094ddc24da7dc32ed4ef6603197b6344dc49fc2d136bbd7b0e28265" Oct 30 20:00:18.635783 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.635760 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz"] Oct 30 20:00:18.669898 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.669870 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr"] Oct 30 20:00:18.695592 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.695502 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j"] Oct 30 20:00:18.706215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.706179 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889840-jc24j"] Oct 30 20:00:18.757681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.757516 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r"] Oct 30 20:00:18.772777 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.770691 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889840-tkh4r"] Oct 30 20:00:18.823527 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.823323 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:18.926187 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:18.926110 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" Oct 30 20:00:19.027401 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.027369 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96e701fd-f225-4347-8d3c-5be59458f0fb-kube-api-access-m6gsq\" (UniqueName: \"kubernetes.io/projected/96e701fd-f225-4347-8d3c-5be59458f0fb-kube-api-access-m6gsq\") pod \"96e701fd-f225-4347-8d3c-5be59458f0fb\" (UID: \"96e701fd-f225-4347-8d3c-5be59458f0fb\") " Oct 30 20:00:19.027572 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.027418 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96e701fd-f225-4347-8d3c-5be59458f0fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e701fd-f225-4347-8d3c-5be59458f0fb-maas-api-service-ca\") pod \"96e701fd-f225-4347-8d3c-5be59458f0fb\" (UID: \"96e701fd-f225-4347-8d3c-5be59458f0fb\") " Oct 30 20:00:19.028090 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.028061 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e701fd-f225-4347-8d3c-5be59458f0fb-maas-api-service-ca" pod "96e701fd-f225-4347-8d3c-5be59458f0fb" (UID: "96e701fd-f225-4347-8d3c-5be59458f0fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:19.028293 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.028271 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96e701fd-f225-4347-8d3c-5be59458f0fb-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:19.030538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.030508 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e701fd-f225-4347-8d3c-5be59458f0fb-kube-api-access-m6gsq" pod "96e701fd-f225-4347-8d3c-5be59458f0fb" (UID: "96e701fd-f225-4347-8d3c-5be59458f0fb"). InnerVolumeSpecName "kube-api-access-m6gsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:19.065300 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.065278 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" Oct 30 20:00:19.129570 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.129536 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3161e3-ac92-40a9-bd2f-18384362eeb8-kube-api-access-p22lp\" (UniqueName: \"kubernetes.io/projected/0f3161e3-ac92-40a9-bd2f-18384362eeb8-kube-api-access-p22lp\") pod \"0f3161e3-ac92-40a9-bd2f-18384362eeb8\" (UID: \"0f3161e3-ac92-40a9-bd2f-18384362eeb8\") " Oct 30 20:00:19.129744 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.129719 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f3161e3-ac92-40a9-bd2f-18384362eeb8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3161e3-ac92-40a9-bd2f-18384362eeb8-maas-api-service-ca\") pod \"0f3161e3-ac92-40a9-bd2f-18384362eeb8\" (UID: \"0f3161e3-ac92-40a9-bd2f-18384362eeb8\") " Oct 30 20:00:19.130748 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.130718 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6gsq\" (UniqueName: \"kubernetes.io/projected/96e701fd-f225-4347-8d3c-5be59458f0fb-kube-api-access-m6gsq\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:19.131133 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.131102 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f3161e3-ac92-40a9-bd2f-18384362eeb8-maas-api-service-ca" pod "0f3161e3-ac92-40a9-bd2f-18384362eeb8" (UID: "0f3161e3-ac92-40a9-bd2f-18384362eeb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:19.137509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.137478 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3161e3-ac92-40a9-bd2f-18384362eeb8-kube-api-access-p22lp" pod "0f3161e3-ac92-40a9-bd2f-18384362eeb8" (UID: "0f3161e3-ac92-40a9-bd2f-18384362eeb8"). InnerVolumeSpecName "kube-api-access-p22lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:19.231706 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.231634 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p22lp\" (UniqueName: \"kubernetes.io/projected/0f3161e3-ac92-40a9-bd2f-18384362eeb8-kube-api-access-p22lp\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:19.231706 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.231664 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f3161e3-ac92-40a9-bd2f-18384362eeb8-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:19.621945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.621914 5091 generic.go:356] "Generic (PLEG): container finished" podID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" containerID="ebc34c849941d7027a8c5ae9c3de5fb9b6567f65aecbdeb6ba149ed667fc74f3" exitCode=60 Oct 30 20:00:19.622406 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.621986 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" event={"ID":"b6f62ee1-1e91-4ed1-a82b-c5371d65af02","Type":"ContainerDied","Data":"ebc34c849941d7027a8c5ae9c3de5fb9b6567f65aecbdeb6ba149ed667fc74f3"} Oct 30 20:00:19.622406 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.622042 5091 scope.go:122] "RemoveContainer" containerID="5edffccb15039e7e6a70c7e838aabaf7f6df90d70b414c94f017967f1fb34906" Oct 30 20:00:19.622508 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.622409 5091 scope.go:122] "RemoveContainer" containerID="ebc34c849941d7027a8c5ae9c3de5fb9b6567f65aecbdeb6ba149ed667fc74f3" Oct 30 20:00:19.622715 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:19.622692 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m_odh-ai-gateway-infra(b6f62ee1-1e91-4ed1-a82b-c5371d65af02)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" podUID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" Oct 30 20:00:19.625381 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.625359 5091 generic.go:356] "Generic (PLEG): container finished" podID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" containerID="177cfc475f9f2be2715b35fbc455943eebd054389f1d429d188efa9865833cc2" exitCode=60 Oct 30 20:00:19.625487 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.625426 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" event={"ID":"005e2b9f-1283-48d3-9b3d-ddd316161e8c","Type":"ContainerDied","Data":"177cfc475f9f2be2715b35fbc455943eebd054389f1d429d188efa9865833cc2"} Oct 30 20:00:19.625763 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.625748 5091 scope.go:122] "RemoveContainer" containerID="177cfc475f9f2be2715b35fbc455943eebd054389f1d429d188efa9865833cc2" Oct 30 20:00:19.626033 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:19.626003 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889840-4stn7_odh-ai-gateway-infra(005e2b9f-1283-48d3-9b3d-ddd316161e8c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" podUID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" Oct 30 20:00:19.626760 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.626745 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" Oct 30 20:00:19.626856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.626766 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds" event={"ID":"96e701fd-f225-4347-8d3c-5be59458f0fb","Type":"ContainerDied","Data":"fd5b6ab69e72586dae640e76a34595ad4a39c22b497c105c9ccb96aa6509578d"} Oct 30 20:00:19.628642 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.628395 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" Oct 30 20:00:19.628642 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.628468 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr" event={"ID":"0f3161e3-ac92-40a9-bd2f-18384362eeb8","Type":"ContainerDied","Data":"6a746fb5ac769e46b75610dabe5df8384a56dc67ad0103f81e64a6b3172a045d"} Oct 30 20:00:19.635952 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.635927 5091 generic.go:356] "Generic (PLEG): container finished" podID="125a6e67-31de-4110-8bb8-cdc0af811e46" containerID="08a86ff817f36f2db8305f773b0c87c8dc48b160d73752b3cc0cb243d7e5fc5d" exitCode=60 Oct 30 20:00:19.636049 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.636003 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" event={"ID":"125a6e67-31de-4110-8bb8-cdc0af811e46","Type":"ContainerDied","Data":"08a86ff817f36f2db8305f773b0c87c8dc48b160d73752b3cc0cb243d7e5fc5d"} Oct 30 20:00:19.636437 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.636420 5091 scope.go:122] "RemoveContainer" containerID="08a86ff817f36f2db8305f773b0c87c8dc48b160d73752b3cc0cb243d7e5fc5d" Oct 30 20:00:19.636725 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:19.636703 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff_odh-ai-gateway-infra(125a6e67-31de-4110-8bb8-cdc0af811e46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" podUID="125a6e67-31de-4110-8bb8-cdc0af811e46" Oct 30 20:00:19.640757 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.640737 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7"] Oct 30 20:00:19.645563 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.645406 5091 scope.go:122] "RemoveContainer" containerID="dc3b62e50bb1b0d11612996deb5b6b1a1c9e66a7f72631c15e259fe6d6b94a27" Oct 30 20:00:19.669479 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.669456 5091 scope.go:122] "RemoveContainer" containerID="5a0f92682a9641fa6b729191ac6b91cbc284e4af79055c5492ee41d3e5592004" Oct 30 20:00:19.682667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.682644 5091 scope.go:122] "RemoveContainer" containerID="33432fe1627c42bf50f75309ab356446cda223c4430f1cc072361e6e751cd2f7" Oct 30 20:00:19.699409 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.699370 5091 scope.go:122] "RemoveContainer" containerID="7576a9d86c409bd486a12e74c1692952d36e7dceca28df12cf034689ad0165d5" Oct 30 20:00:19.709432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.709373 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr"] Oct 30 20:00:19.713575 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.713550 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889840-vbmnr"] Oct 30 20:00:19.740250 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.740220 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds"] Oct 30 20:00:19.745257 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.745228 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-0eeba5-29889840-dv8ds"] Oct 30 20:00:19.821902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.821875 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" Oct 30 20:00:19.952187 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.950793 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:19.956874 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.956838 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254c3d23-0de3-46c3-bb52-92502e789102-kube-api-access-6t9ss\" (UniqueName: \"kubernetes.io/projected/254c3d23-0de3-46c3-bb52-92502e789102-kube-api-access-6t9ss\") pod \"254c3d23-0de3-46c3-bb52-92502e789102\" (UID: \"254c3d23-0de3-46c3-bb52-92502e789102\") " Oct 30 20:00:19.957081 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.956923 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254c3d23-0de3-46c3-bb52-92502e789102-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254c3d23-0de3-46c3-bb52-92502e789102-maas-api-service-ca\") pod \"254c3d23-0de3-46c3-bb52-92502e789102\" (UID: \"254c3d23-0de3-46c3-bb52-92502e789102\") " Oct 30 20:00:19.958392 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.958266 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254c3d23-0de3-46c3-bb52-92502e789102-maas-api-service-ca" pod "254c3d23-0de3-46c3-bb52-92502e789102" (UID: "254c3d23-0de3-46c3-bb52-92502e789102"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:19.965973 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:19.965945 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254c3d23-0de3-46c3-bb52-92502e789102-kube-api-access-6t9ss" pod "254c3d23-0de3-46c3-bb52-92502e789102" (UID: "254c3d23-0de3-46c3-bb52-92502e789102"). InnerVolumeSpecName "kube-api-access-6t9ss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:20.059374 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:20.059293 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/254c3d23-0de3-46c3-bb52-92502e789102-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.059374 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:20.059323 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t9ss\" (UniqueName: \"kubernetes.io/projected/254c3d23-0de3-46c3-bb52-92502e789102-kube-api-access-6t9ss\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:20.445840 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:20.445762 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" path="/var/lib/kubelet/pods/0f3161e3-ac92-40a9-bd2f-18384362eeb8/volumes" Oct 30 20:00:20.446310 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:20.446295 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65a64db5-dcf8-436d-b44a-3a4891da6bde" path="/var/lib/kubelet/pods/65a64db5-dcf8-436d-b44a-3a4891da6bde/volumes" Oct 30 20:00:20.446715 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:20.446702 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" path="/var/lib/kubelet/pods/7ca322c7-6812-4c40-af1b-62a23ebb4ffe/volumes" Oct 30 20:00:20.447433 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:20.447417 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96e701fd-f225-4347-8d3c-5be59458f0fb" path="/var/lib/kubelet/pods/96e701fd-f225-4347-8d3c-5be59458f0fb/volumes" Oct 30 20:00:21.050197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.047230 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m"] Oct 30 20:00:21.081864 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.081274 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" event={"ID":"254c3d23-0de3-46c3-bb52-92502e789102","Type":"ContainerDied","Data":"52373f0523a530ee18015b730aec2ccec45c80194f98adbd8aac4f301d02a313"} Oct 30 20:00:21.081864 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.081353 5091 scope.go:122] "RemoveContainer" containerID="864c372fc47e976fca37b7c64772e62038ad36ca12a7aee77a9a7e3488b6945c" Oct 30 20:00:21.081864 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.081505 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz" Oct 30 20:00:21.105837 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.103442 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff"] Oct 30 20:00:21.147406 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.147333 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz"] Oct 30 20:00:21.198212 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.188112 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:21.198539 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.198256 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889840-dlqfz"] Oct 30 20:00:21.594372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.594291 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" Oct 30 20:00:21.657604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.657567 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6f62ee1-1e91-4ed1-a82b-c5371d65af02-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f62ee1-1e91-4ed1-a82b-c5371d65af02-maas-api-service-ca\") pod \"b6f62ee1-1e91-4ed1-a82b-c5371d65af02\" (UID: \"b6f62ee1-1e91-4ed1-a82b-c5371d65af02\") " Oct 30 20:00:21.657758 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.657671 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6f62ee1-1e91-4ed1-a82b-c5371d65af02-kube-api-access-5888g\" (UniqueName: \"kubernetes.io/projected/b6f62ee1-1e91-4ed1-a82b-c5371d65af02-kube-api-access-5888g\") pod \"b6f62ee1-1e91-4ed1-a82b-c5371d65af02\" (UID: \"b6f62ee1-1e91-4ed1-a82b-c5371d65af02\") " Oct 30 20:00:21.659126 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.659082 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6f62ee1-1e91-4ed1-a82b-c5371d65af02-maas-api-service-ca" pod "b6f62ee1-1e91-4ed1-a82b-c5371d65af02" (UID: "b6f62ee1-1e91-4ed1-a82b-c5371d65af02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:21.663715 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.663637 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6f62ee1-1e91-4ed1-a82b-c5371d65af02-kube-api-access-5888g" pod "b6f62ee1-1e91-4ed1-a82b-c5371d65af02" (UID: "b6f62ee1-1e91-4ed1-a82b-c5371d65af02"). InnerVolumeSpecName "kube-api-access-5888g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:21.703644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.703617 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" Oct 30 20:00:21.705897 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.705863 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" Oct 30 20:00:21.759301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.759261 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/125a6e67-31de-4110-8bb8-cdc0af811e46-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/125a6e67-31de-4110-8bb8-cdc0af811e46-maas-api-service-ca\") pod \"125a6e67-31de-4110-8bb8-cdc0af811e46\" (UID: \"125a6e67-31de-4110-8bb8-cdc0af811e46\") " Oct 30 20:00:21.759517 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.759380 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/005e2b9f-1283-48d3-9b3d-ddd316161e8c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/005e2b9f-1283-48d3-9b3d-ddd316161e8c-maas-api-service-ca\") pod \"005e2b9f-1283-48d3-9b3d-ddd316161e8c\" (UID: \"005e2b9f-1283-48d3-9b3d-ddd316161e8c\") " Oct 30 20:00:21.759770 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.759743 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125a6e67-31de-4110-8bb8-cdc0af811e46-kube-api-access-s7x89\" (UniqueName: \"kubernetes.io/projected/125a6e67-31de-4110-8bb8-cdc0af811e46-kube-api-access-s7x89\") pod \"125a6e67-31de-4110-8bb8-cdc0af811e46\" (UID: \"125a6e67-31de-4110-8bb8-cdc0af811e46\") " Oct 30 20:00:21.759884 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.759762 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125a6e67-31de-4110-8bb8-cdc0af811e46-maas-api-service-ca" pod "125a6e67-31de-4110-8bb8-cdc0af811e46" (UID: "125a6e67-31de-4110-8bb8-cdc0af811e46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:21.759884 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.759804 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/005e2b9f-1283-48d3-9b3d-ddd316161e8c-kube-api-access-2g2ht\" (UniqueName: \"kubernetes.io/projected/005e2b9f-1283-48d3-9b3d-ddd316161e8c-kube-api-access-2g2ht\") pod \"005e2b9f-1283-48d3-9b3d-ddd316161e8c\" (UID: \"005e2b9f-1283-48d3-9b3d-ddd316161e8c\") " Oct 30 20:00:21.760650 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.760573 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6f62ee1-1e91-4ed1-a82b-c5371d65af02-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:21.760650 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.760600 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5888g\" (UniqueName: \"kubernetes.io/projected/b6f62ee1-1e91-4ed1-a82b-c5371d65af02-kube-api-access-5888g\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:21.760650 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.760624 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/125a6e67-31de-4110-8bb8-cdc0af811e46-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:21.762677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.762644 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125a6e67-31de-4110-8bb8-cdc0af811e46-kube-api-access-s7x89" pod "125a6e67-31de-4110-8bb8-cdc0af811e46" (UID: "125a6e67-31de-4110-8bb8-cdc0af811e46"). InnerVolumeSpecName "kube-api-access-s7x89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:21.762677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.762654 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/005e2b9f-1283-48d3-9b3d-ddd316161e8c-maas-api-service-ca" pod "005e2b9f-1283-48d3-9b3d-ddd316161e8c" (UID: "005e2b9f-1283-48d3-9b3d-ddd316161e8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:21.764774 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.764743 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/005e2b9f-1283-48d3-9b3d-ddd316161e8c-kube-api-access-2g2ht" pod "005e2b9f-1283-48d3-9b3d-ddd316161e8c" (UID: "005e2b9f-1283-48d3-9b3d-ddd316161e8c"). InnerVolumeSpecName "kube-api-access-2g2ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:21.862883 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.862802 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7x89\" (UniqueName: \"kubernetes.io/projected/125a6e67-31de-4110-8bb8-cdc0af811e46-kube-api-access-s7x89\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:21.862883 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.862841 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g2ht\" (UniqueName: \"kubernetes.io/projected/005e2b9f-1283-48d3-9b3d-ddd316161e8c-kube-api-access-2g2ht\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:21.862883 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:21.862857 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/005e2b9f-1283-48d3-9b3d-ddd316161e8c-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:22.109499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.109469 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" Oct 30 20:00:22.109943 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.109507 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m" event={"ID":"b6f62ee1-1e91-4ed1-a82b-c5371d65af02","Type":"ContainerDied","Data":"a437c2a0d0e051a24abe818f5341c2f43c88fa7c73971f8314e1bf42af65208c"} Oct 30 20:00:22.109943 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.109567 5091 scope.go:122] "RemoveContainer" containerID="ebc34c849941d7027a8c5ae9c3de5fb9b6567f65aecbdeb6ba149ed667fc74f3" Oct 30 20:00:22.111252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.111227 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" Oct 30 20:00:22.111252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.111232 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7" event={"ID":"005e2b9f-1283-48d3-9b3d-ddd316161e8c","Type":"ContainerDied","Data":"1c856bd88bd75a71f0ef0dc213db4a2da8dcce798b40b2e09c65f1ffe3b5f9fa"} Oct 30 20:00:22.115795 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.113107 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" Oct 30 20:00:22.115795 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.113187 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff" event={"ID":"125a6e67-31de-4110-8bb8-cdc0af811e46","Type":"ContainerDied","Data":"ca44f22c51d5bb636a4baedab6bc3808e1e94ce7acb93e2a04fc4c55b779efb6"} Oct 30 20:00:22.124862 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.124785 5091 scope.go:122] "RemoveContainer" containerID="177cfc475f9f2be2715b35fbc455943eebd054389f1d429d188efa9865833cc2" Oct 30 20:00:22.135959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.135940 5091 scope.go:122] "RemoveContainer" containerID="08a86ff817f36f2db8305f773b0c87c8dc48b160d73752b3cc0cb243d7e5fc5d" Oct 30 20:00:22.165245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.165134 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m"] Oct 30 20:00:22.171110 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.171080 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7cbcbb-29889840-gvh8m"] Oct 30 20:00:22.179732 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.179698 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7"] Oct 30 20:00:22.185263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.185239 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889840-4stn7"] Oct 30 20:00:22.204026 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.203997 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff"] Oct 30 20:00:22.207846 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.207823 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889840-65xff"] Oct 30 20:00:22.397251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.392183 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:22.444870 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.444832 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" path="/var/lib/kubelet/pods/005e2b9f-1283-48d3-9b3d-ddd316161e8c/volumes" Oct 30 20:00:22.445418 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.445394 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="125a6e67-31de-4110-8bb8-cdc0af811e46" path="/var/lib/kubelet/pods/125a6e67-31de-4110-8bb8-cdc0af811e46/volumes" Oct 30 20:00:22.445886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.445867 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254c3d23-0de3-46c3-bb52-92502e789102" path="/var/lib/kubelet/pods/254c3d23-0de3-46c3-bb52-92502e789102/volumes" Oct 30 20:00:22.446700 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.446679 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" path="/var/lib/kubelet/pods/b6f62ee1-1e91-4ed1-a82b-c5371d65af02/volumes" Oct 30 20:00:22.801283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.801246 5091 scope.go:122] "RemoveContainer" containerID="9cfd0aaab7d69f0d22a7ef768663e3cc4723dff94f7bfc9361c8220daf9fa388" Oct 30 20:00:22.814226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.814203 5091 scope.go:122] "RemoveContainer" containerID="a373363934285a0ee559a6768851e533d4f61bfdde92622be57b03d57ae5e8cc" Oct 30 20:00:22.824583 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.824564 5091 scope.go:122] "RemoveContainer" containerID="8695f41b72b9de9d21ab6105adffc56d91ccda4a08cd29afc67bf457ad39bf9f" Oct 30 20:00:22.836103 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.836082 5091 scope.go:122] "RemoveContainer" containerID="2f6c13daaa37583a71a3a31386ee21021f9047d4b2ffb26a3d71b4e6c6b0f79e" Oct 30 20:00:22.849286 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.849261 5091 scope.go:122] "RemoveContainer" containerID="973a8c1d8a749d965e04ada6fc78914ef387e1cf94301f615a47ee9d5d6b44cb" Oct 30 20:00:22.859474 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.859449 5091 scope.go:122] "RemoveContainer" containerID="73dff09f417534f2d65c40dfae97c863044c2faf50c4374943c9ac9b8f759caf" Oct 30 20:00:22.868284 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.868268 5091 scope.go:122] "RemoveContainer" containerID="cee45743584c77e8a1873446ad66f99f5de40faaca1782e6d4b53e0dce13da23" Oct 30 20:00:22.879491 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:22.879474 5091 scope.go:122] "RemoveContainer" containerID="9b71f996462a070b6c207d1672118d220a7325f45f9d409244761dda777fd1c5" Oct 30 20:00:24.081087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:24.081048 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc"] Oct 30 20:00:24.084986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:24.084958 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-401c9b63-44qzc"] Oct 30 20:00:24.165951 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:24.165902 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:24.443085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:24.442999 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa6a678b-bb04-4755-b0c1-7284663f0455" path="/var/lib/kubelet/pods/aa6a678b-bb04-4755-b0c1-7284663f0455/volumes" Oct 30 20:00:24.716726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:24.716174 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:25.599471 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:25.599436 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:26.509180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:26.505207 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:27.386446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:27.386411 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:28.325681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:28.324996 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:29.331176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:29.320087 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:30.384177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:30.380781 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:31.198689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.198425 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:31.351582 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.351551 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7325-14a184"] Oct 30 20:00:31.352259 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352240 5091 cpu_manager.go:414] "containerMap: removing container" podUID="65a64db5-dcf8-436d-b44a-3a4891da6bde" containerName="cleanup" Oct 30 20:00:31.352259 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352258 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a64db5-dcf8-436d-b44a-3a4891da6bde" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352269 5091 cpu_manager.go:414] "containerMap: removing container" podUID="65a64db5-dcf8-436d-b44a-3a4891da6bde" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352274 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a64db5-dcf8-436d-b44a-3a4891da6bde" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352284 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352289 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352302 5091 cpu_manager.go:414] "containerMap: removing container" podUID="96e701fd-f225-4347-8d3c-5be59458f0fb" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352306 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e701fd-f225-4347-8d3c-5be59458f0fb" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352313 5091 cpu_manager.go:414] "containerMap: removing container" podUID="65a64db5-dcf8-436d-b44a-3a4891da6bde" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352318 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a64db5-dcf8-436d-b44a-3a4891da6bde" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352328 5091 cpu_manager.go:414] "containerMap: removing container" podUID="254c3d23-0de3-46c3-bb52-92502e789102" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352332 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254c3d23-0de3-46c3-bb52-92502e789102" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352338 5091 cpu_manager.go:414] "containerMap: removing container" podUID="96e701fd-f225-4347-8d3c-5be59458f0fb" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352342 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e701fd-f225-4347-8d3c-5be59458f0fb" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352354 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352359 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352370 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352375 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352383 5091 cpu_manager.go:414] "containerMap: removing container" podUID="96e701fd-f225-4347-8d3c-5be59458f0fb" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352387 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96e701fd-f225-4347-8d3c-5be59458f0fb" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352397 5091 cpu_manager.go:414] "containerMap: removing container" podUID="125a6e67-31de-4110-8bb8-cdc0af811e46" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352401 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125a6e67-31de-4110-8bb8-cdc0af811e46" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352409 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352413 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352427 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352431 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352437 5091 cpu_manager.go:414] "containerMap: removing container" podUID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352442 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352447 5091 cpu_manager.go:414] "containerMap: removing container" podUID="125a6e67-31de-4110-8bb8-cdc0af811e46" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352452 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125a6e67-31de-4110-8bb8-cdc0af811e46" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352465 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" containerName="cleanup" Oct 30 20:00:31.352452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352469 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352476 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352480 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352487 5091 cpu_manager.go:414] "containerMap: removing container" podUID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352493 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352503 5091 cpu_manager.go:414] "containerMap: removing container" podUID="254c3d23-0de3-46c3-bb52-92502e789102" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352507 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254c3d23-0de3-46c3-bb52-92502e789102" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352515 5091 cpu_manager.go:414] "containerMap: removing container" podUID="124d537d-04bc-45bf-866e-d0bf638a8cd2" containerName="main" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352519 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="124d537d-04bc-45bf-866e-d0bf638a8cd2" containerName="main" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352529 5091 cpu_manager.go:414] "containerMap: removing container" podUID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352533 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352541 5091 cpu_manager.go:414] "containerMap: removing container" podUID="125a6e67-31de-4110-8bb8-cdc0af811e46" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352545 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125a6e67-31de-4110-8bb8-cdc0af811e46" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352651 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352664 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352670 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e701fd-f225-4347-8d3c-5be59458f0fb" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352678 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="125a6e67-31de-4110-8bb8-cdc0af811e46" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352683 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352691 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352697 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352703 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="125a6e67-31de-4110-8bb8-cdc0af811e46" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352709 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352715 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352719 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352726 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352731 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="254c3d23-0de3-46c3-bb52-92502e789102" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352737 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="254c3d23-0de3-46c3-bb52-92502e789102" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352742 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="124d537d-04bc-45bf-866e-d0bf638a8cd2" containerName="main" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352747 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="005e2b9f-1283-48d3-9b3d-ddd316161e8c" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352752 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352759 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a64db5-dcf8-436d-b44a-3a4891da6bde" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352763 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a64db5-dcf8-436d-b44a-3a4891da6bde" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352771 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="254c3d23-0de3-46c3-bb52-92502e789102" containerName="cleanup" Oct 30 20:00:31.353992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.352777 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e701fd-f225-4347-8d3c-5be59458f0fb" containerName="cleanup" Oct 30 20:00:31.360326 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.360301 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-14a184" Oct 30 20:00:31.367620 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.367591 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-14a184"] Oct 30 20:00:31.370430 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.370409 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dqx9r\"" Oct 30 20:00:31.395371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.395323 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:31.467133 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.467041 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l9wd\" (UniqueName: \"kubernetes.io/projected/7d11e4f0-1e56-416d-8a1d-56329456cb75-kube-api-access-9l9wd\") pod \"test-curl-7325-14a184\" (UID: \"7d11e4f0-1e56-416d-8a1d-56329456cb75\") " pod="opendatahub/test-curl-7325-14a184" Oct 30 20:00:31.569188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.569108 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l9wd\" (UniqueName: \"kubernetes.io/projected/7d11e4f0-1e56-416d-8a1d-56329456cb75-kube-api-access-9l9wd\") pod \"test-curl-7325-14a184\" (UID: \"7d11e4f0-1e56-416d-8a1d-56329456cb75\") " pod="opendatahub/test-curl-7325-14a184" Oct 30 20:00:31.584471 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.584439 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l9wd\" (UniqueName: \"kubernetes.io/projected/7d11e4f0-1e56-416d-8a1d-56329456cb75-kube-api-access-9l9wd\") pod \"test-curl-7325-14a184\" (UID: \"7d11e4f0-1e56-416d-8a1d-56329456cb75\") " pod="opendatahub/test-curl-7325-14a184" Oct 30 20:00:31.685767 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.685734 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-14a184" Oct 30 20:00:31.971276 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:00:31.971234 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7d11e4f0_1e56_416d_8a1d_56329456cb75.slice/crio-76bbd3fd3e3eeb1617f864f47528cf17b4054ca49e56e59f073c367dfb4b7a8e WatchSource:0}: Error finding container 76bbd3fd3e3eeb1617f864f47528cf17b4054ca49e56e59f073c367dfb4b7a8e: Status 404 returned error can't find the container with id 76bbd3fd3e3eeb1617f864f47528cf17b4054ca49e56e59f073c367dfb4b7a8e Oct 30 20:00:31.973372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.973351 5091 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 20:00:31.974297 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:31.974261 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-14a184"] Oct 30 20:00:32.210173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:32.201680 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:32.253080 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:32.253048 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-14a184" event={"ID":"7d11e4f0-1e56-416d-8a1d-56329456cb75","Type":"ContainerStarted","Data":"76bbd3fd3e3eeb1617f864f47528cf17b4054ca49e56e59f073c367dfb4b7a8e"} Oct 30 20:00:33.135276 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:33.135169 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:33.908194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:33.907963 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:34.032117 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:34.032081 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz"] Oct 30 20:00:34.036953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:34.036922 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d799a878-hm5vz"] Oct 30 20:00:34.444984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:34.444953 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be3d162b-06c8-4418-a135-0bd64038f695" path="/var/lib/kubelet/pods/be3d162b-06c8-4418-a135-0bd64038f695/volumes" Oct 30 20:00:34.886184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:34.884816 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:35.285841 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:35.285800 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-14a184" event={"ID":"7d11e4f0-1e56-416d-8a1d-56329456cb75","Type":"ContainerStarted","Data":"a653a243b0a1a73d81ffac777ced033bc92d476b38a4fef3608544b7b0cf535c"} Oct 30 20:00:35.306584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:35.306530 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7325-14a184" podStartSLOduration=1.344360633 podStartE2EDuration="4.306516321s" totalImagesPullingTime="2.962155688s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:35.306396092 +0000 UTC m=+1755.001604049" watchObservedRunningTime="2026-10-30 20:00:35.306516321 +0000 UTC m=+1755.001724279" Oct 30 20:00:35.682849 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:35.682705 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:36.029382 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.029345 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7325-14a184"] Oct 30 20:00:36.035909 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.035876 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7325-14a184"] Oct 30 20:00:36.208160 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.208108 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7325-2b3c41"] Oct 30 20:00:36.209233 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.209211 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7d11e4f0-1e56-416d-8a1d-56329456cb75" containerName="test-curl-7325-14a184" Oct 30 20:00:36.209295 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.209236 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d11e4f0-1e56-416d-8a1d-56329456cb75" containerName="test-curl-7325-14a184" Oct 30 20:00:36.209295 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.209260 5091 cpu_manager.go:414] "containerMap: removing container" podUID="254c3d23-0de3-46c3-bb52-92502e789102" containerName="cleanup" Oct 30 20:00:36.209295 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.209268 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254c3d23-0de3-46c3-bb52-92502e789102" containerName="cleanup" Oct 30 20:00:36.209395 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.209326 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" containerName="cleanup" Oct 30 20:00:36.209395 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.209333 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3161e3-ac92-40a9-bd2f-18384362eeb8" containerName="cleanup" Oct 30 20:00:36.209395 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.209347 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" containerName="cleanup" Oct 30 20:00:36.209395 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.209354 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6f62ee1-1e91-4ed1-a82b-c5371d65af02" containerName="cleanup" Oct 30 20:00:36.209520 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.209488 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a64db5-dcf8-436d-b44a-3a4891da6bde" containerName="cleanup" Oct 30 20:00:36.209520 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.209506 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d11e4f0-1e56-416d-8a1d-56329456cb75" containerName="test-curl-7325-14a184" Oct 30 20:00:36.209582 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.209519 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="96e701fd-f225-4347-8d3c-5be59458f0fb" containerName="cleanup" Oct 30 20:00:36.209582 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.209530 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="125a6e67-31de-4110-8bb8-cdc0af811e46" containerName="cleanup" Oct 30 20:00:36.209582 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.209541 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca322c7-6812-4c40-af1b-62a23ebb4ffe" containerName="cleanup" Oct 30 20:00:36.214034 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.214008 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-2b3c41" Oct 30 20:00:36.217602 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.217578 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-2b3c41"] Oct 30 20:00:36.254018 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.253983 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mgdf\" (UniqueName: \"kubernetes.io/projected/f3d160a8-c929-47bb-a0cc-1dd463ad20a2-kube-api-access-7mgdf\") pod \"test-curl-7325-2b3c41\" (UID: \"f3d160a8-c929-47bb-a0cc-1dd463ad20a2\") " pod="opendatahub/test-curl-7325-2b3c41" Oct 30 20:00:36.355922 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.355834 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mgdf\" (UniqueName: \"kubernetes.io/projected/f3d160a8-c929-47bb-a0cc-1dd463ad20a2-kube-api-access-7mgdf\") pod \"test-curl-7325-2b3c41\" (UID: \"f3d160a8-c929-47bb-a0cc-1dd463ad20a2\") " pod="opendatahub/test-curl-7325-2b3c41" Oct 30 20:00:36.370526 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.370494 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mgdf\" (UniqueName: \"kubernetes.io/projected/f3d160a8-c929-47bb-a0cc-1dd463ad20a2-kube-api-access-7mgdf\") pod \"test-curl-7325-2b3c41\" (UID: \"f3d160a8-c929-47bb-a0cc-1dd463ad20a2\") " pod="opendatahub/test-curl-7325-2b3c41" Oct 30 20:00:36.532174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.531387 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-2b3c41" Oct 30 20:00:36.579215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.577132 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:36.873644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:36.873603 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7325-2b3c41"] Oct 30 20:00:36.874134 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:00:36.874095 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3d160a8_c929_47bb_a0cc_1dd463ad20a2.slice/crio-87b6d030d9201c7416717e800e7dd1481daf60bec8c470951d60041053085363 WatchSource:0}: Error finding container 87b6d030d9201c7416717e800e7dd1481daf60bec8c470951d60041053085363: Status 404 returned error can't find the container with id 87b6d030d9201c7416717e800e7dd1481daf60bec8c470951d60041053085363 Oct 30 20:00:37.310499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:37.310458 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-2b3c41" event={"ID":"f3d160a8-c929-47bb-a0cc-1dd463ad20a2","Type":"ContainerStarted","Data":"725fe06b0fca7c7d2257412351689b6ae922313f1ecedc952a3fc6da0dcbe9a8"} Oct 30 20:00:37.310499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:37.310504 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7325-2b3c41" event={"ID":"f3d160a8-c929-47bb-a0cc-1dd463ad20a2","Type":"ContainerStarted","Data":"87b6d030d9201c7416717e800e7dd1481daf60bec8c470951d60041053085363"} Oct 30 20:00:37.310838 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:37.310566 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7325-14a184" podUID="7d11e4f0-1e56-416d-8a1d-56329456cb75" containerName="test-curl-7325-14a184" containerID="cri-o://a653a243b0a1a73d81ffac777ced033bc92d476b38a4fef3608544b7b0cf535c" gracePeriod=2 Oct 30 20:00:37.333399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:37.333355 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7325-2b3c41" podStartSLOduration=1.333342839 podStartE2EDuration="1.333342839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:37.332436313 +0000 UTC m=+1757.027644272" watchObservedRunningTime="2026-10-30 20:00:37.333342839 +0000 UTC m=+1757.028550798" Oct 30 20:00:37.890233 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:37.890135 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:38.236899 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:38.236816 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7325-2b3c41"] Oct 30 20:00:38.245435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:38.245400 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7325-2b3c41"] Oct 30 20:00:38.744991 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:38.744951 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:38.923169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:38.921841 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:38.982155 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:38.982108 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h"] Oct 30 20:00:38.982880 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:38.982862 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f3d160a8-c929-47bb-a0cc-1dd463ad20a2" containerName="test-curl-7325-2b3c41" Oct 30 20:00:38.982880 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:38.982879 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3d160a8-c929-47bb-a0cc-1dd463ad20a2" containerName="test-curl-7325-2b3c41" Oct 30 20:00:38.983082 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:38.982985 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3d160a8-c929-47bb-a0cc-1dd463ad20a2" containerName="test-curl-7325-2b3c41" Oct 30 20:00:38.986829 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:38.986807 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" Oct 30 20:00:39.000208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:38.993938 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:00:39.000208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:38.994080 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:00:39.000208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:38.997319 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h"] Oct 30 20:00:39.054608 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.054568 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efa7f1b-69d0-4bac-a180-89e102205b24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8520dc94-mg76h\" (UID: \"7efa7f1b-69d0-4bac-a180-89e102205b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" Oct 30 20:00:39.054831 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.054797 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpzp8\" (UniqueName: \"kubernetes.io/projected/7efa7f1b-69d0-4bac-a180-89e102205b24-kube-api-access-mpzp8\") pod \"maas-api-revoke-sub-8520dc94-mg76h\" (UID: \"7efa7f1b-69d0-4bac-a180-89e102205b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" Oct 30 20:00:39.054949 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.054931 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efa7f1b-69d0-4bac-a180-89e102205b24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8520dc94-mg76h\" (UID: \"7efa7f1b-69d0-4bac-a180-89e102205b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" Oct 30 20:00:39.089293 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.089257 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:39.156514 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.156479 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpzp8\" (UniqueName: \"kubernetes.io/projected/7efa7f1b-69d0-4bac-a180-89e102205b24-kube-api-access-mpzp8\") pod \"maas-api-revoke-sub-8520dc94-mg76h\" (UID: \"7efa7f1b-69d0-4bac-a180-89e102205b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" Oct 30 20:00:39.156714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.156575 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efa7f1b-69d0-4bac-a180-89e102205b24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8520dc94-mg76h\" (UID: \"7efa7f1b-69d0-4bac-a180-89e102205b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" Oct 30 20:00:39.156776 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.156732 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efa7f1b-69d0-4bac-a180-89e102205b24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8520dc94-mg76h\" (UID: \"7efa7f1b-69d0-4bac-a180-89e102205b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" Oct 30 20:00:39.157955 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.157927 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efa7f1b-69d0-4bac-a180-89e102205b24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8520dc94-mg76h\" (UID: \"7efa7f1b-69d0-4bac-a180-89e102205b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" Oct 30 20:00:39.159393 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.159367 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efa7f1b-69d0-4bac-a180-89e102205b24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8520dc94-mg76h\" (UID: \"7efa7f1b-69d0-4bac-a180-89e102205b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" Oct 30 20:00:39.172396 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.172369 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpzp8\" (UniqueName: \"kubernetes.io/projected/7efa7f1b-69d0-4bac-a180-89e102205b24-kube-api-access-mpzp8\") pod \"maas-api-revoke-sub-8520dc94-mg76h\" (UID: \"7efa7f1b-69d0-4bac-a180-89e102205b24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" Oct 30 20:00:39.317782 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.317744 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" Oct 30 20:00:39.349345 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.348140 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm"] Oct 30 20:00:39.349345 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.348215 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7325-2b3c41" podUID="f3d160a8-c929-47bb-a0cc-1dd463ad20a2" containerName="test-curl-7325-2b3c41" containerID="cri-o://725fe06b0fca7c7d2257412351689b6ae922313f1ecedc952a3fc6da0dcbe9a8" gracePeriod=2 Oct 30 20:00:39.359788 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.359761 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.372682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.372652 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-4d2317-openshift-default-dockercfg-qqqtl\"" Oct 30 20:00:39.532172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.476109 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-envoy\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.532172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.476251 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c2kc\" (UniqueName: \"kubernetes.io/projected/ec317404-f9f4-499c-bdbe-65109b6a18d2-kube-api-access-5c2kc\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.532172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.476322 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-token\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.532172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.476419 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-workload-socket\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.532172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.476482 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-workload-certs\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.532172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.476561 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec317404-f9f4-499c-bdbe-65109b6a18d2-istiod-ca-cert\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.532172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.476626 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-credential-socket\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.532172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.476653 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-podinfo\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.532172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.476700 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-data\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.698170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.616138 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-workload-certs\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.698170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.616266 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec317404-f9f4-499c-bdbe-65109b6a18d2-istiod-ca-cert\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.698170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.616327 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-credential-socket\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.698170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.616353 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-podinfo\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.698170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.616399 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-data\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.698170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.616482 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-envoy\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.698170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.616574 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c2kc\" (UniqueName: \"kubernetes.io/projected/ec317404-f9f4-499c-bdbe-65109b6a18d2-kube-api-access-5c2kc\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.698170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.631729 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-credential-socket\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.698170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.632471 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-workload-certs\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.698170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.633269 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec317404-f9f4-499c-bdbe-65109b6a18d2-istiod-ca-cert\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.698170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.633536 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-data\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.698170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.651545 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-token\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.698170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.651680 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-workload-socket\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.698170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.652139 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-workload-socket\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.712261 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.712225 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm"] Oct 30 20:00:39.763891 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.739864 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-podinfo\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.763891 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.743407 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-envoy\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.763891 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.748803 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c2kc\" (UniqueName: \"kubernetes.io/projected/ec317404-f9f4-499c-bdbe-65109b6a18d2-kube-api-access-5c2kc\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.763891 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.749810 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:39.763891 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.762636 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-token\") pod \"e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:39.910026 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:00:39.909994 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7efa7f1b_69d0_4bac_a180_89e102205b24.slice/crio-c68ac7af15c1b0d3713f9a76de98bb29b8522c399ef0de2a4da4e5655779f5b8 WatchSource:0}: Error finding container c68ac7af15c1b0d3713f9a76de98bb29b8522c399ef0de2a4da4e5655779f5b8: Status 404 returned error can't find the container with id c68ac7af15c1b0d3713f9a76de98bb29b8522c399ef0de2a4da4e5655779f5b8 Oct 30 20:00:39.913558 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.913527 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h"] Oct 30 20:00:39.924511 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.924483 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:39.957129 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.957104 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-14a184" Oct 30 20:00:39.964510 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.964483 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d11e4f0-1e56-416d-8a1d-56329456cb75-kube-api-access-9l9wd\" (UniqueName: \"kubernetes.io/projected/7d11e4f0-1e56-416d-8a1d-56329456cb75-kube-api-access-9l9wd\") pod \"7d11e4f0-1e56-416d-8a1d-56329456cb75\" (UID: \"7d11e4f0-1e56-416d-8a1d-56329456cb75\") " Oct 30 20:00:39.966609 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:39.966579 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d11e4f0-1e56-416d-8a1d-56329456cb75-kube-api-access-9l9wd" pod "7d11e4f0-1e56-416d-8a1d-56329456cb75" (UID: "7d11e4f0-1e56-416d-8a1d-56329456cb75"). InnerVolumeSpecName "kube-api-access-9l9wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:40.035826 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.035782 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg"] Oct 30 20:00:40.046206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.046175 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da0b084c-s4jrg"] Oct 30 20:00:40.061042 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.061014 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:40.067366 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.067334 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9l9wd\" (UniqueName: \"kubernetes.io/projected/7d11e4f0-1e56-416d-8a1d-56329456cb75-kube-api-access-9l9wd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:40.111628 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.109376 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:40.358460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.358431 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" event={"ID":"7efa7f1b-69d0-4bac-a180-89e102205b24","Type":"ContainerStarted","Data":"c68ac7af15c1b0d3713f9a76de98bb29b8522c399ef0de2a4da4e5655779f5b8"} Oct 30 20:00:40.360055 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.360031 5091 generic.go:356] "Generic (PLEG): container finished" podID="7d11e4f0-1e56-416d-8a1d-56329456cb75" containerID="a653a243b0a1a73d81ffac777ced033bc92d476b38a4fef3608544b7b0cf535c" exitCode=137 Oct 30 20:00:40.360191 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.360083 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-14a184" Oct 30 20:00:40.360191 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.360081 5091 scope.go:122] "RemoveContainer" containerID="a653a243b0a1a73d81ffac777ced033bc92d476b38a4fef3608544b7b0cf535c" Oct 30 20:00:40.371223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.371205 5091 scope.go:122] "RemoveContainer" containerID="a653a243b0a1a73d81ffac777ced033bc92d476b38a4fef3608544b7b0cf535c" Oct 30 20:00:40.371499 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:40.371476 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a653a243b0a1a73d81ffac777ced033bc92d476b38a4fef3608544b7b0cf535c\": container with ID starting with a653a243b0a1a73d81ffac777ced033bc92d476b38a4fef3608544b7b0cf535c not found: ID does not exist" containerID="a653a243b0a1a73d81ffac777ced033bc92d476b38a4fef3608544b7b0cf535c" Oct 30 20:00:40.371627 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.371505 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a653a243b0a1a73d81ffac777ced033bc92d476b38a4fef3608544b7b0cf535c"} err="failed to get container status \"a653a243b0a1a73d81ffac777ced033bc92d476b38a4fef3608544b7b0cf535c\": rpc error: code = NotFound desc = could not find container \"a653a243b0a1a73d81ffac777ced033bc92d476b38a4fef3608544b7b0cf535c\": container with ID starting with a653a243b0a1a73d81ffac777ced033bc92d476b38a4fef3608544b7b0cf535c not found: ID does not exist" Oct 30 20:00:40.373618 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:40.373595 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7325-14a184\" is forbidden: User \"system:node:ip-10-0-132-214.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-214.us-east-2.compute.internal' and this object" podUID="7d11e4f0-1e56-416d-8a1d-56329456cb75" pod="opendatahub/test-curl-7325-14a184" Oct 30 20:00:40.426130 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:00:40.426105 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec317404_f9f4_499c_bdbe_65109b6a18d2.slice/crio-b817866e9fd5bc2ce8ee3e947f25f7e3f4c29d49686ebffe9e25638748094b24 WatchSource:0}: Error finding container b817866e9fd5bc2ce8ee3e947f25f7e3f4c29d49686ebffe9e25638748094b24: Status 404 returned error can't find the container with id b817866e9fd5bc2ce8ee3e947f25f7e3f4c29d49686ebffe9e25638748094b24 Oct 30 20:00:40.427080 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.427057 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm"] Oct 30 20:00:40.428834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.428804 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:00:40.428911 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.428880 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:00:40.428949 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.428909 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:00:40.453476 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:40.448243 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7325-14a184\" is forbidden: User \"system:node:ip-10-0-132-214.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-214.us-east-2.compute.internal' and this object" podUID="7d11e4f0-1e56-416d-8a1d-56329456cb75" pod="opendatahub/test-curl-7325-14a184" Oct 30 20:00:40.461550 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.461526 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d11e4f0-1e56-416d-8a1d-56329456cb75" path="/var/lib/kubelet/pods/7d11e4f0-1e56-416d-8a1d-56329456cb75/volumes" Oct 30 20:00:40.462036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:40.462021 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a" path="/var/lib/kubelet/pods/8cd93c0a-27fa-4b3f-a54d-9e3ef0712f2a/volumes" Oct 30 20:00:41.151039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:41.150984 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:41.372900 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:41.372860 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" event={"ID":"ec317404-f9f4-499c-bdbe-65109b6a18d2","Type":"ContainerStarted","Data":"c6e5709981ad9b33dfea33bf266708bda762034c9a1072c405e98bd7d0a21d90"} Oct 30 20:00:41.373054 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:41.372934 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" event={"ID":"ec317404-f9f4-499c-bdbe-65109b6a18d2","Type":"ContainerStarted","Data":"b817866e9fd5bc2ce8ee3e947f25f7e3f4c29d49686ebffe9e25638748094b24"} Oct 30 20:00:41.374784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:41.374755 5091 generic.go:356] "Generic (PLEG): container finished" podID="7efa7f1b-69d0-4bac-a180-89e102205b24" containerID="3f2600e43b0c20a30f644256b056c491efbe5998fb796435e2941663b185abe3" exitCode=0 Oct 30 20:00:41.374904 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:41.374842 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" event={"ID":"7efa7f1b-69d0-4bac-a180-89e102205b24","Type":"ContainerDied","Data":"3f2600e43b0c20a30f644256b056c491efbe5998fb796435e2941663b185abe3"} Oct 30 20:00:41.416573 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:41.416524 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" podStartSLOduration=2.416509726 podStartE2EDuration="2.416509726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:41.413371842 +0000 UTC m=+1761.108579805" watchObservedRunningTime="2026-10-30 20:00:41.416509726 +0000 UTC m=+1761.111717683" Oct 30 20:00:41.667737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:41.667710 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-2b3c41" Oct 30 20:00:41.691281 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:41.691256 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3d160a8-c929-47bb-a0cc-1dd463ad20a2-kube-api-access-7mgdf\" (UniqueName: \"kubernetes.io/projected/f3d160a8-c929-47bb-a0cc-1dd463ad20a2-kube-api-access-7mgdf\") pod \"f3d160a8-c929-47bb-a0cc-1dd463ad20a2\" (UID: \"f3d160a8-c929-47bb-a0cc-1dd463ad20a2\") " Oct 30 20:00:41.693293 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:41.693268 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d160a8-c929-47bb-a0cc-1dd463ad20a2-kube-api-access-7mgdf" pod "f3d160a8-c929-47bb-a0cc-1dd463ad20a2" (UID: "f3d160a8-c929-47bb-a0cc-1dd463ad20a2"). InnerVolumeSpecName "kube-api-access-7mgdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:41.793013 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:41.792969 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mgdf\" (UniqueName: \"kubernetes.io/projected/f3d160a8-c929-47bb-a0cc-1dd463ad20a2-kube-api-access-7mgdf\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:42.081746 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.070209 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:42.137509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.137476 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:42.242717 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.242677 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:42.387716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.387609 5091 generic.go:356] "Generic (PLEG): container finished" podID="f3d160a8-c929-47bb-a0cc-1dd463ad20a2" containerID="725fe06b0fca7c7d2257412351689b6ae922313f1ecedc952a3fc6da0dcbe9a8" exitCode=137 Oct 30 20:00:42.387716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.387660 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7325-2b3c41" Oct 30 20:00:42.387716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.387703 5091 scope.go:122] "RemoveContainer" containerID="725fe06b0fca7c7d2257412351689b6ae922313f1ecedc952a3fc6da0dcbe9a8" Oct 30 20:00:42.388435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.388355 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:42.389594 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.389437 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:00:42.403326 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.403308 5091 scope.go:122] "RemoveContainer" containerID="725fe06b0fca7c7d2257412351689b6ae922313f1ecedc952a3fc6da0dcbe9a8" Oct 30 20:00:42.403661 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:42.403635 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"725fe06b0fca7c7d2257412351689b6ae922313f1ecedc952a3fc6da0dcbe9a8\": container with ID starting with 725fe06b0fca7c7d2257412351689b6ae922313f1ecedc952a3fc6da0dcbe9a8 not found: ID does not exist" containerID="725fe06b0fca7c7d2257412351689b6ae922313f1ecedc952a3fc6da0dcbe9a8" Oct 30 20:00:42.403729 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.403672 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"725fe06b0fca7c7d2257412351689b6ae922313f1ecedc952a3fc6da0dcbe9a8"} err="failed to get container status \"725fe06b0fca7c7d2257412351689b6ae922313f1ecedc952a3fc6da0dcbe9a8\": rpc error: code = NotFound desc = could not find container \"725fe06b0fca7c7d2257412351689b6ae922313f1ecedc952a3fc6da0dcbe9a8\": container with ID starting with 725fe06b0fca7c7d2257412351689b6ae922313f1ecedc952a3fc6da0dcbe9a8 not found: ID does not exist" Oct 30 20:00:42.428378 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:42.428334 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7325-2b3c41\" is forbidden: User \"system:node:ip-10-0-132-214.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-214.us-east-2.compute.internal' and this object" podUID="f3d160a8-c929-47bb-a0cc-1dd463ad20a2" pod="opendatahub/test-curl-7325-2b3c41" Oct 30 20:00:42.452254 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.452219 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3d160a8-c929-47bb-a0cc-1dd463ad20a2" path="/var/lib/kubelet/pods/f3d160a8-c929-47bb-a0cc-1dd463ad20a2/volumes" Oct 30 20:00:42.614382 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.614356 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" Oct 30 20:00:42.708579 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.708505 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7efa7f1b-69d0-4bac-a180-89e102205b24-kube-api-access-mpzp8\" (UniqueName: \"kubernetes.io/projected/7efa7f1b-69d0-4bac-a180-89e102205b24-kube-api-access-mpzp8\") pod \"7efa7f1b-69d0-4bac-a180-89e102205b24\" (UID: \"7efa7f1b-69d0-4bac-a180-89e102205b24\") " Oct 30 20:00:42.708579 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.708551 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7efa7f1b-69d0-4bac-a180-89e102205b24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efa7f1b-69d0-4bac-a180-89e102205b24-maas-api-openshift-ca\") pod \"7efa7f1b-69d0-4bac-a180-89e102205b24\" (UID: \"7efa7f1b-69d0-4bac-a180-89e102205b24\") " Oct 30 20:00:42.708825 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.708594 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7efa7f1b-69d0-4bac-a180-89e102205b24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efa7f1b-69d0-4bac-a180-89e102205b24-maas-api-xks-ca\") pod \"7efa7f1b-69d0-4bac-a180-89e102205b24\" (UID: \"7efa7f1b-69d0-4bac-a180-89e102205b24\") " Oct 30 20:00:42.708993 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.708970 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7efa7f1b-69d0-4bac-a180-89e102205b24-maas-api-openshift-ca" pod "7efa7f1b-69d0-4bac-a180-89e102205b24" (UID: "7efa7f1b-69d0-4bac-a180-89e102205b24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:42.709251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.709232 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7efa7f1b-69d0-4bac-a180-89e102205b24-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:42.710866 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.710834 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7efa7f1b-69d0-4bac-a180-89e102205b24-maas-api-xks-ca" pod "7efa7f1b-69d0-4bac-a180-89e102205b24" (UID: "7efa7f1b-69d0-4bac-a180-89e102205b24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:42.710995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.710971 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efa7f1b-69d0-4bac-a180-89e102205b24-kube-api-access-mpzp8" pod "7efa7f1b-69d0-4bac-a180-89e102205b24" (UID: "7efa7f1b-69d0-4bac-a180-89e102205b24"). InnerVolumeSpecName "kube-api-access-mpzp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:42.810388 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.810356 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpzp8\" (UniqueName: \"kubernetes.io/projected/7efa7f1b-69d0-4bac-a180-89e102205b24-kube-api-access-mpzp8\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:42.810388 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:42.810386 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7efa7f1b-69d0-4bac-a180-89e102205b24-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:43.223023 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:43.222981 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:43.399100 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:43.399056 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" event={"ID":"7efa7f1b-69d0-4bac-a180-89e102205b24","Type":"ContainerDied","Data":"c68ac7af15c1b0d3713f9a76de98bb29b8522c399ef0de2a4da4e5655779f5b8"} Oct 30 20:00:43.399100 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:43.399089 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h" Oct 30 20:00:43.399689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:43.399117 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c68ac7af15c1b0d3713f9a76de98bb29b8522c399ef0de2a4da4e5655779f5b8" Oct 30 20:00:43.806251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:43.806212 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:43.961862 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:43.961811 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:44.351172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:44.350764 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:45.096245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.096209 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd"] Oct 30 20:00:45.096942 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.096927 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7efa7f1b-69d0-4bac-a180-89e102205b24" containerName="revoke-keys" Oct 30 20:00:45.096997 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.096943 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7efa7f1b-69d0-4bac-a180-89e102205b24" containerName="revoke-keys" Oct 30 20:00:45.097067 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.097058 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7efa7f1b-69d0-4bac-a180-89e102205b24" containerName="revoke-keys" Oct 30 20:00:45.103894 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.103874 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" Oct 30 20:00:45.115360 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.115339 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-4d2317\"" Oct 30 20:00:45.119034 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.119013 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd"] Oct 30 20:00:45.167213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.167169 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26191153-8910-4ca1-ad13-54f97b744a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd\" (UID: \"26191153-8910-4ca1-ad13-54f97b744a37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" Oct 30 20:00:45.167400 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.167283 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckcss\" (UniqueName: \"kubernetes.io/projected/26191153-8910-4ca1-ad13-54f97b744a37-kube-api-access-ckcss\") pod \"payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd\" (UID: \"26191153-8910-4ca1-ad13-54f97b744a37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" Oct 30 20:00:45.204755 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.204674 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:45.269014 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.268964 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26191153-8910-4ca1-ad13-54f97b744a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd\" (UID: \"26191153-8910-4ca1-ad13-54f97b744a37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" Oct 30 20:00:45.269228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.269026 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckcss\" (UniqueName: \"kubernetes.io/projected/26191153-8910-4ca1-ad13-54f97b744a37-kube-api-access-ckcss\") pod \"payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd\" (UID: \"26191153-8910-4ca1-ad13-54f97b744a37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" Oct 30 20:00:45.269779 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.269749 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26191153-8910-4ca1-ad13-54f97b744a37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd\" (UID: \"26191153-8910-4ca1-ad13-54f97b744a37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" Oct 30 20:00:45.285864 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.285830 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckcss\" (UniqueName: \"kubernetes.io/projected/26191153-8910-4ca1-ad13-54f97b744a37-kube-api-access-ckcss\") pod \"payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd\" (UID: \"26191153-8910-4ca1-ad13-54f97b744a37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" Oct 30 20:00:45.425598 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.423986 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" Oct 30 20:00:45.426523 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.426492 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:45.910995 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:00:45.910963 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26191153_8910_4ca1_ad13_54f97b744a37.slice/crio-6ecd75ba9151459e6d4a29a4c443d0ca394df93c93b2ce969e72a981a1b48b9d WatchSource:0}: Error finding container 6ecd75ba9151459e6d4a29a4c443d0ca394df93c93b2ce969e72a981a1b48b9d: Status 404 returned error can't find the container with id 6ecd75ba9151459e6d4a29a4c443d0ca394df93c93b2ce969e72a981a1b48b9d Oct 30 20:00:45.912140 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:45.912112 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd"] Oct 30 20:00:45.982231 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:00:45.982194 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:00:46.385390 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:46.385353 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:46.614037 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:46.613997 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" event={"ID":"26191153-8910-4ca1-ad13-54f97b744a37","Type":"ContainerStarted","Data":"8bd9c240cd6031686fb903e8d906579554e1afce39786023b048b028a1f05ec7"} Oct 30 20:00:46.614308 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:46.614066 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" Oct 30 20:00:46.614308 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:46.614093 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" event={"ID":"26191153-8910-4ca1-ad13-54f97b744a37","Type":"ContainerStarted","Data":"6ecd75ba9151459e6d4a29a4c443d0ca394df93c93b2ce969e72a981a1b48b9d"} Oct 30 20:00:46.640117 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:46.640015 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" podStartSLOduration=1.6399986819999999 podStartE2EDuration="1.639998682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:46.639817808 +0000 UTC m=+1766.335025767" watchObservedRunningTime="2026-10-30 20:00:46.639998682 +0000 UTC m=+1766.335206642" Oct 30 20:00:47.563172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:47.557683 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:48.038141 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:48.038107 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz"] Oct 30 20:00:48.041927 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:48.041906 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-09beadb9-zlkqz"] Oct 30 20:00:48.442955 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:48.442875 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8acc1040-cd5f-4233-8372-b74b4309229e" path="/var/lib/kubelet/pods/8acc1040-cd5f-4233-8372-b74b4309229e/volumes" Oct 30 20:00:48.483121 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:48.483074 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:49.229575 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.229540 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4"] Oct 30 20:00:49.235304 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.235283 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" Oct 30 20:00:49.243166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.243120 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:00:49.244791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.243350 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:00:49.246961 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.246938 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4"] Oct 30 20:00:49.373965 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.373928 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h95cd\" (UniqueName: \"kubernetes.io/projected/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-kube-api-access-h95cd\") pod \"maas-api-revoke-sub-70882eb3-5dtf4\" (UID: \"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" Oct 30 20:00:49.374190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.374129 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70882eb3-5dtf4\" (UID: \"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" Oct 30 20:00:49.374280 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.374218 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70882eb3-5dtf4\" (UID: \"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" Oct 30 20:00:49.486215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.478272 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h95cd\" (UniqueName: \"kubernetes.io/projected/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-kube-api-access-h95cd\") pod \"maas-api-revoke-sub-70882eb3-5dtf4\" (UID: \"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" Oct 30 20:00:49.486215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.478470 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70882eb3-5dtf4\" (UID: \"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" Oct 30 20:00:49.486215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.478516 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70882eb3-5dtf4\" (UID: \"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" Oct 30 20:00:49.492454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.492401 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70882eb3-5dtf4\" (UID: \"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" Oct 30 20:00:49.534318 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.533747 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70882eb3-5dtf4\" (UID: \"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" Oct 30 20:00:49.535332 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.535168 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:49.543038 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.543005 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h95cd\" (UniqueName: \"kubernetes.io/projected/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-kube-api-access-h95cd\") pod \"maas-api-revoke-sub-70882eb3-5dtf4\" (UID: \"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" Oct 30 20:00:49.576344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.576314 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" Oct 30 20:00:49.870177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:49.870109 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4"] Oct 30 20:00:49.870487 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:00:49.870466 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb0ae25_c4ad_4001_bbd9_494d7dc610ba.slice/crio-5673cd08a77b1c3acc9491e43d071f32d2a9d091aa7eca42d315574f237d3ac5 WatchSource:0}: Error finding container 5673cd08a77b1c3acc9491e43d071f32d2a9d091aa7eca42d315574f237d3ac5: Status 404 returned error can't find the container with id 5673cd08a77b1c3acc9491e43d071f32d2a9d091aa7eca42d315574f237d3ac5 Oct 30 20:00:50.442067 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:50.441977 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:50.659802 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:50.659768 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" event={"ID":"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba","Type":"ContainerStarted","Data":"3c0a1f8067e47010be8b331ee71554d59dc15ef14aadf46d4d785efb3c3e05f1"} Oct 30 20:00:50.659977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:50.659824 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" event={"ID":"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba","Type":"ContainerStarted","Data":"5673cd08a77b1c3acc9491e43d071f32d2a9d091aa7eca42d315574f237d3ac5"} Oct 30 20:00:50.689694 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:50.689644 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" podStartSLOduration=1.68962989 podStartE2EDuration="1.68962989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:00:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:00:50.685381258 +0000 UTC m=+1770.380589217" watchObservedRunningTime="2026-10-30 20:00:50.68962989 +0000 UTC m=+1770.384837850" Oct 30 20:00:51.414026 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:51.411185 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:51.670981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:51.670883 5091 generic.go:356] "Generic (PLEG): container finished" podID="4bb0ae25-c4ad-4001-bbd9-494d7dc610ba" containerID="3c0a1f8067e47010be8b331ee71554d59dc15ef14aadf46d4d785efb3c3e05f1" exitCode=0 Oct 30 20:00:51.670981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:51.670922 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" event={"ID":"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba","Type":"ContainerDied","Data":"3c0a1f8067e47010be8b331ee71554d59dc15ef14aadf46d4d785efb3c3e05f1"} Oct 30 20:00:52.314174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:52.314069 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:52.869259 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:52.869234 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" Oct 30 20:00:52.950414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:52.950321 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-maas-api-openshift-ca\") pod \"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba\" (UID: \"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba\") " Oct 30 20:00:52.950596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:52.950458 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-kube-api-access-h95cd\" (UniqueName: \"kubernetes.io/projected/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-kube-api-access-h95cd\") pod \"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba\" (UID: \"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba\") " Oct 30 20:00:52.950661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:52.950590 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-maas-api-xks-ca\") pod \"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba\" (UID: \"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba\") " Oct 30 20:00:52.951174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:52.951064 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-maas-api-openshift-ca" pod "4bb0ae25-c4ad-4001-bbd9-494d7dc610ba" (UID: "4bb0ae25-c4ad-4001-bbd9-494d7dc610ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:00:52.951527 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:52.951344 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:52.954216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:52.954191 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-kube-api-access-h95cd" pod "4bb0ae25-c4ad-4001-bbd9-494d7dc610ba" (UID: "4bb0ae25-c4ad-4001-bbd9-494d7dc610ba"). InnerVolumeSpecName "kube-api-access-h95cd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:00:52.954453 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:52.954406 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-maas-api-xks-ca" pod "4bb0ae25-c4ad-4001-bbd9-494d7dc610ba" (UID: "4bb0ae25-c4ad-4001-bbd9-494d7dc610ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:00:53.052710 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:53.052659 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h95cd\" (UniqueName: \"kubernetes.io/projected/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-kube-api-access-h95cd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:53.052710 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:53.052707 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:00:53.159225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:53.159181 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:53.691087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:53.691051 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" event={"ID":"4bb0ae25-c4ad-4001-bbd9-494d7dc610ba","Type":"ContainerDied","Data":"5673cd08a77b1c3acc9491e43d071f32d2a9d091aa7eca42d315574f237d3ac5"} Oct 30 20:00:53.691087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:53.691081 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4" Oct 30 20:00:53.691393 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:53.691101 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5673cd08a77b1c3acc9491e43d071f32d2a9d091aa7eca42d315574f237d3ac5" Oct 30 20:00:54.044180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:54.037454 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:54.852940 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:54.852901 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:55.041950 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:55.041911 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl"] Oct 30 20:00:55.046386 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:55.046356 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98fda6a7-n8wdl"] Oct 30 20:00:56.114688 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:56.114648 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:56.448607 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:56.448513 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="397b980b-d738-4eee-b685-3404c364fb1e" path="/var/lib/kubelet/pods/397b980b-d738-4eee-b685-3404c364fb1e/volumes" Oct 30 20:00:57.119209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:57.119168 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:57.303109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:57.303069 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:57.626133 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:57.626101 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" Oct 30 20:00:58.126189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:58.126111 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:00:59.215312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:00:59.215273 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:00.281171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:00.280534 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:01.190601 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:01.190562 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:01.345309 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:01.345079 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:02.273185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:02.273082 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:03.343169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:03.339625 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:04.410171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:04.401831 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:05.044657 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:05.044618 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx"] Oct 30 20:01:05.049426 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:05.049394 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e183ab3d-gvklx"] Oct 30 20:01:05.596172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:05.595101 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:06.034439 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:06.034407 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl"] Oct 30 20:01:06.038065 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:06.038042 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0f32d51-s5hhl"] Oct 30 20:01:06.444421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:06.444348 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96550e8d-b40a-43f1-9942-9eb094b9293d" path="/var/lib/kubelet/pods/96550e8d-b40a-43f1-9942-9eb094b9293d/volumes" Oct 30 20:01:06.444937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:06.444920 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c48f98ff-5a17-411b-bc08-328bf002e862" path="/var/lib/kubelet/pods/c48f98ff-5a17-411b-bc08-328bf002e862/volumes" Oct 30 20:01:06.547175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:06.547114 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:07.014889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:07.014852 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt"] Oct 30 20:01:07.153179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:07.149691 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:07.279322 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:07.279244 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7334-54897d"] Oct 30 20:01:07.279995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:07.279980 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4bb0ae25-c4ad-4001-bbd9-494d7dc610ba" containerName="revoke-keys" Oct 30 20:01:07.280047 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:07.279996 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb0ae25-c4ad-4001-bbd9-494d7dc610ba" containerName="revoke-keys" Oct 30 20:01:07.280108 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:07.280099 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb0ae25-c4ad-4001-bbd9-494d7dc610ba" containerName="revoke-keys" Oct 30 20:01:07.285463 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:07.285443 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-54897d" Oct 30 20:01:07.295551 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:07.295480 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dqx9r\"" Oct 30 20:01:07.297710 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:07.297679 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7334-54897d"] Oct 30 20:01:07.398191 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:07.398157 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lrq6\" (UniqueName: \"kubernetes.io/projected/93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0-kube-api-access-2lrq6\") pod \"test-curl-7334-54897d\" (UID: \"93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0\") " pod="opendatahub/test-curl-7334-54897d" Oct 30 20:01:07.500045 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:07.500008 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lrq6\" (UniqueName: \"kubernetes.io/projected/93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0-kube-api-access-2lrq6\") pod \"test-curl-7334-54897d\" (UID: \"93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0\") " pod="opendatahub/test-curl-7334-54897d" Oct 30 20:01:07.513820 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:07.513794 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lrq6\" (UniqueName: \"kubernetes.io/projected/93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0-kube-api-access-2lrq6\") pod \"test-curl-7334-54897d\" (UID: \"93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0\") " pod="opendatahub/test-curl-7334-54897d" Oct 30 20:01:07.612225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:07.610889 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:07.612225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:07.611443 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-54897d" Oct 30 20:01:07.975195 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:01:07.969059 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93cbe40c_5ba8_4d12_afb5_3c3ab07da9a0.slice/crio-9d3e280305b9f867df37a1c0e2d4f1187d4e8d6dc93f9820cef5778f3450cbd0 WatchSource:0}: Error finding container 9d3e280305b9f867df37a1c0e2d4f1187d4e8d6dc93f9820cef5778f3450cbd0: Status 404 returned error can't find the container with id 9d3e280305b9f867df37a1c0e2d4f1187d4e8d6dc93f9820cef5778f3450cbd0 Oct 30 20:01:07.975195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:07.970634 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7334-54897d"] Oct 30 20:01:08.707170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:08.704114 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:08.886204 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:08.881896 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7334-54897d" event={"ID":"93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0","Type":"ContainerStarted","Data":"319510e7d349f1557c0c1666d6b46d8e5cb35d9f1cfe490301e795cf53da68c2"} Oct 30 20:01:08.886204 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:08.881952 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7334-54897d" event={"ID":"93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0","Type":"ContainerStarted","Data":"9d3e280305b9f867df37a1c0e2d4f1187d4e8d6dc93f9820cef5778f3450cbd0"} Oct 30 20:01:08.904300 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:08.904258 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7334-54897d" podStartSLOduration=1.904246486 podStartE2EDuration="1.904246486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:08.90308854 +0000 UTC m=+1788.598296497" watchObservedRunningTime="2026-10-30 20:01:08.904246486 +0000 UTC m=+1788.599454444" Oct 30 20:01:09.741982 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:09.737182 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:09.838885 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:09.838853 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7334-54897d"] Oct 30 20:01:09.855220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:09.852875 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7334-54897d"] Oct 30 20:01:10.027439 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.027406 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7334-1b6145"] Oct 30 20:01:10.028188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.028166 5091 cpu_manager.go:414] "containerMap: removing container" podUID="93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0" containerName="test-curl-7334-54897d" Oct 30 20:01:10.028188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.028188 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0" containerName="test-curl-7334-54897d" Oct 30 20:01:10.028368 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.028356 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0" containerName="test-curl-7334-54897d" Oct 30 20:01:10.032637 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.032230 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-1b6145" Oct 30 20:01:10.036738 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.036716 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7334-1b6145"] Oct 30 20:01:10.147252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.147206 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g86l9\" (UniqueName: \"kubernetes.io/projected/d2fcc610-9ace-49bc-a1dd-f90b7078f22c-kube-api-access-g86l9\") pod \"test-curl-7334-1b6145\" (UID: \"d2fcc610-9ace-49bc-a1dd-f90b7078f22c\") " pod="opendatahub/test-curl-7334-1b6145" Oct 30 20:01:10.248982 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.248946 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g86l9\" (UniqueName: \"kubernetes.io/projected/d2fcc610-9ace-49bc-a1dd-f90b7078f22c-kube-api-access-g86l9\") pod \"test-curl-7334-1b6145\" (UID: \"d2fcc610-9ace-49bc-a1dd-f90b7078f22c\") " pod="opendatahub/test-curl-7334-1b6145" Oct 30 20:01:10.271779 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.271746 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g86l9\" (UniqueName: \"kubernetes.io/projected/d2fcc610-9ace-49bc-a1dd-f90b7078f22c-kube-api-access-g86l9\") pod \"test-curl-7334-1b6145\" (UID: \"d2fcc610-9ace-49bc-a1dd-f90b7078f22c\") " pod="opendatahub/test-curl-7334-1b6145" Oct 30 20:01:10.348036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.347963 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-1b6145" Oct 30 20:01:10.714012 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.713906 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:10.719523 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.719492 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7334-1b6145"] Oct 30 20:01:10.721604 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:01:10.721573 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2fcc610_9ace_49bc_a1dd_f90b7078f22c.slice/crio-ce6754a9a9c5abee91fb29c3db9062432afdb3b99d83ae28927b0a0d0e369645 WatchSource:0}: Error finding container ce6754a9a9c5abee91fb29c3db9062432afdb3b99d83ae28927b0a0d0e369645: Status 404 returned error can't find the container with id ce6754a9a9c5abee91fb29c3db9062432afdb3b99d83ae28927b0a0d0e369645 Oct 30 20:01:10.904365 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.904324 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7334-1b6145" event={"ID":"d2fcc610-9ace-49bc-a1dd-f90b7078f22c","Type":"ContainerStarted","Data":"b270e0001d958cb8ba232b8a882027e1187e6e4acb9b835c6837bff5408812eb"} Oct 30 20:01:10.904365 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.904369 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7334-1b6145" event={"ID":"d2fcc610-9ace-49bc-a1dd-f90b7078f22c","Type":"ContainerStarted","Data":"ce6754a9a9c5abee91fb29c3db9062432afdb3b99d83ae28927b0a0d0e369645"} Oct 30 20:01:10.904876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.904479 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7334-54897d" podUID="93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0" containerName="test-curl-7334-54897d" containerID="cri-o://319510e7d349f1557c0c1666d6b46d8e5cb35d9f1cfe490301e795cf53da68c2" gracePeriod=2 Oct 30 20:01:10.940972 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:10.940924 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7334-1b6145" podStartSLOduration=0.940910988 podStartE2EDuration="940.910988ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:10.934523638 +0000 UTC m=+1790.629731596" watchObservedRunningTime="2026-10-30 20:01:10.940910988 +0000 UTC m=+1790.636118945" Oct 30 20:01:11.709548 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:11.709510 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:11.792247 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:11.792209 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7334-1b6145"] Oct 30 20:01:11.798040 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:11.798008 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7334-1b6145"] Oct 30 20:01:11.970924 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:11.970849 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7334-2c1f58"] Oct 30 20:01:11.971624 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:11.971605 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d2fcc610-9ace-49bc-a1dd-f90b7078f22c" containerName="test-curl-7334-1b6145" Oct 30 20:01:11.971624 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:11.971624 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2fcc610-9ace-49bc-a1dd-f90b7078f22c" containerName="test-curl-7334-1b6145" Oct 30 20:01:11.971764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:11.971737 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2fcc610-9ace-49bc-a1dd-f90b7078f22c" containerName="test-curl-7334-1b6145" Oct 30 20:01:11.975430 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:11.975405 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-2c1f58" Oct 30 20:01:11.983306 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:11.983278 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7334-2c1f58"] Oct 30 20:01:12.093012 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:12.092970 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8fd5\" (UniqueName: \"kubernetes.io/projected/29e7e31e-211d-40ad-a3c2-dd4e1006b0e2-kube-api-access-r8fd5\") pod \"test-curl-7334-2c1f58\" (UID: \"29e7e31e-211d-40ad-a3c2-dd4e1006b0e2\") " pod="opendatahub/test-curl-7334-2c1f58" Oct 30 20:01:12.194590 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:12.194549 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8fd5\" (UniqueName: \"kubernetes.io/projected/29e7e31e-211d-40ad-a3c2-dd4e1006b0e2-kube-api-access-r8fd5\") pod \"test-curl-7334-2c1f58\" (UID: \"29e7e31e-211d-40ad-a3c2-dd4e1006b0e2\") " pod="opendatahub/test-curl-7334-2c1f58" Oct 30 20:01:12.210466 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:12.210438 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8fd5\" (UniqueName: \"kubernetes.io/projected/29e7e31e-211d-40ad-a3c2-dd4e1006b0e2-kube-api-access-r8fd5\") pod \"test-curl-7334-2c1f58\" (UID: \"29e7e31e-211d-40ad-a3c2-dd4e1006b0e2\") " pod="opendatahub/test-curl-7334-2c1f58" Oct 30 20:01:12.296705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:12.296677 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-2c1f58" Oct 30 20:01:12.666769 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:01:12.666742 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29e7e31e_211d_40ad_a3c2_dd4e1006b0e2.slice/crio-8bdd0adcbe1d2fb02632b313d706936283a99e44ab66406643d973ae869fd5d4 WatchSource:0}: Error finding container 8bdd0adcbe1d2fb02632b313d706936283a99e44ab66406643d973ae869fd5d4: Status 404 returned error can't find the container with id 8bdd0adcbe1d2fb02632b313d706936283a99e44ab66406643d973ae869fd5d4 Oct 30 20:01:12.668139 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:12.668110 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7334-2c1f58"] Oct 30 20:01:12.976802 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:12.976764 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7334-2c1f58" event={"ID":"29e7e31e-211d-40ad-a3c2-dd4e1006b0e2","Type":"ContainerStarted","Data":"6f5cd24fff94ba40a7623f65758710941dd90d8a72cd4346667846bf1ca6b582"} Oct 30 20:01:12.977217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:12.976818 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7334-2c1f58" event={"ID":"29e7e31e-211d-40ad-a3c2-dd4e1006b0e2","Type":"ContainerStarted","Data":"8bdd0adcbe1d2fb02632b313d706936283a99e44ab66406643d973ae869fd5d4"} Oct 30 20:01:12.977217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:12.977079 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7334-1b6145" podUID="d2fcc610-9ace-49bc-a1dd-f90b7078f22c" containerName="test-curl-7334-1b6145" containerID="cri-o://b270e0001d958cb8ba232b8a882027e1187e6e4acb9b835c6837bff5408812eb" gracePeriod=2 Oct 30 20:01:13.003362 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:13.003292 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7334-2c1f58" podStartSLOduration=2.003275238 podStartE2EDuration="2.003275238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:12.999827146 +0000 UTC m=+1792.695035105" watchObservedRunningTime="2026-10-30 20:01:13.003275238 +0000 UTC m=+1792.698483198" Oct 30 20:01:13.018696 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:13.018660 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:13.135116 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:13.135093 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-54897d" Oct 30 20:01:13.316444 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:13.316411 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0-kube-api-access-2lrq6\" (UniqueName: \"kubernetes.io/projected/93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0-kube-api-access-2lrq6\") pod \"93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0\" (UID: \"93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0\") " Oct 30 20:01:13.319056 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:13.319026 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0-kube-api-access-2lrq6" pod "93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0" (UID: "93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0"). InnerVolumeSpecName "kube-api-access-2lrq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:13.418480 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:13.418448 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lrq6\" (UniqueName: \"kubernetes.io/projected/93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0-kube-api-access-2lrq6\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:13.430863 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:13.430819 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:13.928341 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:13.924380 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7334-2c1f58"] Oct 30 20:01:14.046340 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:13.990246 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7334-2c1f58"] Oct 30 20:01:14.197219 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.183215 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7334-68df11"] Oct 30 20:01:14.215201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.197467 5091 cpu_manager.go:414] "containerMap: removing container" podUID="29e7e31e-211d-40ad-a3c2-dd4e1006b0e2" containerName="test-curl-7334-2c1f58" Oct 30 20:01:14.215201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.197504 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29e7e31e-211d-40ad-a3c2-dd4e1006b0e2" containerName="test-curl-7334-2c1f58" Oct 30 20:01:14.215201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.197739 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="29e7e31e-211d-40ad-a3c2-dd4e1006b0e2" containerName="test-curl-7334-2c1f58" Oct 30 20:01:14.233612 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.232317 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:14.234715 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.234531 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7334-68df11"] Oct 30 20:01:14.234715 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.234610 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-68df11" Oct 30 20:01:14.235059 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.234902 5091 generic.go:356] "Generic (PLEG): container finished" podID="93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0" containerID="319510e7d349f1557c0c1666d6b46d8e5cb35d9f1cfe490301e795cf53da68c2" exitCode=137 Oct 30 20:01:14.235177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.235081 5091 scope.go:122] "RemoveContainer" containerID="319510e7d349f1557c0c1666d6b46d8e5cb35d9f1cfe490301e795cf53da68c2" Oct 30 20:01:14.237516 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.235759 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-54897d" Oct 30 20:01:14.262836 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:01:14.262803 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7334-54897d\" is forbidden: User \"system:node:ip-10-0-132-214.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-214.us-east-2.compute.internal' and this object" podUID="93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0" pod="opendatahub/test-curl-7334-54897d" Oct 30 20:01:14.263424 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.263403 5091 scope.go:122] "RemoveContainer" containerID="319510e7d349f1557c0c1666d6b46d8e5cb35d9f1cfe490301e795cf53da68c2" Oct 30 20:01:14.263737 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:01:14.263713 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"319510e7d349f1557c0c1666d6b46d8e5cb35d9f1cfe490301e795cf53da68c2\": container with ID starting with 319510e7d349f1557c0c1666d6b46d8e5cb35d9f1cfe490301e795cf53da68c2 not found: ID does not exist" containerID="319510e7d349f1557c0c1666d6b46d8e5cb35d9f1cfe490301e795cf53da68c2" Oct 30 20:01:14.263817 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.263746 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"319510e7d349f1557c0c1666d6b46d8e5cb35d9f1cfe490301e795cf53da68c2"} err="failed to get container status \"319510e7d349f1557c0c1666d6b46d8e5cb35d9f1cfe490301e795cf53da68c2\": rpc error: code = NotFound desc = could not find container \"319510e7d349f1557c0c1666d6b46d8e5cb35d9f1cfe490301e795cf53da68c2\": container with ID starting with 319510e7d349f1557c0c1666d6b46d8e5cb35d9f1cfe490301e795cf53da68c2 not found: ID does not exist" Oct 30 20:01:14.317033 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.317004 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb6cq\" (UniqueName: \"kubernetes.io/projected/efbca5e8-57ff-4240-8350-7b433f1c9e4d-kube-api-access-pb6cq\") pod \"test-curl-7334-68df11\" (UID: \"efbca5e8-57ff-4240-8350-7b433f1c9e4d\") " pod="opendatahub/test-curl-7334-68df11" Oct 30 20:01:14.418025 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.417992 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb6cq\" (UniqueName: \"kubernetes.io/projected/efbca5e8-57ff-4240-8350-7b433f1c9e4d-kube-api-access-pb6cq\") pod \"test-curl-7334-68df11\" (UID: \"efbca5e8-57ff-4240-8350-7b433f1c9e4d\") " pod="opendatahub/test-curl-7334-68df11" Oct 30 20:01:14.432477 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.432454 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb6cq\" (UniqueName: \"kubernetes.io/projected/efbca5e8-57ff-4240-8350-7b433f1c9e4d-kube-api-access-pb6cq\") pod \"test-curl-7334-68df11\" (UID: \"efbca5e8-57ff-4240-8350-7b433f1c9e4d\") " pod="opendatahub/test-curl-7334-68df11" Oct 30 20:01:14.442639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.442616 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0" path="/var/lib/kubelet/pods/93cbe40c-5ba8-4d12-afb5-3c3ab07da9a0/volumes" Oct 30 20:01:14.552672 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.552638 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-68df11" Oct 30 20:01:14.859379 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:01:14.859344 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefbca5e8_57ff_4240_8350_7b433f1c9e4d.slice/crio-7ce2777badbdf97bb4a44cc85de3bd416e07c0c0ad9f71c3f4088ad6f600147c WatchSource:0}: Error finding container 7ce2777badbdf97bb4a44cc85de3bd416e07c0c0ad9f71c3f4088ad6f600147c: Status 404 returned error can't find the container with id 7ce2777badbdf97bb4a44cc85de3bd416e07c0c0ad9f71c3f4088ad6f600147c Oct 30 20:01:14.863027 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:14.862984 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7334-68df11"] Oct 30 20:01:15.174531 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:15.174498 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-1b6145" Oct 30 20:01:15.231091 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:15.231062 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2fcc610-9ace-49bc-a1dd-f90b7078f22c-kube-api-access-g86l9\" (UniqueName: \"kubernetes.io/projected/d2fcc610-9ace-49bc-a1dd-f90b7078f22c-kube-api-access-g86l9\") pod \"d2fcc610-9ace-49bc-a1dd-f90b7078f22c\" (UID: \"d2fcc610-9ace-49bc-a1dd-f90b7078f22c\") " Oct 30 20:01:15.233139 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:15.233105 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2fcc610-9ace-49bc-a1dd-f90b7078f22c-kube-api-access-g86l9" pod "d2fcc610-9ace-49bc-a1dd-f90b7078f22c" (UID: "d2fcc610-9ace-49bc-a1dd-f90b7078f22c"). InnerVolumeSpecName "kube-api-access-g86l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:15.244905 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:15.244878 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7334-68df11" event={"ID":"efbca5e8-57ff-4240-8350-7b433f1c9e4d","Type":"ContainerStarted","Data":"25e0d0813243a12b485c303aeb09deb4b0e673214b30507a271521ff3558027d"} Oct 30 20:01:15.245036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:15.244919 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7334-68df11" event={"ID":"efbca5e8-57ff-4240-8350-7b433f1c9e4d","Type":"ContainerStarted","Data":"7ce2777badbdf97bb4a44cc85de3bd416e07c0c0ad9f71c3f4088ad6f600147c"} Oct 30 20:01:15.247664 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:15.247639 5091 generic.go:356] "Generic (PLEG): container finished" podID="d2fcc610-9ace-49bc-a1dd-f90b7078f22c" containerID="b270e0001d958cb8ba232b8a882027e1187e6e4acb9b835c6837bff5408812eb" exitCode=137 Oct 30 20:01:15.247786 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:15.247679 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-1b6145" Oct 30 20:01:15.247786 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:15.247717 5091 scope.go:122] "RemoveContainer" containerID="b270e0001d958cb8ba232b8a882027e1187e6e4acb9b835c6837bff5408812eb" Oct 30 20:01:15.248037 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:15.248011 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7334-2c1f58" podUID="29e7e31e-211d-40ad-a3c2-dd4e1006b0e2" containerName="test-curl-7334-2c1f58" containerID="cri-o://6f5cd24fff94ba40a7623f65758710941dd90d8a72cd4346667846bf1ca6b582" gracePeriod=2 Oct 30 20:01:15.262594 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:15.262571 5091 scope.go:122] "RemoveContainer" containerID="b270e0001d958cb8ba232b8a882027e1187e6e4acb9b835c6837bff5408812eb" Oct 30 20:01:15.262898 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:01:15.262876 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b270e0001d958cb8ba232b8a882027e1187e6e4acb9b835c6837bff5408812eb\": container with ID starting with b270e0001d958cb8ba232b8a882027e1187e6e4acb9b835c6837bff5408812eb not found: ID does not exist" containerID="b270e0001d958cb8ba232b8a882027e1187e6e4acb9b835c6837bff5408812eb" Oct 30 20:01:15.263000 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:15.262904 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b270e0001d958cb8ba232b8a882027e1187e6e4acb9b835c6837bff5408812eb"} err="failed to get container status \"b270e0001d958cb8ba232b8a882027e1187e6e4acb9b835c6837bff5408812eb\": rpc error: code = NotFound desc = could not find container \"b270e0001d958cb8ba232b8a882027e1187e6e4acb9b835c6837bff5408812eb\": container with ID starting with b270e0001d958cb8ba232b8a882027e1187e6e4acb9b835c6837bff5408812eb not found: ID does not exist" Oct 30 20:01:15.314271 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:01:15.314234 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7334-1b6145\" is forbidden: User \"system:node:ip-10-0-132-214.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-214.us-east-2.compute.internal' and this object" podUID="d2fcc610-9ace-49bc-a1dd-f90b7078f22c" pod="opendatahub/test-curl-7334-1b6145" Oct 30 20:01:15.333676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:15.333645 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g86l9\" (UniqueName: \"kubernetes.io/projected/d2fcc610-9ace-49bc-a1dd-f90b7078f22c-kube-api-access-g86l9\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:15.470377 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:15.470137 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:15.978041 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:01:15.977998 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:01:16.241221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:16.239490 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7334-68df11" podStartSLOduration=2.239467796 podStartE2EDuration="2.239467796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:15.310489871 +0000 UTC m=+1795.005697831" watchObservedRunningTime="2026-10-30 20:01:16.239467796 +0000 UTC m=+1795.934675760" Oct 30 20:01:16.253167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:16.252206 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7334-68df11"] Oct 30 20:01:16.276503 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:16.276449 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7334-68df11"] Oct 30 20:01:16.450334 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:16.448883 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2fcc610-9ace-49bc-a1dd-f90b7078f22c" path="/var/lib/kubelet/pods/d2fcc610-9ace-49bc-a1dd-f90b7078f22c/volumes" Oct 30 20:01:16.726189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:16.707996 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:17.329944 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:17.329904 5091 generic.go:356] "Generic (PLEG): container finished" podID="29e7e31e-211d-40ad-a3c2-dd4e1006b0e2" containerID="6f5cd24fff94ba40a7623f65758710941dd90d8a72cd4346667846bf1ca6b582" exitCode=137 Oct 30 20:01:17.330406 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:17.330101 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7334-68df11" podUID="efbca5e8-57ff-4240-8350-7b433f1c9e4d" containerName="test-curl-7334-68df11" containerID="cri-o://25e0d0813243a12b485c303aeb09deb4b0e673214b30507a271521ff3558027d" gracePeriod=2 Oct 30 20:01:17.536766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:17.536740 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-2c1f58" Oct 30 20:01:17.601698 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:17.601651 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29e7e31e-211d-40ad-a3c2-dd4e1006b0e2-kube-api-access-r8fd5\" (UniqueName: \"kubernetes.io/projected/29e7e31e-211d-40ad-a3c2-dd4e1006b0e2-kube-api-access-r8fd5\") pod \"29e7e31e-211d-40ad-a3c2-dd4e1006b0e2\" (UID: \"29e7e31e-211d-40ad-a3c2-dd4e1006b0e2\") " Oct 30 20:01:17.604243 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:17.604213 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29e7e31e-211d-40ad-a3c2-dd4e1006b0e2-kube-api-access-r8fd5" pod "29e7e31e-211d-40ad-a3c2-dd4e1006b0e2" (UID: "29e7e31e-211d-40ad-a3c2-dd4e1006b0e2"). InnerVolumeSpecName "kube-api-access-r8fd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:17.704031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:17.703998 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8fd5\" (UniqueName: \"kubernetes.io/projected/29e7e31e-211d-40ad-a3c2-dd4e1006b0e2-kube-api-access-r8fd5\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:18.052596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.045923 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:18.269168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.268551 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:18.340522 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.340443 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-2c1f58" Oct 30 20:01:18.340522 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.340483 5091 scope.go:122] "RemoveContainer" containerID="6f5cd24fff94ba40a7623f65758710941dd90d8a72cd4346667846bf1ca6b582" Oct 30 20:01:18.377171 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:01:18.375401 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7334-2c1f58\" is forbidden: User \"system:node:ip-10-0-132-214.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-214.us-east-2.compute.internal' and this object" podUID="29e7e31e-211d-40ad-a3c2-dd4e1006b0e2" pod="opendatahub/test-curl-7334-2c1f58" Oct 30 20:01:18.443425 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.443392 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29e7e31e-211d-40ad-a3c2-dd4e1006b0e2" path="/var/lib/kubelet/pods/29e7e31e-211d-40ad-a3c2-dd4e1006b0e2/volumes" Oct 30 20:01:18.521432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.521397 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz"] Oct 30 20:01:18.522451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.522424 5091 cpu_manager.go:414] "containerMap: removing container" podUID="efbca5e8-57ff-4240-8350-7b433f1c9e4d" containerName="test-curl-7334-68df11" Oct 30 20:01:18.522451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.522446 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efbca5e8-57ff-4240-8350-7b433f1c9e4d" containerName="test-curl-7334-68df11" Oct 30 20:01:18.522730 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.522657 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="efbca5e8-57ff-4240-8350-7b433f1c9e4d" containerName="test-curl-7334-68df11" Oct 30 20:01:18.532616 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.532586 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.544909 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.544839 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c9bce8-openshift-default-dockercfg-6mb7x\"" Oct 30 20:01:18.597410 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.597338 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz"] Oct 30 20:01:18.642471 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.642426 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxzqk\" (UniqueName: \"kubernetes.io/projected/374daf82-137e-4a07-ad5f-b205bcb571fc-kube-api-access-wxzqk\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.642749 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.642714 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-envoy\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.642901 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.642884 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-token\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.643034 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.643010 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374daf82-137e-4a07-ad5f-b205bcb571fc-istiod-ca-cert\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.643195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.643177 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-podinfo\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.643372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.643349 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-workload-certs\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.643563 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.643543 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-workload-socket\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.646265 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.646238 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-credential-socket\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.656169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.646475 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-data\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.806332 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.806285 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-credential-socket\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.806724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.806656 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-data\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.817167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.807057 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxzqk\" (UniqueName: \"kubernetes.io/projected/374daf82-137e-4a07-ad5f-b205bcb571fc-kube-api-access-wxzqk\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.867805 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.867722 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-envoy\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.868074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.868049 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-token\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.868207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.868188 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374daf82-137e-4a07-ad5f-b205bcb571fc-istiod-ca-cert\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.868340 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.868326 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-podinfo\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.868482 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.868471 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-workload-certs\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.868644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.868631 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-workload-socket\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.869243 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.869222 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-workload-socket\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.869394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.810844 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:18.872275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.811271 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-credential-socket\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.872814 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.811504 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-data\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.873361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.873339 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-workload-certs\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.922705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.919688 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxzqk\" (UniqueName: \"kubernetes.io/projected/374daf82-137e-4a07-ad5f-b205bcb571fc-kube-api-access-wxzqk\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.941174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.924872 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374daf82-137e-4a07-ad5f-b205bcb571fc-istiod-ca-cert\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.941552 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.932665 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-podinfo\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.957172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.955474 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-envoy\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:18.980352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:18.967782 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-token\") pod \"e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:19.197666 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:19.197575 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:19.272179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:19.265265 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:19.459357 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:19.447498 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:19.689362 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:19.689332 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz"] Oct 30 20:01:19.700791 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:01:19.698861 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374daf82_137e_4a07_ad5f_b205bcb571fc.slice/crio-0d561ce8724f5a45e8534792b347e897d604f46b6b120ff2515836e820a5db70 WatchSource:0}: Error finding container 0d561ce8724f5a45e8534792b347e897d604f46b6b120ff2515836e820a5db70: Status 404 returned error can't find the container with id 0d561ce8724f5a45e8534792b347e897d604f46b6b120ff2515836e820a5db70 Oct 30 20:01:19.701080 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:19.700878 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:19.702926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:19.702887 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:01:19.703045 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:19.702977 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:01:19.703045 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:19.703023 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:01:19.743439 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:19.743295 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-68df11" Oct 30 20:01:19.886173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:19.885155 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efbca5e8-57ff-4240-8350-7b433f1c9e4d-kube-api-access-pb6cq\" (UniqueName: \"kubernetes.io/projected/efbca5e8-57ff-4240-8350-7b433f1c9e4d-kube-api-access-pb6cq\") pod \"efbca5e8-57ff-4240-8350-7b433f1c9e4d\" (UID: \"efbca5e8-57ff-4240-8350-7b433f1c9e4d\") " Oct 30 20:01:19.909731 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:19.909695 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efbca5e8-57ff-4240-8350-7b433f1c9e4d-kube-api-access-pb6cq" pod "efbca5e8-57ff-4240-8350-7b433f1c9e4d" (UID: "efbca5e8-57ff-4240-8350-7b433f1c9e4d"). InnerVolumeSpecName "kube-api-access-pb6cq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:19.987961 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:19.987846 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb6cq\" (UniqueName: \"kubernetes.io/projected/efbca5e8-57ff-4240-8350-7b433f1c9e4d-kube-api-access-pb6cq\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:19.998480 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:19.998443 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:20.405591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.405549 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r"] Oct 30 20:01:20.411703 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.411680 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.425018 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.413933 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r"] Oct 30 20:01:20.425432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.419922 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-75b361-openshift-default-dockercfg-mcqdv\"" Oct 30 20:01:20.443824 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.443794 5091 generic.go:356] "Generic (PLEG): container finished" podID="efbca5e8-57ff-4240-8350-7b433f1c9e4d" containerID="25e0d0813243a12b485c303aeb09deb4b0e673214b30507a271521ff3558027d" exitCode=137 Oct 30 20:01:20.444893 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.444323 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7334-68df11" Oct 30 20:01:20.829794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.828789 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-podinfo\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.829794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.828888 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-workload-socket\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.829794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.828937 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-data\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.829794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.828972 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-credential-socket\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.829794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.829043 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-token\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.829794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.829321 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-workload-certs\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.829794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.829377 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-envoy\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.829794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.829411 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vxp4\" (UniqueName: \"kubernetes.io/projected/2bcb265a-c599-404a-b848-b62b6e632f2a-kube-api-access-9vxp4\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.829794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.829441 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcb265a-c599-404a-b848-b62b6e632f2a-istiod-ca-cert\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.993171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.966902 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-workload-socket\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.993486 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.993457 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-data\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.993642 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.993624 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-credential-socket\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.993803 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.993789 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-token\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.994170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.994131 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-workload-certs\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.994353 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.994337 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-envoy\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.994484 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.994467 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vxp4\" (UniqueName: \"kubernetes.io/projected/2bcb265a-c599-404a-b848-b62b6e632f2a-kube-api-access-9vxp4\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.994603 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.994589 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcb265a-c599-404a-b848-b62b6e632f2a-istiod-ca-cert\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.994736 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.994721 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-podinfo\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.995231 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.969349 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-workload-socket\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.995739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.995712 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-data\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.995855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.969652 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:20.996335 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.996313 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-credential-socket\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.998002 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.997100 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-workload-certs\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.998002 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.997483 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcb265a-c599-404a-b848-b62b6e632f2a-istiod-ca-cert\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:20.999754 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:20.999730 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-envoy\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:21.001017 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.000999 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-podinfo\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:21.017278 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.015186 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efbca5e8-57ff-4240-8350-7b433f1c9e4d" path="/var/lib/kubelet/pods/efbca5e8-57ff-4240-8350-7b433f1c9e4d/volumes" Oct 30 20:01:21.017278 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.016378 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" event={"ID":"374daf82-137e-4a07-ad5f-b205bcb571fc","Type":"ContainerStarted","Data":"f22c1b8878712b749f290f1e15da4c381305985e674b688ad3285f6b18ec360a"} Oct 30 20:01:21.017278 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.016432 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" event={"ID":"374daf82-137e-4a07-ad5f-b205bcb571fc","Type":"ContainerStarted","Data":"0d561ce8724f5a45e8534792b347e897d604f46b6b120ff2515836e820a5db70"} Oct 30 20:01:21.022295 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.022271 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vxp4\" (UniqueName: \"kubernetes.io/projected/2bcb265a-c599-404a-b848-b62b6e632f2a-kube-api-access-9vxp4\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:21.022480 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.022276 5091 scope.go:122] "RemoveContainer" containerID="25e0d0813243a12b485c303aeb09deb4b0e673214b30507a271521ff3558027d" Oct 30 20:01:21.022931 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.022904 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-token\") pod \"e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:21.040195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.040084 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-75b361-openshift-default-dockercfg-mcqdv\"" Oct 30 20:01:21.054184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.045789 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:21.103691 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.103667 5091 scope.go:122] "RemoveContainer" containerID="25e0d0813243a12b485c303aeb09deb4b0e673214b30507a271521ff3558027d" Oct 30 20:01:21.104020 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:01:21.103990 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e0d0813243a12b485c303aeb09deb4b0e673214b30507a271521ff3558027d\": container with ID starting with 25e0d0813243a12b485c303aeb09deb4b0e673214b30507a271521ff3558027d not found: ID does not exist" containerID="25e0d0813243a12b485c303aeb09deb4b0e673214b30507a271521ff3558027d" Oct 30 20:01:21.104187 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.104162 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e0d0813243a12b485c303aeb09deb4b0e673214b30507a271521ff3558027d"} err="failed to get container status \"25e0d0813243a12b485c303aeb09deb4b0e673214b30507a271521ff3558027d\": rpc error: code = NotFound desc = could not find container \"25e0d0813243a12b485c303aeb09deb4b0e673214b30507a271521ff3558027d\": container with ID starting with 25e0d0813243a12b485c303aeb09deb4b0e673214b30507a271521ff3558027d not found: ID does not exist" Oct 30 20:01:21.126651 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.126624 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 20:01:21.135252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.135184 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 20:01:21.216170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.208625 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:21.255810 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.255700 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:21.322718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.322650 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" podStartSLOduration=3.322630378 podStartE2EDuration="3.322630378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:21.082513603 +0000 UTC m=+1800.777721574" watchObservedRunningTime="2026-10-30 20:01:21.322630378 +0000 UTC m=+1801.017838337" Oct 30 20:01:21.338588 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.331271 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:21.493406 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.493357 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv"] Oct 30 20:01:21.500493 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.500469 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" Oct 30 20:01:21.507677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.507647 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv"] Oct 30 20:01:21.518461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.518438 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:01:21.518583 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.518491 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:01:21.585168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.585119 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r"] Oct 30 20:01:21.595568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.590236 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:01:21.595568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.590317 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:01:21.595568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.590362 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:01:21.615315 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.615287 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef30b5ec-0384-47c7-9251-6763ed94fe90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05290867-64ttv\" (UID: \"ef30b5ec-0384-47c7-9251-6763ed94fe90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" Oct 30 20:01:21.615470 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.615343 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6l5w\" (UniqueName: \"kubernetes.io/projected/ef30b5ec-0384-47c7-9251-6763ed94fe90-kube-api-access-q6l5w\") pod \"maas-api-revoke-sub-05290867-64ttv\" (UID: \"ef30b5ec-0384-47c7-9251-6763ed94fe90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" Oct 30 20:01:21.615470 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.615421 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef30b5ec-0384-47c7-9251-6763ed94fe90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05290867-64ttv\" (UID: \"ef30b5ec-0384-47c7-9251-6763ed94fe90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" Oct 30 20:01:21.716600 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.716574 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef30b5ec-0384-47c7-9251-6763ed94fe90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05290867-64ttv\" (UID: \"ef30b5ec-0384-47c7-9251-6763ed94fe90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" Oct 30 20:01:21.716715 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.716623 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6l5w\" (UniqueName: \"kubernetes.io/projected/ef30b5ec-0384-47c7-9251-6763ed94fe90-kube-api-access-q6l5w\") pod \"maas-api-revoke-sub-05290867-64ttv\" (UID: \"ef30b5ec-0384-47c7-9251-6763ed94fe90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" Oct 30 20:01:21.716715 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.716669 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef30b5ec-0384-47c7-9251-6763ed94fe90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05290867-64ttv\" (UID: \"ef30b5ec-0384-47c7-9251-6763ed94fe90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" Oct 30 20:01:21.717268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.717245 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef30b5ec-0384-47c7-9251-6763ed94fe90-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05290867-64ttv\" (UID: \"ef30b5ec-0384-47c7-9251-6763ed94fe90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" Oct 30 20:01:21.718880 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.718860 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef30b5ec-0384-47c7-9251-6763ed94fe90-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05290867-64ttv\" (UID: \"ef30b5ec-0384-47c7-9251-6763ed94fe90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" Oct 30 20:01:21.738446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.738420 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6l5w\" (UniqueName: \"kubernetes.io/projected/ef30b5ec-0384-47c7-9251-6763ed94fe90-kube-api-access-q6l5w\") pod \"maas-api-revoke-sub-05290867-64ttv\" (UID: \"ef30b5ec-0384-47c7-9251-6763ed94fe90\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" Oct 30 20:01:21.818179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.818128 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" Oct 30 20:01:21.898871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:21.895106 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:22.022776 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.022713 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" event={"ID":"2bcb265a-c599-404a-b848-b62b6e632f2a","Type":"ContainerStarted","Data":"b6ee1b25d608772915087d0d56222d6449e4450181ecb3053876f93e3f314c99"} Oct 30 20:01:22.022993 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.022792 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" event={"ID":"2bcb265a-c599-404a-b848-b62b6e632f2a","Type":"ContainerStarted","Data":"a1cf944f327bee06f80248a99ba607a98f109a77c4acd76a47d9dd8bac2fcc79"} Oct 30 20:01:22.033170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.030204 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:22.033170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.032788 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:01:22.040039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.038729 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" podUID="6794b918-54df-4217-b67a-26fec2eb5646" containerName="main" containerID="cri-o://9be74be70a9602855343bbfaf9dc95b9e04f7f3afb85e043c1c339048abbc4b0" gracePeriod=45 Oct 30 20:01:22.134608 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.087782 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:22.134608 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.097572 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 20:01:22.134608 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.097627 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 20:01:22.202211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.135446 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" podStartSLOduration=2.135423263 podStartE2EDuration="2.135423263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:22.084527159 +0000 UTC m=+1801.779735139" watchObservedRunningTime="2026-10-30 20:01:22.135423263 +0000 UTC m=+1801.830631222" Oct 30 20:01:22.539073 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.538374 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv"] Oct 30 20:01:22.587202 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.583306 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" podUID="6794b918-54df-4217-b67a-26fec2eb5646" containerName="main" probeResult="failure" output="Get \"https://10.132.0.225:8000/ready\": dial tcp 10.132.0.225:8000: connect: connection refused" Oct 30 20:01:22.801903 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.801877 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 20:01:22.892902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.892858 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6794b918-54df-4217-b67a-26fec2eb5646-kube-api-access-f6ds7\" (UniqueName: \"kubernetes.io/projected/6794b918-54df-4217-b67a-26fec2eb5646-kube-api-access-f6ds7\") pod \"6794b918-54df-4217-b67a-26fec2eb5646\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " Oct 30 20:01:22.893127 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.892930 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-tmp-dir\") pod \"6794b918-54df-4217-b67a-26fec2eb5646\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " Oct 30 20:01:22.893127 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.892975 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6794b918-54df-4217-b67a-26fec2eb5646-tls-certs\" (UniqueName: \"kubernetes.io/secret/6794b918-54df-4217-b67a-26fec2eb5646-tls-certs\") pod \"6794b918-54df-4217-b67a-26fec2eb5646\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " Oct 30 20:01:22.893331 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.893240 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-home\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-home\") pod \"6794b918-54df-4217-b67a-26fec2eb5646\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " Oct 30 20:01:22.893424 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.893400 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-model-cache\") pod \"6794b918-54df-4217-b67a-26fec2eb5646\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " Oct 30 20:01:22.893493 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.893477 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-dshm\") pod \"6794b918-54df-4217-b67a-26fec2eb5646\" (UID: \"6794b918-54df-4217-b67a-26fec2eb5646\") " Oct 30 20:01:22.895505 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.895474 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-model-cache" pod "6794b918-54df-4217-b67a-26fec2eb5646" (UID: "6794b918-54df-4217-b67a-26fec2eb5646"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:22.895644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.895591 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-tmp-dir" pod "6794b918-54df-4217-b67a-26fec2eb5646" (UID: "6794b918-54df-4217-b67a-26fec2eb5646"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:22.895714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.895653 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-home" pod "6794b918-54df-4217-b67a-26fec2eb5646" (UID: "6794b918-54df-4217-b67a-26fec2eb5646"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:22.897863 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.897837 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-dshm" pod "6794b918-54df-4217-b67a-26fec2eb5646" (UID: "6794b918-54df-4217-b67a-26fec2eb5646"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:01:22.898043 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.898022 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6794b918-54df-4217-b67a-26fec2eb5646-tls-certs" pod "6794b918-54df-4217-b67a-26fec2eb5646" (UID: "6794b918-54df-4217-b67a-26fec2eb5646"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:22.898136 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.898081 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6794b918-54df-4217-b67a-26fec2eb5646-kube-api-access-f6ds7" pod "6794b918-54df-4217-b67a-26fec2eb5646" (UID: "6794b918-54df-4217-b67a-26fec2eb5646"). InnerVolumeSpecName "kube-api-access-f6ds7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:22.995999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.995956 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6ds7\" (UniqueName: \"kubernetes.io/projected/6794b918-54df-4217-b67a-26fec2eb5646-kube-api-access-f6ds7\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:22.995999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.996002 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-tmp-dir\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:22.996597 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.996017 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6794b918-54df-4217-b67a-26fec2eb5646-tls-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:22.996597 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.996031 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-home\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:22.996597 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.996047 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-model-cache\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:22.996597 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:22.996061 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6794b918-54df-4217-b67a-26fec2eb5646-dshm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:23.009585 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.009560 5091 scope.go:122] "RemoveContainer" containerID="ae2e80493e7836f9c9caf4637eebb1462e3106a21f82bd1aeda9ff3c38394ec5" Oct 30 20:01:23.023263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.023243 5091 scope.go:122] "RemoveContainer" containerID="cbced8b0b125a518e47944f60d9fb1e46c281d0ff4bcc223b81180ce2eed4607" Oct 30 20:01:23.036140 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.036118 5091 scope.go:122] "RemoveContainer" containerID="39715085beaccacf3ad731be63b51feb88fd46c1ddc139f4a4e9c23b2eead8a2" Oct 30 20:01:23.086172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.072990 5091 generic.go:356] "Generic (PLEG): container finished" podID="6794b918-54df-4217-b67a-26fec2eb5646" containerID="9be74be70a9602855343bbfaf9dc95b9e04f7f3afb85e043c1c339048abbc4b0" exitCode=0 Oct 30 20:01:23.086172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.073195 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" event={"ID":"6794b918-54df-4217-b67a-26fec2eb5646","Type":"ContainerDied","Data":"9be74be70a9602855343bbfaf9dc95b9e04f7f3afb85e043c1c339048abbc4b0"} Oct 30 20:01:23.086172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.073243 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" event={"ID":"6794b918-54df-4217-b67a-26fec2eb5646","Type":"ContainerDied","Data":"390b2a773a489283705a0cf36a15dee406cb86d5773591afe820f115e10593e4"} Oct 30 20:01:23.086172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.073267 5091 scope.go:122] "RemoveContainer" containerID="9be74be70a9602855343bbfaf9dc95b9e04f7f3afb85e043c1c339048abbc4b0" Oct 30 20:01:23.086172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.073400 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt" Oct 30 20:01:23.142775 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.131365 5091 scope.go:122] "RemoveContainer" containerID="81e65f592713f2edbbc648860ca8835e249e3f376692a8f07534da79262b4132" Oct 30 20:01:23.142775 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.132299 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:23.161029 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.158063 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" event={"ID":"ef30b5ec-0384-47c7-9251-6763ed94fe90","Type":"ContainerStarted","Data":"31faeac38941e593f1646adff7558374fb4d90620875e3c8bc75a52a999d69e8"} Oct 30 20:01:23.161029 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.158136 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:23.183171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.174426 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:01:23.183171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.182598 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:23.212921 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.212897 5091 scope.go:122] "RemoveContainer" containerID="9be74be70a9602855343bbfaf9dc95b9e04f7f3afb85e043c1c339048abbc4b0" Oct 30 20:01:23.213318 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:01:23.213297 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be74be70a9602855343bbfaf9dc95b9e04f7f3afb85e043c1c339048abbc4b0\": container with ID starting with 9be74be70a9602855343bbfaf9dc95b9e04f7f3afb85e043c1c339048abbc4b0 not found: ID does not exist" containerID="9be74be70a9602855343bbfaf9dc95b9e04f7f3afb85e043c1c339048abbc4b0" Oct 30 20:01:23.213460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.213433 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be74be70a9602855343bbfaf9dc95b9e04f7f3afb85e043c1c339048abbc4b0"} err="failed to get container status \"9be74be70a9602855343bbfaf9dc95b9e04f7f3afb85e043c1c339048abbc4b0\": rpc error: code = NotFound desc = could not find container \"9be74be70a9602855343bbfaf9dc95b9e04f7f3afb85e043c1c339048abbc4b0\": container with ID starting with 9be74be70a9602855343bbfaf9dc95b9e04f7f3afb85e043c1c339048abbc4b0 not found: ID does not exist" Oct 30 20:01:23.214586 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.214563 5091 scope.go:122] "RemoveContainer" containerID="92a7a3b781d8ea84a2d94f68638197296f11f4b34a179d07e40f7831c317b91d" Oct 30 20:01:23.226014 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.225990 5091 scope.go:122] "RemoveContainer" containerID="522d41964eba19a08c8c8537cd19f8fbe0007793ba90db9c71535647bb4f0c20" Oct 30 20:01:23.236953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.236932 5091 scope.go:122] "RemoveContainer" containerID="f3e5b8531ea2df0b8f22f148d17cc1027d3d1e7802bed3ea0058cceb99caaf03" Oct 30 20:01:23.275309 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.270425 5091 scope.go:122] "RemoveContainer" containerID="de54bd877557292d6cead3ca1e00b99563746167da6124c1e137ba10d9c91fc2" Oct 30 20:01:23.335086 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.334986 5091 scope.go:122] "RemoveContainer" containerID="38379709aa8c13c2a6832ad79935fcf577456cbbf2946d4644f1109d53459b2d" Oct 30 20:01:23.335086 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.335038 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt"] Oct 30 20:01:23.361171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.356908 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-779fa3/auth-test-model-779fa3-kserve-6cb59d65b5-kbcgt"] Oct 30 20:01:23.403850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.402510 5091 scope.go:122] "RemoveContainer" containerID="47360f08022999c3947ac261b80b1fc0968ae0e1668787703aaa5ea23c641fbd" Oct 30 20:01:23.417017 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.416997 5091 scope.go:122] "RemoveContainer" containerID="65347e8ce1b71b107934c01bc5ea0dc0fd0313852cb970bc87073e87d87ce10e" Oct 30 20:01:23.712200 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.712157 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:23.943708 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:23.943668 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:24.063932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:24.063899 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp"] Oct 30 20:01:24.080576 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:24.080546 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c4f759c-hnksp"] Oct 30 20:01:24.169507 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:24.169419 5091 generic.go:356] "Generic (PLEG): container finished" podID="ef30b5ec-0384-47c7-9251-6763ed94fe90" containerID="48220fe0de7b3e7984998a5e12b99b704bd0fff4625db00aa18bc67e3f2d85a3" exitCode=0 Oct 30 20:01:24.169668 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:24.169497 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" event={"ID":"ef30b5ec-0384-47c7-9251-6763ed94fe90","Type":"ContainerDied","Data":"48220fe0de7b3e7984998a5e12b99b704bd0fff4625db00aa18bc67e3f2d85a3"} Oct 30 20:01:24.444786 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:24.444713 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6794b918-54df-4217-b67a-26fec2eb5646" path="/var/lib/kubelet/pods/6794b918-54df-4217-b67a-26fec2eb5646/volumes" Oct 30 20:01:24.445289 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:24.445271 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e917c4e9-9a21-48fa-b866-291661ad2b22" path="/var/lib/kubelet/pods/e917c4e9-9a21-48fa-b866-291661ad2b22/volumes" Oct 30 20:01:24.612559 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:24.612524 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:24.818913 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:24.818868 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:25.110138 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:25.110054 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:25.496219 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:25.495484 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:25.704387 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:25.704354 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" Oct 30 20:01:25.751663 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:25.751453 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef30b5ec-0384-47c7-9251-6763ed94fe90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef30b5ec-0384-47c7-9251-6763ed94fe90-maas-api-xks-ca\") pod \"ef30b5ec-0384-47c7-9251-6763ed94fe90\" (UID: \"ef30b5ec-0384-47c7-9251-6763ed94fe90\") " Oct 30 20:01:25.751663 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:25.751586 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef30b5ec-0384-47c7-9251-6763ed94fe90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef30b5ec-0384-47c7-9251-6763ed94fe90-maas-api-openshift-ca\") pod \"ef30b5ec-0384-47c7-9251-6763ed94fe90\" (UID: \"ef30b5ec-0384-47c7-9251-6763ed94fe90\") " Oct 30 20:01:25.751895 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:25.751666 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef30b5ec-0384-47c7-9251-6763ed94fe90-kube-api-access-q6l5w\" (UniqueName: \"kubernetes.io/projected/ef30b5ec-0384-47c7-9251-6763ed94fe90-kube-api-access-q6l5w\") pod \"ef30b5ec-0384-47c7-9251-6763ed94fe90\" (UID: \"ef30b5ec-0384-47c7-9251-6763ed94fe90\") " Oct 30 20:01:25.754238 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:25.754209 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef30b5ec-0384-47c7-9251-6763ed94fe90-maas-api-openshift-ca" pod "ef30b5ec-0384-47c7-9251-6763ed94fe90" (UID: "ef30b5ec-0384-47c7-9251-6763ed94fe90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:25.772196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:25.763621 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef30b5ec-0384-47c7-9251-6763ed94fe90-kube-api-access-q6l5w" pod "ef30b5ec-0384-47c7-9251-6763ed94fe90" (UID: "ef30b5ec-0384-47c7-9251-6763ed94fe90"). InnerVolumeSpecName "kube-api-access-q6l5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:25.772196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:25.763980 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef30b5ec-0384-47c7-9251-6763ed94fe90-maas-api-xks-ca" pod "ef30b5ec-0384-47c7-9251-6763ed94fe90" (UID: "ef30b5ec-0384-47c7-9251-6763ed94fe90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:25.854176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:25.854126 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef30b5ec-0384-47c7-9251-6763ed94fe90-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:25.854176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:25.854177 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6l5w\" (UniqueName: \"kubernetes.io/projected/ef30b5ec-0384-47c7-9251-6763ed94fe90-kube-api-access-q6l5w\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:25.854403 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:25.854194 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef30b5ec-0384-47c7-9251-6763ed94fe90-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:26.051230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.046270 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:26.196509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.195574 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" Oct 30 20:01:26.196509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.195604 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv" event={"ID":"ef30b5ec-0384-47c7-9251-6763ed94fe90","Type":"ContainerDied","Data":"31faeac38941e593f1646adff7558374fb4d90620875e3c8bc75a52a999d69e8"} Oct 30 20:01:26.196509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.195654 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31faeac38941e593f1646adff7558374fb4d90620875e3c8bc75a52a999d69e8" Oct 30 20:01:26.295051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.295013 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:26.412794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.412716 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7"] Oct 30 20:01:26.413485 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.413468 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ef30b5ec-0384-47c7-9251-6763ed94fe90" containerName="revoke-keys" Oct 30 20:01:26.413551 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.413486 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef30b5ec-0384-47c7-9251-6763ed94fe90" containerName="revoke-keys" Oct 30 20:01:26.413551 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.413543 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6794b918-54df-4217-b67a-26fec2eb5646" containerName="main" Oct 30 20:01:26.413551 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.413549 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6794b918-54df-4217-b67a-26fec2eb5646" containerName="main" Oct 30 20:01:26.413691 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.413681 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6794b918-54df-4217-b67a-26fec2eb5646" containerName="main" Oct 30 20:01:26.413725 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.413700 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef30b5ec-0384-47c7-9251-6763ed94fe90" containerName="revoke-keys" Oct 30 20:01:26.417735 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.417714 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" Oct 30 20:01:26.448801 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.448673 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-75b361\"" Oct 30 20:01:26.507289 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.449403 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-75b361-dockercfg-qq5xw\"" Oct 30 20:01:26.507540 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.475180 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z8d9\" (UniqueName: \"kubernetes.io/projected/432c8f27-589b-4be4-aac3-e37148e70ef9-kube-api-access-5z8d9\") pod \"payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7\" (UID: \"432c8f27-589b-4be4-aac3-e37148e70ef9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" Oct 30 20:01:26.508065 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.508039 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432c8f27-589b-4be4-aac3-e37148e70ef9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7\" (UID: \"432c8f27-589b-4be4-aac3-e37148e70ef9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" Oct 30 20:01:26.558260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.543754 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7"] Oct 30 20:01:26.634264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.618071 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:26.663593 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.663501 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432c8f27-589b-4be4-aac3-e37148e70ef9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7\" (UID: \"432c8f27-589b-4be4-aac3-e37148e70ef9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" Oct 30 20:01:26.703946 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.703906 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z8d9\" (UniqueName: \"kubernetes.io/projected/432c8f27-589b-4be4-aac3-e37148e70ef9-kube-api-access-5z8d9\") pod \"payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7\" (UID: \"432c8f27-589b-4be4-aac3-e37148e70ef9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" Oct 30 20:01:26.704708 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.676862 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432c8f27-589b-4be4-aac3-e37148e70ef9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7\" (UID: \"432c8f27-589b-4be4-aac3-e37148e70ef9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" Oct 30 20:01:26.735167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.732168 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z8d9\" (UniqueName: \"kubernetes.io/projected/432c8f27-589b-4be4-aac3-e37148e70ef9-kube-api-access-5z8d9\") pod \"payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7\" (UID: \"432c8f27-589b-4be4-aac3-e37148e70ef9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" Oct 30 20:01:26.750065 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:26.746413 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" Oct 30 20:01:27.100241 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:01:27.100206 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432c8f27_589b_4be4_aac3_e37148e70ef9.slice/crio-dce2e30e687e4eb20b773b54ef60b03f45eb55479f3cb584ebedb584eabb6cfe WatchSource:0}: Error finding container dce2e30e687e4eb20b773b54ef60b03f45eb55479f3cb584ebedb584eabb6cfe: Status 404 returned error can't find the container with id dce2e30e687e4eb20b773b54ef60b03f45eb55479f3cb584ebedb584eabb6cfe Oct 30 20:01:27.107973 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:27.104581 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7"] Oct 30 20:01:27.233046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:27.230924 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" event={"ID":"432c8f27-589b-4be4-aac3-e37148e70ef9","Type":"ContainerStarted","Data":"dce2e30e687e4eb20b773b54ef60b03f45eb55479f3cb584ebedb584eabb6cfe"} Oct 30 20:01:27.326858 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:27.326822 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:28.260648 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:28.260589 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" event={"ID":"432c8f27-589b-4be4-aac3-e37148e70ef9","Type":"ContainerStarted","Data":"fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4"} Oct 30 20:01:28.261513 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:28.261480 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" Oct 30 20:01:28.289376 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:28.289315 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" podStartSLOduration=2.289298353 podStartE2EDuration="2.289298353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:28.286233851 +0000 UTC m=+1807.981441826" watchObservedRunningTime="2026-10-30 20:01:28.289298353 +0000 UTC m=+1807.984506312" Oct 30 20:01:28.475523 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:28.430420 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:01:28.475523 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:28.430557 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" podUID="ec317404-f9f4-499c-bdbe-65109b6a18d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:01:28.485682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:28.485653 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:29.272553 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:29.272512 5091 generic.go:356] "Generic (PLEG): container finished" podID="ec317404-f9f4-499c-bdbe-65109b6a18d2" containerID="c6e5709981ad9b33dfea33bf266708bda762034c9a1072c405e98bd7d0a21d90" exitCode=0 Oct 30 20:01:29.273042 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:29.272569 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" event={"ID":"ec317404-f9f4-499c-bdbe-65109b6a18d2","Type":"ContainerDied","Data":"c6e5709981ad9b33dfea33bf266708bda762034c9a1072c405e98bd7d0a21d90"} Oct 30 20:01:29.273355 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:29.273335 5091 scope.go:122] "RemoveContainer" containerID="c6e5709981ad9b33dfea33bf266708bda762034c9a1072c405e98bd7d0a21d90" Oct 30 20:01:29.275187 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:29.275139 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:01:29.275313 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:29.275226 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:01:29.275313 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:29.275256 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:01:29.531214 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:29.513806 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:30.291619 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:30.291581 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" event={"ID":"ec317404-f9f4-499c-bdbe-65109b6a18d2","Type":"ContainerStarted","Data":"4012fdf9bce4045b0905513abedc184e3277bd538e5ff2dbcb8fa38a21195e76"} Oct 30 20:01:30.499566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:30.499523 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:31.063169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.061543 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:01:31.063169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.061701 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 20:01:31.063169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.061747 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" podUID="ec317404-f9f4-499c-bdbe-65109b6a18d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 20:01:31.381591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.375273 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:31.798134 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.798098 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t"] Oct 30 20:01:31.802787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.802768 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" Oct 30 20:01:31.809988 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.809961 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:01:31.810131 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.810053 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:01:31.813231 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.813048 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t"] Oct 30 20:01:31.837869 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.837838 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pggcj\" (UniqueName: \"kubernetes.io/projected/be352a3e-b001-4260-b466-fe15f3d5037d-kube-api-access-pggcj\") pod \"maas-api-revoke-sub-02cca21c-v9z8t\" (UID: \"be352a3e-b001-4260-b466-fe15f3d5037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" Oct 30 20:01:31.837997 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.837909 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be352a3e-b001-4260-b466-fe15f3d5037d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02cca21c-v9z8t\" (UID: \"be352a3e-b001-4260-b466-fe15f3d5037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" Oct 30 20:01:31.838111 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.838081 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be352a3e-b001-4260-b466-fe15f3d5037d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02cca21c-v9z8t\" (UID: \"be352a3e-b001-4260-b466-fe15f3d5037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" Oct 30 20:01:31.939389 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.939354 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be352a3e-b001-4260-b466-fe15f3d5037d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02cca21c-v9z8t\" (UID: \"be352a3e-b001-4260-b466-fe15f3d5037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" Oct 30 20:01:31.939542 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.939496 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be352a3e-b001-4260-b466-fe15f3d5037d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02cca21c-v9z8t\" (UID: \"be352a3e-b001-4260-b466-fe15f3d5037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" Oct 30 20:01:31.939691 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.939674 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pggcj\" (UniqueName: \"kubernetes.io/projected/be352a3e-b001-4260-b466-fe15f3d5037d-kube-api-access-pggcj\") pod \"maas-api-revoke-sub-02cca21c-v9z8t\" (UID: \"be352a3e-b001-4260-b466-fe15f3d5037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" Oct 30 20:01:31.940205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.940180 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be352a3e-b001-4260-b466-fe15f3d5037d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-02cca21c-v9z8t\" (UID: \"be352a3e-b001-4260-b466-fe15f3d5037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" Oct 30 20:01:31.941926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.941902 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be352a3e-b001-4260-b466-fe15f3d5037d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-02cca21c-v9z8t\" (UID: \"be352a3e-b001-4260-b466-fe15f3d5037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" Oct 30 20:01:31.978207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:31.977636 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pggcj\" (UniqueName: \"kubernetes.io/projected/be352a3e-b001-4260-b466-fe15f3d5037d-kube-api-access-pggcj\") pod \"maas-api-revoke-sub-02cca21c-v9z8t\" (UID: \"be352a3e-b001-4260-b466-fe15f3d5037d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" Oct 30 20:01:32.075224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:32.073632 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 20:01:32.075224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:32.073721 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" podUID="ec317404-f9f4-499c-bdbe-65109b6a18d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 20:01:32.120470 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:32.120438 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" Oct 30 20:01:32.296213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:32.291652 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:32.457168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:32.445749 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t"] Oct 30 20:01:33.064179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:33.063462 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 20:01:33.064179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:33.063521 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" podUID="ec317404-f9f4-499c-bdbe-65109b6a18d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 20:01:33.259217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:33.259181 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:33.409947 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:33.409908 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" event={"ID":"be352a3e-b001-4260-b466-fe15f3d5037d","Type":"ContainerStarted","Data":"516cc1f28b5d90936a314734534a3ee142e42cb68311d0b5f099736597e2ecd4"} Oct 30 20:01:33.410106 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:33.409965 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" event={"ID":"be352a3e-b001-4260-b466-fe15f3d5037d","Type":"ContainerStarted","Data":"35b9f7dcc05bc5f85ffddb77e42beb4e807954dc7cef30e2c59a80b04f18db83"} Oct 30 20:01:33.433672 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:33.433624 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" podStartSLOduration=2.433612206 podStartE2EDuration="2.433612206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:33.432875077 +0000 UTC m=+1813.128083035" watchObservedRunningTime="2026-10-30 20:01:33.433612206 +0000 UTC m=+1813.128820193" Oct 30 20:01:34.065789 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:34.065753 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 20:01:34.066423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:34.066385 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" podUID="ec317404-f9f4-499c-bdbe-65109b6a18d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 20:01:34.222971 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:34.222923 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:34.419563 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:34.419471 5091 generic.go:356] "Generic (PLEG): container finished" podID="be352a3e-b001-4260-b466-fe15f3d5037d" containerID="516cc1f28b5d90936a314734534a3ee142e42cb68311d0b5f099736597e2ecd4" exitCode=0 Oct 30 20:01:34.419730 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:34.419563 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" event={"ID":"be352a3e-b001-4260-b466-fe15f3d5037d","Type":"ContainerDied","Data":"516cc1f28b5d90936a314734534a3ee142e42cb68311d0b5f099736597e2ecd4"} Oct 30 20:01:35.070291 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:35.070252 5091 patch_prober.go:29] interesting pod/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Oct 30 20:01:35.070883 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:35.070844 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" podUID="ec317404-f9f4-499c-bdbe-65109b6a18d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Oct 30 20:01:35.257170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:35.254785 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:35.638165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:35.638125 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" Oct 30 20:01:35.645879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:35.645848 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be352a3e-b001-4260-b466-fe15f3d5037d-kube-api-access-pggcj\" (UniqueName: \"kubernetes.io/projected/be352a3e-b001-4260-b466-fe15f3d5037d-kube-api-access-pggcj\") pod \"be352a3e-b001-4260-b466-fe15f3d5037d\" (UID: \"be352a3e-b001-4260-b466-fe15f3d5037d\") " Oct 30 20:01:35.646028 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:35.645993 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be352a3e-b001-4260-b466-fe15f3d5037d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be352a3e-b001-4260-b466-fe15f3d5037d-maas-api-openshift-ca\") pod \"be352a3e-b001-4260-b466-fe15f3d5037d\" (UID: \"be352a3e-b001-4260-b466-fe15f3d5037d\") " Oct 30 20:01:35.646105 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:35.646032 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be352a3e-b001-4260-b466-fe15f3d5037d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be352a3e-b001-4260-b466-fe15f3d5037d-maas-api-xks-ca\") pod \"be352a3e-b001-4260-b466-fe15f3d5037d\" (UID: \"be352a3e-b001-4260-b466-fe15f3d5037d\") " Oct 30 20:01:35.646514 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:35.646444 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be352a3e-b001-4260-b466-fe15f3d5037d-maas-api-openshift-ca" pod "be352a3e-b001-4260-b466-fe15f3d5037d" (UID: "be352a3e-b001-4260-b466-fe15f3d5037d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:35.646694 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:35.646675 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be352a3e-b001-4260-b466-fe15f3d5037d-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:35.648714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:35.648688 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be352a3e-b001-4260-b466-fe15f3d5037d-kube-api-access-pggcj" pod "be352a3e-b001-4260-b466-fe15f3d5037d" (UID: "be352a3e-b001-4260-b466-fe15f3d5037d"). InnerVolumeSpecName "kube-api-access-pggcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:35.649287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:35.649255 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be352a3e-b001-4260-b466-fe15f3d5037d-maas-api-xks-ca" pod "be352a3e-b001-4260-b466-fe15f3d5037d" (UID: "be352a3e-b001-4260-b466-fe15f3d5037d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:35.748596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:35.748557 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be352a3e-b001-4260-b466-fe15f3d5037d-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:35.748596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:35.748588 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pggcj\" (UniqueName: \"kubernetes.io/projected/be352a3e-b001-4260-b466-fe15f3d5037d-kube-api-access-pggcj\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:36.137444 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:36.134623 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:01:36.137444 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:36.136261 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:01:36.147169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:36.142195 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:36.147169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:36.144312 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:01:36.440399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:36.440322 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" Oct 30 20:01:36.451165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:36.443195 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t" event={"ID":"be352a3e-b001-4260-b466-fe15f3d5037d","Type":"ContainerDied","Data":"35b9f7dcc05bc5f85ffddb77e42beb4e807954dc7cef30e2c59a80b04f18db83"} Oct 30 20:01:36.451165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:36.443250 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b9f7dcc05bc5f85ffddb77e42beb4e807954dc7cef30e2c59a80b04f18db83" Oct 30 20:01:37.033699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:37.033658 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5"] Oct 30 20:01:37.038039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:37.038010 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8ee4d261-m86d5"] Oct 30 20:01:37.087121 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:37.087082 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:37.972325 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:37.972290 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:38.442935 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:38.442905 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a" path="/var/lib/kubelet/pods/9d0a8048-1a7a-49a3-b7c0-4e7d6d8c4e5a/volumes" Oct 30 20:01:38.942201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:38.942162 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:39.168050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:39.168011 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:39.276158 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:39.276116 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" Oct 30 20:01:40.019894 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:40.019852 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:41.104292 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:41.104251 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:41.266737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:41.266699 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:42.177173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:42.175164 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:43.086172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:43.081781 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:43.922011 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:43.921972 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:44.105167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:44.093787 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:44.987169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:44.986408 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:46.003708 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:01:46.003579 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:01:46.018293 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:46.018089 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:46.416550 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:46.416467 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg"] Oct 30 20:01:46.505801 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:46.505756 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:47.264715 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:47.264667 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:48.038747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.038710 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:48.291396 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.291299 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:48.431797 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.431742 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp"] Oct 30 20:01:48.433423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.433397 5091 cpu_manager.go:414] "containerMap: removing container" podUID="be352a3e-b001-4260-b466-fe15f3d5037d" containerName="revoke-keys" Oct 30 20:01:48.433423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.433423 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be352a3e-b001-4260-b466-fe15f3d5037d" containerName="revoke-keys" Oct 30 20:01:48.433717 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.433698 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="be352a3e-b001-4260-b466-fe15f3d5037d" containerName="revoke-keys" Oct 30 20:01:48.439239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.438435 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.461467 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.461442 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-8c8241-openshift-default-dockercfg-mjvbn\"" Oct 30 20:01:48.461689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.461662 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp"] Oct 30 20:01:48.596604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.590369 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-credential-socket\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.596604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.590432 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e11004-d729-4eec-8626-28eba4b01e90-istiod-ca-cert\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.596604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.590542 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxp8\" (UniqueName: \"kubernetes.io/projected/18e11004-d729-4eec-8626-28eba4b01e90-kube-api-access-tlxp8\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.596604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.590581 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-istio-envoy\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.596604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.590605 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-istio-data\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.596604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.590716 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e11004-d729-4eec-8626-28eba4b01e90-istio-token\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.596604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.590821 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-workload-socket\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.596604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.590886 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-workload-certs\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.596604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.590977 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e11004-d729-4eec-8626-28eba4b01e90-istio-podinfo\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.646166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.630822 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:48.693179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.692184 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e11004-d729-4eec-8626-28eba4b01e90-istio-token\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.693179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.692311 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-workload-socket\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.693179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.692376 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-workload-certs\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.693179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.692503 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e11004-d729-4eec-8626-28eba4b01e90-istio-podinfo\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.693179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.692612 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-credential-socket\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.693179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.692649 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e11004-d729-4eec-8626-28eba4b01e90-istiod-ca-cert\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.693751 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.693338 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-workload-certs\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.747669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.705394 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxp8\" (UniqueName: \"kubernetes.io/projected/18e11004-d729-4eec-8626-28eba4b01e90-kube-api-access-tlxp8\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.747669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.705451 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-istio-envoy\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.747669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.705476 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-istio-data\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.747669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.705916 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-istio-data\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.747669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.706504 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-workload-socket\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.747669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.707548 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e11004-d729-4eec-8626-28eba4b01e90-istiod-ca-cert\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.747669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.707782 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-credential-socket\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.747669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.726982 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e11004-d729-4eec-8626-28eba4b01e90-istio-podinfo\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.747669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.741200 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-istio-envoy\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.763180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.754352 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxp8\" (UniqueName: \"kubernetes.io/projected/18e11004-d729-4eec-8626-28eba4b01e90-kube-api-access-tlxp8\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.763180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.754746 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e11004-d729-4eec-8626-28eba4b01e90-istio-token\") pod \"e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.764010 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.763766 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:48.934336 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:48.926125 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:49.307170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:49.306840 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp"] Oct 30 20:01:49.365170 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:01:49.363227 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e11004_d729_4eec_8626_28eba4b01e90.slice/crio-fd51b63d7390c04a6480d04510e8bc8e50a2ea0d7a5308a649c6745d08f321d6 WatchSource:0}: Error finding container fd51b63d7390c04a6480d04510e8bc8e50a2ea0d7a5308a649c6745d08f321d6: Status 404 returned error can't find the container with id fd51b63d7390c04a6480d04510e8bc8e50a2ea0d7a5308a649c6745d08f321d6 Oct 30 20:01:49.369176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:49.366854 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:01:49.369176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:49.366948 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:01:49.369176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:49.366995 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:01:49.515670 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:49.515248 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:49.711435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:49.711401 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" event={"ID":"18e11004-d729-4eec-8626-28eba4b01e90","Type":"ContainerStarted","Data":"b832152b92057244bcc5e2d9da58874ecd2cee48a3954953a6a67e39f106d133"} Oct 30 20:01:49.711655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:49.711640 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" event={"ID":"18e11004-d729-4eec-8626-28eba4b01e90","Type":"ContainerStarted","Data":"fd51b63d7390c04a6480d04510e8bc8e50a2ea0d7a5308a649c6745d08f321d6"} Oct 30 20:01:49.746203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:49.746140 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" podStartSLOduration=1.746124994 podStartE2EDuration="1.746124994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:49.743226224 +0000 UTC m=+1829.438434185" watchObservedRunningTime="2026-10-30 20:01:49.746124994 +0000 UTC m=+1829.441332949" Oct 30 20:01:49.765406 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:49.765379 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:50.481170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:50.477672 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:50.771720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:50.771689 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:51.518701 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.518666 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj"] Oct 30 20:01:51.523519 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.523499 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" Oct 30 20:01:51.530174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.530125 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:01:51.530296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.530229 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:01:51.534481 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.534457 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj"] Oct 30 20:01:51.593781 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.593741 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a386a76c-8c8c-4de2-a5f4-86560053751f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f428cc16-zf7zj\" (UID: \"a386a76c-8c8c-4de2-a5f4-86560053751f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" Oct 30 20:01:51.593976 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.593801 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wdb5\" (UniqueName: \"kubernetes.io/projected/a386a76c-8c8c-4de2-a5f4-86560053751f-kube-api-access-7wdb5\") pod \"maas-api-revoke-sub-f428cc16-zf7zj\" (UID: \"a386a76c-8c8c-4de2-a5f4-86560053751f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" Oct 30 20:01:51.594049 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.594021 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a386a76c-8c8c-4de2-a5f4-86560053751f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f428cc16-zf7zj\" (UID: \"a386a76c-8c8c-4de2-a5f4-86560053751f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" Oct 30 20:01:51.720301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.720252 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a386a76c-8c8c-4de2-a5f4-86560053751f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f428cc16-zf7zj\" (UID: \"a386a76c-8c8c-4de2-a5f4-86560053751f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" Oct 30 20:01:51.720519 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.720319 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wdb5\" (UniqueName: \"kubernetes.io/projected/a386a76c-8c8c-4de2-a5f4-86560053751f-kube-api-access-7wdb5\") pod \"maas-api-revoke-sub-f428cc16-zf7zj\" (UID: \"a386a76c-8c8c-4de2-a5f4-86560053751f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" Oct 30 20:01:51.720590 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.720566 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a386a76c-8c8c-4de2-a5f4-86560053751f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f428cc16-zf7zj\" (UID: \"a386a76c-8c8c-4de2-a5f4-86560053751f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" Oct 30 20:01:51.722788 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.722729 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a386a76c-8c8c-4de2-a5f4-86560053751f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f428cc16-zf7zj\" (UID: \"a386a76c-8c8c-4de2-a5f4-86560053751f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" Oct 30 20:01:51.741845 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.741812 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wdb5\" (UniqueName: \"kubernetes.io/projected/a386a76c-8c8c-4de2-a5f4-86560053751f-kube-api-access-7wdb5\") pod \"maas-api-revoke-sub-f428cc16-zf7zj\" (UID: \"a386a76c-8c8c-4de2-a5f4-86560053751f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" Oct 30 20:01:51.744420 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.742756 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:51.745753 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.745445 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a386a76c-8c8c-4de2-a5f4-86560053751f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f428cc16-zf7zj\" (UID: \"a386a76c-8c8c-4de2-a5f4-86560053751f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" Oct 30 20:01:51.750498 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.750473 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:51.751823 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.751802 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:01:51.847074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:51.843273 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" Oct 30 20:01:52.169268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:52.169176 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:52.188038 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:01:52.187995 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda386a76c_8c8c_4de2_a5f4_86560053751f.slice/crio-2be0ea29c0705234f03caa61cd1174c22a72789681822f760033375642a36d29 WatchSource:0}: Error finding container 2be0ea29c0705234f03caa61cd1174c22a72789681822f760033375642a36d29: Status 404 returned error can't find the container with id 2be0ea29c0705234f03caa61cd1174c22a72789681822f760033375642a36d29 Oct 30 20:01:52.196066 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:52.196037 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj"] Oct 30 20:01:52.346697 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:52.346661 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:52.764967 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:52.764931 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" event={"ID":"a386a76c-8c8c-4de2-a5f4-86560053751f","Type":"ContainerStarted","Data":"63946c076c0b6bd62dbf063d506906cd8595712156d1271ac21c9de2c5789f89"} Oct 30 20:01:52.765454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:52.764994 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" event={"ID":"a386a76c-8c8c-4de2-a5f4-86560053751f","Type":"ContainerStarted","Data":"2be0ea29c0705234f03caa61cd1174c22a72789681822f760033375642a36d29"} Oct 30 20:01:52.822216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:52.812653 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" podStartSLOduration=1.81263064 podStartE2EDuration="1.81263064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:52.807595712 +0000 UTC m=+1832.502803672" watchObservedRunningTime="2026-10-30 20:01:52.81263064 +0000 UTC m=+1832.507838595" Oct 30 20:01:52.822216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:52.821997 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:52.937271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:52.937236 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7337-fbeb2b"] Oct 30 20:01:52.941939 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:52.941918 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-fbeb2b" Oct 30 20:01:52.953447 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:52.953423 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-fbeb2b"] Oct 30 20:01:52.960896 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:52.960868 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dqx9r\"" Oct 30 20:01:53.053370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:53.053327 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qncg\" (UniqueName: \"kubernetes.io/projected/24eae921-a70b-49f3-bf34-27608ba3ed22-kube-api-access-4qncg\") pod \"test-curl-7337-fbeb2b\" (UID: \"24eae921-a70b-49f3-bf34-27608ba3ed22\") " pod="opendatahub/test-curl-7337-fbeb2b" Oct 30 20:01:53.155373 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:53.155278 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qncg\" (UniqueName: \"kubernetes.io/projected/24eae921-a70b-49f3-bf34-27608ba3ed22-kube-api-access-4qncg\") pod \"test-curl-7337-fbeb2b\" (UID: \"24eae921-a70b-49f3-bf34-27608ba3ed22\") " pod="opendatahub/test-curl-7337-fbeb2b" Oct 30 20:01:53.172874 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:53.172836 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qncg\" (UniqueName: \"kubernetes.io/projected/24eae921-a70b-49f3-bf34-27608ba3ed22-kube-api-access-4qncg\") pod \"test-curl-7337-fbeb2b\" (UID: \"24eae921-a70b-49f3-bf34-27608ba3ed22\") " pod="opendatahub/test-curl-7337-fbeb2b" Oct 30 20:01:53.263260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:53.263230 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-fbeb2b" Oct 30 20:01:53.526075 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:53.526037 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:53.598089 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:01:53.598026 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24eae921_a70b_49f3_bf34_27608ba3ed22.slice/crio-ab5a984c89d41fbb833c831b660324e9e7d03c4de1ccebcaff5e524147585712 WatchSource:0}: Error finding container ab5a984c89d41fbb833c831b660324e9e7d03c4de1ccebcaff5e524147585712: Status 404 returned error can't find the container with id ab5a984c89d41fbb833c831b660324e9e7d03c4de1ccebcaff5e524147585712 Oct 30 20:01:53.598395 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:53.598352 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7337-fbeb2b"] Oct 30 20:01:53.819333 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:53.811544 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:54.028040 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:54.027918 5091 generic.go:356] "Generic (PLEG): container finished" podID="a386a76c-8c8c-4de2-a5f4-86560053751f" containerID="63946c076c0b6bd62dbf063d506906cd8595712156d1271ac21c9de2c5789f89" exitCode=0 Oct 30 20:01:54.028200 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:54.028097 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" event={"ID":"a386a76c-8c8c-4de2-a5f4-86560053751f","Type":"ContainerDied","Data":"63946c076c0b6bd62dbf063d506906cd8595712156d1271ac21c9de2c5789f89"} Oct 30 20:01:54.045076 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:54.045048 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-fbeb2b" event={"ID":"24eae921-a70b-49f3-bf34-27608ba3ed22","Type":"ContainerStarted","Data":"ab5a984c89d41fbb833c831b660324e9e7d03c4de1ccebcaff5e524147585712"} Oct 30 20:01:54.239285 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:54.239201 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:55.123244 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:55.123202 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7337-fbeb2b" event={"ID":"24eae921-a70b-49f3-bf34-27608ba3ed22","Type":"ContainerStarted","Data":"2a4c8df94161760f7bbdffd39917f862e6a3f8a1f8636ac6275783ff063f0290"} Oct 30 20:01:55.161041 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:55.160972 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7337-fbeb2b" podStartSLOduration=3.160952008 podStartE2EDuration="3.160952008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:55.153541003 +0000 UTC m=+1834.848748955" watchObservedRunningTime="2026-10-30 20:01:55.160952008 +0000 UTC m=+1834.856159966" Oct 30 20:01:55.327623 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:55.327596 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" Oct 30 20:01:55.500364 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:55.500325 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a386a76c-8c8c-4de2-a5f4-86560053751f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a386a76c-8c8c-4de2-a5f4-86560053751f-maas-api-openshift-ca\") pod \"a386a76c-8c8c-4de2-a5f4-86560053751f\" (UID: \"a386a76c-8c8c-4de2-a5f4-86560053751f\") " Oct 30 20:01:55.500676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:55.500652 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a386a76c-8c8c-4de2-a5f4-86560053751f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a386a76c-8c8c-4de2-a5f4-86560053751f-maas-api-xks-ca\") pod \"a386a76c-8c8c-4de2-a5f4-86560053751f\" (UID: \"a386a76c-8c8c-4de2-a5f4-86560053751f\") " Oct 30 20:01:55.500945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:55.500928 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a386a76c-8c8c-4de2-a5f4-86560053751f-kube-api-access-7wdb5\" (UniqueName: \"kubernetes.io/projected/a386a76c-8c8c-4de2-a5f4-86560053751f-kube-api-access-7wdb5\") pod \"a386a76c-8c8c-4de2-a5f4-86560053751f\" (UID: \"a386a76c-8c8c-4de2-a5f4-86560053751f\") " Oct 30 20:01:55.503608 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:55.503579 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a386a76c-8c8c-4de2-a5f4-86560053751f-maas-api-openshift-ca" pod "a386a76c-8c8c-4de2-a5f4-86560053751f" (UID: "a386a76c-8c8c-4de2-a5f4-86560053751f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:01:55.508838 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:55.508794 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a386a76c-8c8c-4de2-a5f4-86560053751f-maas-api-xks-ca" pod "a386a76c-8c8c-4de2-a5f4-86560053751f" (UID: "a386a76c-8c8c-4de2-a5f4-86560053751f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:01:55.543241 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:55.540525 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a386a76c-8c8c-4de2-a5f4-86560053751f-kube-api-access-7wdb5" pod "a386a76c-8c8c-4de2-a5f4-86560053751f" (UID: "a386a76c-8c8c-4de2-a5f4-86560053751f"). InnerVolumeSpecName "kube-api-access-7wdb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:55.543241 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:55.542919 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:55.604109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:55.603832 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a386a76c-8c8c-4de2-a5f4-86560053751f-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:55.604109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:55.603869 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a386a76c-8c8c-4de2-a5f4-86560053751f-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:55.604109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:55.603887 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wdb5\" (UniqueName: \"kubernetes.io/projected/a386a76c-8c8c-4de2-a5f4-86560053751f-kube-api-access-7wdb5\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:56.111793 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.111750 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7337-fbeb2b"] Oct 30 20:01:56.118676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.118644 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7337-fbeb2b"] Oct 30 20:01:56.133114 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.133085 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" Oct 30 20:01:56.133602 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.133114 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj" event={"ID":"a386a76c-8c8c-4de2-a5f4-86560053751f","Type":"ContainerDied","Data":"2be0ea29c0705234f03caa61cd1174c22a72789681822f760033375642a36d29"} Oct 30 20:01:56.133602 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.133209 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2be0ea29c0705234f03caa61cd1174c22a72789681822f760033375642a36d29" Oct 30 20:01:56.324455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.324421 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd"] Oct 30 20:01:56.325116 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.325102 5091 cpu_manager.go:414] "containerMap: removing container" podUID="24eae921-a70b-49f3-bf34-27608ba3ed22" containerName="test-curl-7337-fbeb2b" Oct 30 20:01:56.325183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.325117 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24eae921-a70b-49f3-bf34-27608ba3ed22" containerName="test-curl-7337-fbeb2b" Oct 30 20:01:56.325183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.325180 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a386a76c-8c8c-4de2-a5f4-86560053751f" containerName="revoke-keys" Oct 30 20:01:56.325260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.325187 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a386a76c-8c8c-4de2-a5f4-86560053751f" containerName="revoke-keys" Oct 30 20:01:56.325353 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.325342 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="24eae921-a70b-49f3-bf34-27608ba3ed22" containerName="test-curl-7337-fbeb2b" Oct 30 20:01:56.325394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.325363 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a386a76c-8c8c-4de2-a5f4-86560053751f" containerName="revoke-keys" Oct 30 20:01:56.329400 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.329382 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" Oct 30 20:01:56.338948 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.337433 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:01:56.338948 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.337555 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:01:56.339814 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.339790 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd"] Oct 30 20:01:56.526899 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.526862 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb262571-85ce-4f55-a2db-86068388b03f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22a98f80-brltd\" (UID: \"eb262571-85ce-4f55-a2db-86068388b03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" Oct 30 20:01:56.527066 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.526935 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpb57\" (UniqueName: \"kubernetes.io/projected/eb262571-85ce-4f55-a2db-86068388b03f-kube-api-access-tpb57\") pod \"maas-api-revoke-sub-22a98f80-brltd\" (UID: \"eb262571-85ce-4f55-a2db-86068388b03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" Oct 30 20:01:56.527066 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.527005 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb262571-85ce-4f55-a2db-86068388b03f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22a98f80-brltd\" (UID: \"eb262571-85ce-4f55-a2db-86068388b03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" Oct 30 20:01:56.639168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.631215 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb262571-85ce-4f55-a2db-86068388b03f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22a98f80-brltd\" (UID: \"eb262571-85ce-4f55-a2db-86068388b03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" Oct 30 20:01:56.639168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.631329 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpb57\" (UniqueName: \"kubernetes.io/projected/eb262571-85ce-4f55-a2db-86068388b03f-kube-api-access-tpb57\") pod \"maas-api-revoke-sub-22a98f80-brltd\" (UID: \"eb262571-85ce-4f55-a2db-86068388b03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" Oct 30 20:01:56.639168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.631370 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb262571-85ce-4f55-a2db-86068388b03f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22a98f80-brltd\" (UID: \"eb262571-85ce-4f55-a2db-86068388b03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" Oct 30 20:01:56.639168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.632636 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:56.639168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.633377 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb262571-85ce-4f55-a2db-86068388b03f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-22a98f80-brltd\" (UID: \"eb262571-85ce-4f55-a2db-86068388b03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" Oct 30 20:01:56.652605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.652559 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpb57\" (UniqueName: \"kubernetes.io/projected/eb262571-85ce-4f55-a2db-86068388b03f-kube-api-access-tpb57\") pod \"maas-api-revoke-sub-22a98f80-brltd\" (UID: \"eb262571-85ce-4f55-a2db-86068388b03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" Oct 30 20:01:56.667825 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.667795 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb262571-85ce-4f55-a2db-86068388b03f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-22a98f80-brltd\" (UID: \"eb262571-85ce-4f55-a2db-86068388b03f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" Oct 30 20:01:56.953086 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:56.953005 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" Oct 30 20:01:57.140762 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:57.140716 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7337-fbeb2b" podUID="24eae921-a70b-49f3-bf34-27608ba3ed22" containerName="test-curl-7337-fbeb2b" containerID="cri-o://2a4c8df94161760f7bbdffd39917f862e6a3f8a1f8636ac6275783ff063f0290" gracePeriod=2 Oct 30 20:01:57.244132 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:01:57.244095 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb262571_85ce_4f55_a2db_86068388b03f.slice/crio-2f3f6598c8dd9e723c16ea42108e0c312127bfa49d02af02c9d2161066a205cd WatchSource:0}: Error finding container 2f3f6598c8dd9e723c16ea42108e0c312127bfa49d02af02c9d2161066a205cd: Status 404 returned error can't find the container with id 2f3f6598c8dd9e723c16ea42108e0c312127bfa49d02af02c9d2161066a205cd Oct 30 20:01:57.245317 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:57.245292 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd"] Oct 30 20:01:57.481168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:57.474960 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:58.152897 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:58.152858 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" event={"ID":"eb262571-85ce-4f55-a2db-86068388b03f","Type":"ContainerStarted","Data":"1c2fd532b63bde926ee92192e4db9ba4359264cb0336478f203fc1a31c53d7ff"} Oct 30 20:01:58.153446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:58.152942 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" event={"ID":"eb262571-85ce-4f55-a2db-86068388b03f","Type":"ContainerStarted","Data":"2f3f6598c8dd9e723c16ea42108e0c312127bfa49d02af02c9d2161066a205cd"} Oct 30 20:01:58.221246 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:58.219794 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" podStartSLOduration=2.219772071 podStartE2EDuration="2.219772071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:01:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:01:58.183570308 +0000 UTC m=+1837.878778268" watchObservedRunningTime="2026-10-30 20:01:58.219772071 +0000 UTC m=+1837.914980029" Oct 30 20:01:58.508181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:58.503227 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:59.007793 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:59.007753 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:01:59.165472 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:59.165442 5091 generic.go:356] "Generic (PLEG): container finished" podID="eb262571-85ce-4f55-a2db-86068388b03f" containerID="1c2fd532b63bde926ee92192e4db9ba4359264cb0336478f203fc1a31c53d7ff" exitCode=0 Oct 30 20:01:59.165879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:59.165510 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" event={"ID":"eb262571-85ce-4f55-a2db-86068388b03f","Type":"ContainerDied","Data":"1c2fd532b63bde926ee92192e4db9ba4359264cb0336478f203fc1a31c53d7ff"} Oct 30 20:01:59.396133 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:59.395987 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-fbeb2b" Oct 30 20:01:59.488249 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:59.488211 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24eae921-a70b-49f3-bf34-27608ba3ed22-kube-api-access-4qncg\" (UniqueName: \"kubernetes.io/projected/24eae921-a70b-49f3-bf34-27608ba3ed22-kube-api-access-4qncg\") pod \"24eae921-a70b-49f3-bf34-27608ba3ed22\" (UID: \"24eae921-a70b-49f3-bf34-27608ba3ed22\") " Oct 30 20:01:59.490589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:59.490560 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24eae921-a70b-49f3-bf34-27608ba3ed22-kube-api-access-4qncg" pod "24eae921-a70b-49f3-bf34-27608ba3ed22" (UID: "24eae921-a70b-49f3-bf34-27608ba3ed22"). InnerVolumeSpecName "kube-api-access-4qncg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:01:59.591283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:59.591073 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qncg\" (UniqueName: \"kubernetes.io/projected/24eae921-a70b-49f3-bf34-27608ba3ed22-kube-api-access-4qncg\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:01:59.603221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:01:59.592332 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:00.175450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.175411 5091 generic.go:356] "Generic (PLEG): container finished" podID="24eae921-a70b-49f3-bf34-27608ba3ed22" containerID="2a4c8df94161760f7bbdffd39917f862e6a3f8a1f8636ac6275783ff063f0290" exitCode=137 Oct 30 20:02:00.175940 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.175456 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7337-fbeb2b" Oct 30 20:02:00.175940 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.175519 5091 scope.go:122] "RemoveContainer" containerID="2a4c8df94161760f7bbdffd39917f862e6a3f8a1f8636ac6275783ff063f0290" Oct 30 20:02:00.189689 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:02:00.189655 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7337-fbeb2b\" is forbidden: User \"system:node:ip-10-0-132-214.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-214.us-east-2.compute.internal' and this object" podUID="24eae921-a70b-49f3-bf34-27608ba3ed22" pod="opendatahub/test-curl-7337-fbeb2b" Oct 30 20:02:00.191856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.191828 5091 scope.go:122] "RemoveContainer" containerID="2a4c8df94161760f7bbdffd39917f862e6a3f8a1f8636ac6275783ff063f0290" Oct 30 20:02:00.192203 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:02:00.192138 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4c8df94161760f7bbdffd39917f862e6a3f8a1f8636ac6275783ff063f0290\": container with ID starting with 2a4c8df94161760f7bbdffd39917f862e6a3f8a1f8636ac6275783ff063f0290 not found: ID does not exist" containerID="2a4c8df94161760f7bbdffd39917f862e6a3f8a1f8636ac6275783ff063f0290" Oct 30 20:02:00.192311 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.192213 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4c8df94161760f7bbdffd39917f862e6a3f8a1f8636ac6275783ff063f0290"} err="failed to get container status \"2a4c8df94161760f7bbdffd39917f862e6a3f8a1f8636ac6275783ff063f0290\": rpc error: code = NotFound desc = could not find container \"2a4c8df94161760f7bbdffd39917f862e6a3f8a1f8636ac6275783ff063f0290\": container with ID starting with 2a4c8df94161760f7bbdffd39917f862e6a3f8a1f8636ac6275783ff063f0290 not found: ID does not exist" Oct 30 20:02:00.350226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.350162 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:00.402929 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.402894 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" Oct 30 20:02:00.432219 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.420105 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb262571-85ce-4f55-a2db-86068388b03f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb262571-85ce-4f55-a2db-86068388b03f-maas-api-xks-ca\") pod \"eb262571-85ce-4f55-a2db-86068388b03f\" (UID: \"eb262571-85ce-4f55-a2db-86068388b03f\") " Oct 30 20:02:00.478169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.466275 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb262571-85ce-4f55-a2db-86068388b03f-maas-api-xks-ca" pod "eb262571-85ce-4f55-a2db-86068388b03f" (UID: "eb262571-85ce-4f55-a2db-86068388b03f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:00.478169 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:02:00.466464 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7337-fbeb2b\" is forbidden: User \"system:node:ip-10-0-132-214.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-214.us-east-2.compute.internal' and this object" podUID="24eae921-a70b-49f3-bf34-27608ba3ed22" pod="opendatahub/test-curl-7337-fbeb2b" Oct 30 20:02:00.551271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.547726 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb262571-85ce-4f55-a2db-86068388b03f-kube-api-access-tpb57\" (UniqueName: \"kubernetes.io/projected/eb262571-85ce-4f55-a2db-86068388b03f-kube-api-access-tpb57\") pod \"eb262571-85ce-4f55-a2db-86068388b03f\" (UID: \"eb262571-85ce-4f55-a2db-86068388b03f\") " Oct 30 20:02:00.551271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.547821 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb262571-85ce-4f55-a2db-86068388b03f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb262571-85ce-4f55-a2db-86068388b03f-maas-api-openshift-ca\") pod \"eb262571-85ce-4f55-a2db-86068388b03f\" (UID: \"eb262571-85ce-4f55-a2db-86068388b03f\") " Oct 30 20:02:00.551271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.548647 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb262571-85ce-4f55-a2db-86068388b03f-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:00.560898 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.556408 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb262571-85ce-4f55-a2db-86068388b03f-maas-api-openshift-ca" pod "eb262571-85ce-4f55-a2db-86068388b03f" (UID: "eb262571-85ce-4f55-a2db-86068388b03f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:00.605969 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:02:00.605064 5091 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7337-fbeb2b\" is forbidden: User \"system:node:ip-10-0-132-214.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-132-214.us-east-2.compute.internal' and this object" podUID="24eae921-a70b-49f3-bf34-27608ba3ed22" pod="opendatahub/test-curl-7337-fbeb2b" Oct 30 20:02:00.656165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.655104 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb262571-85ce-4f55-a2db-86068388b03f-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:00.673396 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.673330 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb262571-85ce-4f55-a2db-86068388b03f-kube-api-access-tpb57" pod "eb262571-85ce-4f55-a2db-86068388b03f" (UID: "eb262571-85ce-4f55-a2db-86068388b03f"). InnerVolumeSpecName "kube-api-access-tpb57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:00.715798 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.715713 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24eae921-a70b-49f3-bf34-27608ba3ed22" path="/var/lib/kubelet/pods/24eae921-a70b-49f3-bf34-27608ba3ed22/volumes" Oct 30 20:02:00.738941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.737186 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8"] Oct 30 20:02:00.738941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.738714 5091 cpu_manager.go:414] "containerMap: removing container" podUID="eb262571-85ce-4f55-a2db-86068388b03f" containerName="revoke-keys" Oct 30 20:02:00.738941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.738737 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb262571-85ce-4f55-a2db-86068388b03f" containerName="revoke-keys" Oct 30 20:02:00.738941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.738941 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb262571-85ce-4f55-a2db-86068388b03f" containerName="revoke-keys" Oct 30 20:02:00.756195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.746495 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:00.757251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.757212 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpb57\" (UniqueName: \"kubernetes.io/projected/eb262571-85ce-4f55-a2db-86068388b03f-kube-api-access-tpb57\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:00.760240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.760207 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8"] Oct 30 20:02:00.761732 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.761710 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d97cc8-openshift-default-dockercfg-2fhzd\"" Oct 30 20:02:00.767701 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.767559 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:00.874237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.865527 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-data\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:00.874237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.865583 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78cdc\" (UniqueName: \"kubernetes.io/projected/071ef2ea-f7f5-4e12-9287-73c4f832f361-kube-api-access-78cdc\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:00.874237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.865614 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-token\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:00.874237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.865645 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-podinfo\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:00.874237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.865704 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-workload-certs\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:00.874237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.865774 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/071ef2ea-f7f5-4e12-9287-73c4f832f361-istiod-ca-cert\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:00.874237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.865888 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-workload-socket\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:00.874237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.865919 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-envoy\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:00.874237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:00.866181 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-credential-socket\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.098190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.043433 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-workload-socket\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.098190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.043478 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-envoy\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.098190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.043636 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-credential-socket\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.098190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.043701 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-data\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.098190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.043727 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78cdc\" (UniqueName: \"kubernetes.io/projected/071ef2ea-f7f5-4e12-9287-73c4f832f361-kube-api-access-78cdc\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.098190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.043753 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-token\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.098190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.043781 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-podinfo\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.098190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.043824 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-workload-certs\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.098190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.043885 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/071ef2ea-f7f5-4e12-9287-73c4f832f361-istiod-ca-cert\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.098190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.044681 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-data\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.098190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.044792 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/071ef2ea-f7f5-4e12-9287-73c4f832f361-istiod-ca-cert\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.098190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.076024 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-workload-socket\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.098190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.076767 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-credential-socket\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.098190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.077540 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:01.098190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.091753 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-workload-certs\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.194226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.194188 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78cdc\" (UniqueName: \"kubernetes.io/projected/071ef2ea-f7f5-4e12-9287-73c4f832f361-kube-api-access-78cdc\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.231680 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.201873 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-podinfo\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.231680 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.203160 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-envoy\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.233472 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.233436 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-token\") pod \"e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.239669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.239646 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" Oct 30 20:02:01.239784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.239643 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd" event={"ID":"eb262571-85ce-4f55-a2db-86068388b03f","Type":"ContainerDied","Data":"2f3f6598c8dd9e723c16ea42108e0c312127bfa49d02af02c9d2161066a205cd"} Oct 30 20:02:01.239840 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.239789 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f3f6598c8dd9e723c16ea42108e0c312127bfa49d02af02c9d2161066a205cd" Oct 30 20:02:01.359767 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.359669 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:01.368036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.368006 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:01.439524 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.439488 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" podUID="51c139a8-fc30-4991-baa6-75295965d8de" containerName="main" containerID="cri-o://6373822c312f3ea8baf0f6c24888b0d52cd85b288fdfedf84d448b392c744d66" gracePeriod=45 Oct 30 20:02:01.679204 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.679168 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8"] Oct 30 20:02:01.685292 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.685253 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:01.685398 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.685339 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:01.685398 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.685380 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:01.718907 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.718881 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 20:02:01.825875 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.825837 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-model-cache\") pod \"51c139a8-fc30-4991-baa6-75295965d8de\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " Oct 30 20:02:01.826074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.825999 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-dshm\") pod \"51c139a8-fc30-4991-baa6-75295965d8de\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " Oct 30 20:02:01.826074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.826037 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-tmp-dir\") pod \"51c139a8-fc30-4991-baa6-75295965d8de\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " Oct 30 20:02:01.826212 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.826079 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-home\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-home\") pod \"51c139a8-fc30-4991-baa6-75295965d8de\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " Oct 30 20:02:01.826212 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.826183 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51c139a8-fc30-4991-baa6-75295965d8de-kube-api-access-zmd8m\" (UniqueName: \"kubernetes.io/projected/51c139a8-fc30-4991-baa6-75295965d8de-kube-api-access-zmd8m\") pod \"51c139a8-fc30-4991-baa6-75295965d8de\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " Oct 30 20:02:01.826343 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.826300 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51c139a8-fc30-4991-baa6-75295965d8de-tls-certs\" (UniqueName: \"kubernetes.io/secret/51c139a8-fc30-4991-baa6-75295965d8de-tls-certs\") pod \"51c139a8-fc30-4991-baa6-75295965d8de\" (UID: \"51c139a8-fc30-4991-baa6-75295965d8de\") " Oct 30 20:02:01.827001 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.826774 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-model-cache" pod "51c139a8-fc30-4991-baa6-75295965d8de" (UID: "51c139a8-fc30-4991-baa6-75295965d8de"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:01.827001 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.826787 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-home" pod "51c139a8-fc30-4991-baa6-75295965d8de" (UID: "51c139a8-fc30-4991-baa6-75295965d8de"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:01.827001 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.826968 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-home\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:01.827001 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.826982 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-model-cache\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:01.827792 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.827751 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-tmp-dir" pod "51c139a8-fc30-4991-baa6-75295965d8de" (UID: "51c139a8-fc30-4991-baa6-75295965d8de"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:01.831612 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.831522 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51c139a8-fc30-4991-baa6-75295965d8de-tls-certs" pod "51c139a8-fc30-4991-baa6-75295965d8de" (UID: "51c139a8-fc30-4991-baa6-75295965d8de"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:01.831612 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.831556 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-dshm" pod "51c139a8-fc30-4991-baa6-75295965d8de" (UID: "51c139a8-fc30-4991-baa6-75295965d8de"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:01.834201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.834113 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51c139a8-fc30-4991-baa6-75295965d8de-kube-api-access-zmd8m" pod "51c139a8-fc30-4991-baa6-75295965d8de" (UID: "51c139a8-fc30-4991-baa6-75295965d8de"). InnerVolumeSpecName "kube-api-access-zmd8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:01.841474 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.841451 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz"] Oct 30 20:02:01.842450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.842434 5091 cpu_manager.go:414] "containerMap: removing container" podUID="51c139a8-fc30-4991-baa6-75295965d8de" containerName="main" Oct 30 20:02:01.842537 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.842527 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51c139a8-fc30-4991-baa6-75295965d8de" containerName="main" Oct 30 20:02:01.842755 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.842745 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="51c139a8-fc30-4991-baa6-75295965d8de" containerName="main" Oct 30 20:02:01.848516 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.848492 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" Oct 30 20:02:01.863853 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.863664 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:01.869687 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.868246 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz"] Oct 30 20:02:01.928381 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.928283 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c9ff71-ad8e-4eab-8950-8f1034481710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5444ffaa-wffcz\" (UID: \"c2c9ff71-ad8e-4eab-8950-8f1034481710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" Oct 30 20:02:01.928570 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.928420 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp24j\" (UniqueName: \"kubernetes.io/projected/c2c9ff71-ad8e-4eab-8950-8f1034481710-kube-api-access-gp24j\") pod \"maas-api-revoke-sub-5444ffaa-wffcz\" (UID: \"c2c9ff71-ad8e-4eab-8950-8f1034481710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" Oct 30 20:02:01.928570 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.928561 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c9ff71-ad8e-4eab-8950-8f1034481710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5444ffaa-wffcz\" (UID: \"c2c9ff71-ad8e-4eab-8950-8f1034481710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" Oct 30 20:02:01.928720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.928671 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmd8m\" (UniqueName: \"kubernetes.io/projected/51c139a8-fc30-4991-baa6-75295965d8de-kube-api-access-zmd8m\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:01.928720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.928684 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/51c139a8-fc30-4991-baa6-75295965d8de-tls-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:01.928720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.928693 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-dshm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:01.928720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.928702 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51c139a8-fc30-4991-baa6-75295965d8de-tmp-dir\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:01.960837 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:01.960803 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:02:02.030381 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.030348 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp24j\" (UniqueName: \"kubernetes.io/projected/c2c9ff71-ad8e-4eab-8950-8f1034481710-kube-api-access-gp24j\") pod \"maas-api-revoke-sub-5444ffaa-wffcz\" (UID: \"c2c9ff71-ad8e-4eab-8950-8f1034481710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" Oct 30 20:02:02.030557 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.030491 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c9ff71-ad8e-4eab-8950-8f1034481710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5444ffaa-wffcz\" (UID: \"c2c9ff71-ad8e-4eab-8950-8f1034481710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" Oct 30 20:02:02.030666 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.030646 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c9ff71-ad8e-4eab-8950-8f1034481710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5444ffaa-wffcz\" (UID: \"c2c9ff71-ad8e-4eab-8950-8f1034481710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" Oct 30 20:02:02.031388 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.031364 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c9ff71-ad8e-4eab-8950-8f1034481710-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5444ffaa-wffcz\" (UID: \"c2c9ff71-ad8e-4eab-8950-8f1034481710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" Oct 30 20:02:02.033045 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.033025 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c9ff71-ad8e-4eab-8950-8f1034481710-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5444ffaa-wffcz\" (UID: \"c2c9ff71-ad8e-4eab-8950-8f1034481710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" Oct 30 20:02:02.044933 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.044913 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp24j\" (UniqueName: \"kubernetes.io/projected/c2c9ff71-ad8e-4eab-8950-8f1034481710-kube-api-access-gp24j\") pod \"maas-api-revoke-sub-5444ffaa-wffcz\" (UID: \"c2c9ff71-ad8e-4eab-8950-8f1034481710\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" Oct 30 20:02:02.192985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.187263 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:02.197177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.196193 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" Oct 30 20:02:02.259567 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.259530 5091 generic.go:356] "Generic (PLEG): container finished" podID="51c139a8-fc30-4991-baa6-75295965d8de" containerID="6373822c312f3ea8baf0f6c24888b0d52cd85b288fdfedf84d448b392c744d66" exitCode=0 Oct 30 20:02:02.259766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.259611 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" Oct 30 20:02:02.259766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.259717 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" event={"ID":"51c139a8-fc30-4991-baa6-75295965d8de","Type":"ContainerDied","Data":"6373822c312f3ea8baf0f6c24888b0d52cd85b288fdfedf84d448b392c744d66"} Oct 30 20:02:02.259766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.259757 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" event={"ID":"51c139a8-fc30-4991-baa6-75295965d8de","Type":"ContainerDied","Data":"58d0893ad004fff98d85436a474865e152d66ca484f0b5810ac7a65ec040ef9d"} Oct 30 20:02:02.259932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.259778 5091 scope.go:122] "RemoveContainer" containerID="6373822c312f3ea8baf0f6c24888b0d52cd85b288fdfedf84d448b392c744d66" Oct 30 20:02:02.263277 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.262732 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" event={"ID":"071ef2ea-f7f5-4e12-9287-73c4f832f361","Type":"ContainerStarted","Data":"f2f279b992166019e79e7a0bfa6990bdbd4044a1aa3acfcf808ae8cbb196ec17"} Oct 30 20:02:02.263277 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.262782 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" event={"ID":"071ef2ea-f7f5-4e12-9287-73c4f832f361","Type":"ContainerStarted","Data":"12869bdfd0413f14d66bde01a4e1554a77e5ca326620dc033ef7e066a3b7e417"} Oct 30 20:02:02.275930 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.275870 5091 scope.go:122] "RemoveContainer" containerID="6373822c312f3ea8baf0f6c24888b0d52cd85b288fdfedf84d448b392c744d66" Oct 30 20:02:02.276338 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:02:02.276276 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6373822c312f3ea8baf0f6c24888b0d52cd85b288fdfedf84d448b392c744d66\": container with ID starting with 6373822c312f3ea8baf0f6c24888b0d52cd85b288fdfedf84d448b392c744d66 not found: ID does not exist" containerID="6373822c312f3ea8baf0f6c24888b0d52cd85b288fdfedf84d448b392c744d66" Oct 30 20:02:02.276338 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.276309 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6373822c312f3ea8baf0f6c24888b0d52cd85b288fdfedf84d448b392c744d66"} err="failed to get container status \"6373822c312f3ea8baf0f6c24888b0d52cd85b288fdfedf84d448b392c744d66\": rpc error: code = NotFound desc = could not find container \"6373822c312f3ea8baf0f6c24888b0d52cd85b288fdfedf84d448b392c744d66\": container with ID starting with 6373822c312f3ea8baf0f6c24888b0d52cd85b288fdfedf84d448b392c744d66 not found: ID does not exist" Oct 30 20:02:02.313346 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.313284 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" podStartSLOduration=2.313264597 podStartE2EDuration="2.313264597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:02.313079658 +0000 UTC m=+1842.008287617" watchObservedRunningTime="2026-10-30 20:02:02.313264597 +0000 UTC m=+1842.008472557" Oct 30 20:02:02.344819 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.344782 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg"] Oct 30 20:02:02.352469 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.352439 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg"] Oct 30 20:02:02.368350 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.368329 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:02.375723 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.375701 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:02.530208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.529014 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51c139a8-fc30-4991-baa6-75295965d8de" path="/var/lib/kubelet/pods/51c139a8-fc30-4991-baa6-75295965d8de/volumes" Oct 30 20:02:02.548667 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:02:02.548626 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c9ff71_ad8e_4eab_8950_8f1034481710.slice/crio-3bcbde07d303169e258d7310265770752cd1926d40f91998e905b732ced86cbf WatchSource:0}: Error finding container 3bcbde07d303169e258d7310265770752cd1926d40f91998e905b732ced86cbf: Status 404 returned error can't find the container with id 3bcbde07d303169e258d7310265770752cd1926d40f91998e905b732ced86cbf Oct 30 20:02:02.551270 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.551242 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz"] Oct 30 20:02:02.627931 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:02.627898 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-54d4c4/auth-test-model-54d4c4-kserve-7b57bbffb5-trbcg" podUID="51c139a8-fc30-4991-baa6-75295965d8de" containerName="main" probeResult="failure" output="Get \"https://10.132.0.227:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:02:03.193032 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:03.189186 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:03.275973 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:03.275936 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" event={"ID":"c2c9ff71-ad8e-4eab-8950-8f1034481710","Type":"ContainerStarted","Data":"33291e8e5bcdcbdb80807bfa50f9f3f282d9dd31a646516f72f3237544deef95"} Oct 30 20:02:03.276434 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:03.275999 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" event={"ID":"c2c9ff71-ad8e-4eab-8950-8f1034481710","Type":"ContainerStarted","Data":"3bcbde07d303169e258d7310265770752cd1926d40f91998e905b732ced86cbf"} Oct 30 20:02:03.276434 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:03.276038 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:03.277210 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:03.277185 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:02:03.303289 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:03.303241 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" podStartSLOduration=2.303226573 podStartE2EDuration="2.303226573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:03.300944913 +0000 UTC m=+1842.996152870" watchObservedRunningTime="2026-10-30 20:02:03.303226573 +0000 UTC m=+1842.998434596" Oct 30 20:02:04.293162 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:04.293110 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:04.324205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:04.323111 5091 generic.go:356] "Generic (PLEG): container finished" podID="c2c9ff71-ad8e-4eab-8950-8f1034481710" containerID="33291e8e5bcdcbdb80807bfa50f9f3f282d9dd31a646516f72f3237544deef95" exitCode=0 Oct 30 20:02:04.324479 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:04.324443 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" event={"ID":"c2c9ff71-ad8e-4eab-8950-8f1034481710","Type":"ContainerDied","Data":"33291e8e5bcdcbdb80807bfa50f9f3f282d9dd31a646516f72f3237544deef95"} Oct 30 20:02:04.724808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:04.724720 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:04.903826 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:04.903783 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:05.342754 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:05.342721 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:05.547417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:05.547345 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:05.591974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:05.591915 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" Oct 30 20:02:05.641164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:05.641115 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2c9ff71-ad8e-4eab-8950-8f1034481710-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c9ff71-ad8e-4eab-8950-8f1034481710-maas-api-xks-ca\") pod \"c2c9ff71-ad8e-4eab-8950-8f1034481710\" (UID: \"c2c9ff71-ad8e-4eab-8950-8f1034481710\") " Oct 30 20:02:05.641359 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:05.641331 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c9ff71-ad8e-4eab-8950-8f1034481710-kube-api-access-gp24j\" (UniqueName: \"kubernetes.io/projected/c2c9ff71-ad8e-4eab-8950-8f1034481710-kube-api-access-gp24j\") pod \"c2c9ff71-ad8e-4eab-8950-8f1034481710\" (UID: \"c2c9ff71-ad8e-4eab-8950-8f1034481710\") " Oct 30 20:02:05.641486 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:05.641400 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c9ff71-ad8e-4eab-8950-8f1034481710-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c9ff71-ad8e-4eab-8950-8f1034481710-maas-api-openshift-ca\") pod \"c2c9ff71-ad8e-4eab-8950-8f1034481710\" (UID: \"c2c9ff71-ad8e-4eab-8950-8f1034481710\") " Oct 30 20:02:05.642051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:05.642023 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c9ff71-ad8e-4eab-8950-8f1034481710-maas-api-openshift-ca" pod "c2c9ff71-ad8e-4eab-8950-8f1034481710" (UID: "c2c9ff71-ad8e-4eab-8950-8f1034481710"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:05.642768 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:05.642736 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2c9ff71-ad8e-4eab-8950-8f1034481710-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:05.644946 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:05.644905 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c9ff71-ad8e-4eab-8950-8f1034481710-kube-api-access-gp24j" pod "c2c9ff71-ad8e-4eab-8950-8f1034481710" (UID: "c2c9ff71-ad8e-4eab-8950-8f1034481710"). InnerVolumeSpecName "kube-api-access-gp24j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:05.650591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:05.650564 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c9ff71-ad8e-4eab-8950-8f1034481710-maas-api-xks-ca" pod "c2c9ff71-ad8e-4eab-8950-8f1034481710" (UID: "c2c9ff71-ad8e-4eab-8950-8f1034481710"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:05.744139 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:05.744101 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2c9ff71-ad8e-4eab-8950-8f1034481710-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:05.744139 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:05.744132 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gp24j\" (UniqueName: \"kubernetes.io/projected/c2c9ff71-ad8e-4eab-8950-8f1034481710-kube-api-access-gp24j\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:06.058384 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.058347 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4"] Oct 30 20:02:06.059517 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.059496 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c2c9ff71-ad8e-4eab-8950-8f1034481710" containerName="revoke-keys" Oct 30 20:02:06.059587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.059520 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c9ff71-ad8e-4eab-8950-8f1034481710" containerName="revoke-keys" Oct 30 20:02:06.059695 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.059684 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c9ff71-ad8e-4eab-8950-8f1034481710" containerName="revoke-keys" Oct 30 20:02:06.063704 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.063685 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" Oct 30 20:02:06.078308 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.078269 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d97cc8-dockercfg-5ztc6\"" Oct 30 20:02:06.080703 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.080675 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d97cc8\"" Oct 30 20:02:06.080949 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.080926 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4"] Oct 30 20:02:06.084038 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.083941 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:06.152063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.151982 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hw2x\" (UniqueName: \"kubernetes.io/projected/d973dcd0-1e61-4929-a1a0-648efb94577c-kube-api-access-4hw2x\") pod \"payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4\" (UID: \"d973dcd0-1e61-4929-a1a0-648efb94577c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" Oct 30 20:02:06.157621 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.157524 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d973dcd0-1e61-4929-a1a0-648efb94577c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4\" (UID: \"d973dcd0-1e61-4929-a1a0-648efb94577c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" Oct 30 20:02:06.270171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.264716 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hw2x\" (UniqueName: \"kubernetes.io/projected/d973dcd0-1e61-4929-a1a0-648efb94577c-kube-api-access-4hw2x\") pod \"payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4\" (UID: \"d973dcd0-1e61-4929-a1a0-648efb94577c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" Oct 30 20:02:06.270171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.264803 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d973dcd0-1e61-4929-a1a0-648efb94577c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4\" (UID: \"d973dcd0-1e61-4929-a1a0-648efb94577c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" Oct 30 20:02:06.270171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.265845 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d973dcd0-1e61-4929-a1a0-648efb94577c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4\" (UID: \"d973dcd0-1e61-4929-a1a0-648efb94577c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" Oct 30 20:02:06.315624 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.312983 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hw2x\" (UniqueName: \"kubernetes.io/projected/d973dcd0-1e61-4929-a1a0-648efb94577c-kube-api-access-4hw2x\") pod \"payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4\" (UID: \"d973dcd0-1e61-4929-a1a0-648efb94577c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" Oct 30 20:02:06.316026 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.315997 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:06.438171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.394425 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" Oct 30 20:02:06.657092 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.654092 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:06.681390 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.669233 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" Oct 30 20:02:06.681390 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.670196 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx"] Oct 30 20:02:06.686246 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.685626 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx"] Oct 30 20:02:06.686506 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.686482 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz" event={"ID":"c2c9ff71-ad8e-4eab-8950-8f1034481710","Type":"ContainerDied","Data":"3bcbde07d303169e258d7310265770752cd1926d40f91998e905b732ced86cbf"} Oct 30 20:02:06.686698 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.686667 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bcbde07d303169e258d7310265770752cd1926d40f91998e905b732ced86cbf" Oct 30 20:02:06.686825 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.686134 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" Oct 30 20:02:06.693928 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.693652 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:02:06.694354 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.694333 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:02:06.827899 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.827862 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-784a0698-tdjjx\" (UID: \"348fb41b-1181-40c8-9193-9bbcfa3ea9a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" Oct 30 20:02:06.828098 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.828004 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-784a0698-tdjjx\" (UID: \"348fb41b-1181-40c8-9193-9bbcfa3ea9a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" Oct 30 20:02:06.828220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.828101 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nz7d\" (UniqueName: \"kubernetes.io/projected/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-kube-api-access-4nz7d\") pod \"maas-api-revoke-sub-784a0698-tdjjx\" (UID: \"348fb41b-1181-40c8-9193-9bbcfa3ea9a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" Oct 30 20:02:06.930069 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.929981 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-784a0698-tdjjx\" (UID: \"348fb41b-1181-40c8-9193-9bbcfa3ea9a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" Oct 30 20:02:06.930069 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.930057 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-784a0698-tdjjx\" (UID: \"348fb41b-1181-40c8-9193-9bbcfa3ea9a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" Oct 30 20:02:06.930363 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.930090 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nz7d\" (UniqueName: \"kubernetes.io/projected/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-kube-api-access-4nz7d\") pod \"maas-api-revoke-sub-784a0698-tdjjx\" (UID: \"348fb41b-1181-40c8-9193-9bbcfa3ea9a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" Oct 30 20:02:06.930799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.930771 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-784a0698-tdjjx\" (UID: \"348fb41b-1181-40c8-9193-9bbcfa3ea9a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" Oct 30 20:02:06.932412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.932393 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-784a0698-tdjjx\" (UID: \"348fb41b-1181-40c8-9193-9bbcfa3ea9a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" Oct 30 20:02:06.945266 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.945231 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nz7d\" (UniqueName: \"kubernetes.io/projected/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-kube-api-access-4nz7d\") pod \"maas-api-revoke-sub-784a0698-tdjjx\" (UID: \"348fb41b-1181-40c8-9193-9bbcfa3ea9a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" Oct 30 20:02:06.985692 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:06.985657 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4"] Oct 30 20:02:06.986114 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:02:06.986083 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd973dcd0_1e61_4929_a1a0_648efb94577c.slice/crio-f8d01ad2283370c8ab418687c37138a5ffdc6939609d912524434979a7ccc728 WatchSource:0}: Error finding container f8d01ad2283370c8ab418687c37138a5ffdc6939609d912524434979a7ccc728: Status 404 returned error can't find the container with id f8d01ad2283370c8ab418687c37138a5ffdc6939609d912524434979a7ccc728 Oct 30 20:02:07.031023 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:07.030999 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" Oct 30 20:02:07.322836 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:02:07.322807 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod348fb41b_1181_40c8_9193_9bbcfa3ea9a1.slice/crio-6bb6ac3d89a5d521d0b5537b9c5e12ae71732105b04688907097baab4229901d WatchSource:0}: Error finding container 6bb6ac3d89a5d521d0b5537b9c5e12ae71732105b04688907097baab4229901d: Status 404 returned error can't find the container with id 6bb6ac3d89a5d521d0b5537b9c5e12ae71732105b04688907097baab4229901d Oct 30 20:02:07.324276 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:07.324247 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx"] Oct 30 20:02:07.616699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:07.616559 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:07.783172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:07.778985 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" event={"ID":"348fb41b-1181-40c8-9193-9bbcfa3ea9a1","Type":"ContainerStarted","Data":"6bb6ac3d89a5d521d0b5537b9c5e12ae71732105b04688907097baab4229901d"} Oct 30 20:02:07.794877 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:07.794832 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" event={"ID":"d973dcd0-1e61-4929-a1a0-648efb94577c","Type":"ContainerStarted","Data":"d62a88067ef970155a179d2d94cbde6d036b01a624d783e8292398fe3a802857"} Oct 30 20:02:07.795051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:07.794905 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" event={"ID":"d973dcd0-1e61-4929-a1a0-648efb94577c","Type":"ContainerStarted","Data":"f8d01ad2283370c8ab418687c37138a5ffdc6939609d912524434979a7ccc728"} Oct 30 20:02:07.796060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:07.796031 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" Oct 30 20:02:07.798045 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:07.797461 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:07.821309 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:07.821260 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" podStartSLOduration=1.821244979 podStartE2EDuration="1.821244979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:07.82079611 +0000 UTC m=+1847.516004069" watchObservedRunningTime="2026-10-30 20:02:07.821244979 +0000 UTC m=+1847.516452936" Oct 30 20:02:08.251241 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:08.251203 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:08.703789 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:08.703697 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:09.044895 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:09.041281 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" event={"ID":"348fb41b-1181-40c8-9193-9bbcfa3ea9a1","Type":"ContainerStarted","Data":"6ec1950c5c991b0ef0351fb75b865b991c871480c12332f071623d17a2fee018"} Oct 30 20:02:09.174552 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:09.174514 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:09.592142 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:09.592104 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj"] Oct 30 20:02:09.592634 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:09.592587 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" podUID="f28281f9-b9a0-4522-9336-ea91aa6a4ce0" containerName="istio-proxy" containerID="cri-o://3bc3fad560e5f440d0e6b88b03e49522ea90e03df21cc41927de3c8ac0880e27" gracePeriod=30 Oct 30 20:02:09.599087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:09.599063 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5"] Oct 30 20:02:09.600689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:09.599293 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" podUID="34968af2-47bb-4c0a-ba8b-974ae9b70f30" containerName="istio-proxy" containerID="cri-o://b0fb368a71550075b1bbe6b65a5a95525b37a2aa52a484d49b50fed08ed127e8" gracePeriod=30 Oct 30 20:02:10.089169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:10.088814 5091 generic.go:356] "Generic (PLEG): container finished" podID="348fb41b-1181-40c8-9193-9bbcfa3ea9a1" containerID="6ec1950c5c991b0ef0351fb75b865b991c871480c12332f071623d17a2fee018" exitCode=0 Oct 30 20:02:10.105268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:10.090628 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" event={"ID":"348fb41b-1181-40c8-9193-9bbcfa3ea9a1","Type":"ContainerDied","Data":"6ec1950c5c991b0ef0351fb75b865b991c871480c12332f071623d17a2fee018"} Oct 30 20:02:10.430217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:10.429577 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:10.729729 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:10.729652 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" Oct 30 20:02:10.821463 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:10.821297 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-maas-api-openshift-ca\") pod \"348fb41b-1181-40c8-9193-9bbcfa3ea9a1\" (UID: \"348fb41b-1181-40c8-9193-9bbcfa3ea9a1\") " Oct 30 20:02:10.821463 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:10.821448 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-maas-api-xks-ca\") pod \"348fb41b-1181-40c8-9193-9bbcfa3ea9a1\" (UID: \"348fb41b-1181-40c8-9193-9bbcfa3ea9a1\") " Oct 30 20:02:10.821704 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:10.821687 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-kube-api-access-4nz7d\" (UniqueName: \"kubernetes.io/projected/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-kube-api-access-4nz7d\") pod \"348fb41b-1181-40c8-9193-9bbcfa3ea9a1\" (UID: \"348fb41b-1181-40c8-9193-9bbcfa3ea9a1\") " Oct 30 20:02:10.823030 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:10.823006 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-maas-api-openshift-ca" pod "348fb41b-1181-40c8-9193-9bbcfa3ea9a1" (UID: "348fb41b-1181-40c8-9193-9bbcfa3ea9a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:10.824544 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:10.824516 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-kube-api-access-4nz7d" pod "348fb41b-1181-40c8-9193-9bbcfa3ea9a1" (UID: "348fb41b-1181-40c8-9193-9bbcfa3ea9a1"). InnerVolumeSpecName "kube-api-access-4nz7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:10.825067 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:10.825045 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-maas-api-xks-ca" pod "348fb41b-1181-40c8-9193-9bbcfa3ea9a1" (UID: "348fb41b-1181-40c8-9193-9bbcfa3ea9a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:10.923711 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:10.923678 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:10.923711 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:10.923709 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nz7d\" (UniqueName: \"kubernetes.io/projected/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-kube-api-access-4nz7d\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:10.923711 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:10.923719 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/348fb41b-1181-40c8-9193-9bbcfa3ea9a1-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:11.110550 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:11.110514 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" event={"ID":"348fb41b-1181-40c8-9193-9bbcfa3ea9a1","Type":"ContainerDied","Data":"6bb6ac3d89a5d521d0b5537b9c5e12ae71732105b04688907097baab4229901d"} Oct 30 20:02:11.111036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:11.110571 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bb6ac3d89a5d521d0b5537b9c5e12ae71732105b04688907097baab4229901d" Oct 30 20:02:11.111036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:11.110633 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx" Oct 30 20:02:11.523965 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:11.523923 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:11.712170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:11.695839 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:11.916748 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:11.916666 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:12.123334 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:12.123298 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:13.161660 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.161589 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:13.538379 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.538325 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql"] Oct 30 20:02:13.544865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.540705 5091 cpu_manager.go:414] "containerMap: removing container" podUID="348fb41b-1181-40c8-9193-9bbcfa3ea9a1" containerName="revoke-keys" Oct 30 20:02:13.544865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.540739 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="348fb41b-1181-40c8-9193-9bbcfa3ea9a1" containerName="revoke-keys" Oct 30 20:02:13.544865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.540999 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="348fb41b-1181-40c8-9193-9bbcfa3ea9a1" containerName="revoke-keys" Oct 30 20:02:13.545166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.545076 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" Oct 30 20:02:13.598170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.580195 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:13.620173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.618724 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-222480\"" Oct 30 20:02:13.646175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.640927 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgxmn\" (UniqueName: \"kubernetes.io/projected/ca54001a-b870-4f69-a748-8e71e28c8241-kube-api-access-qgxmn\") pod \"payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql\" (UID: \"ca54001a-b870-4f69-a748-8e71e28c8241\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" Oct 30 20:02:13.646175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.641099 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca54001a-b870-4f69-a748-8e71e28c8241-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql\" (UID: \"ca54001a-b870-4f69-a748-8e71e28c8241\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" Oct 30 20:02:13.732774 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.732742 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql"] Oct 30 20:02:13.737659 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.737629 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx"] Oct 30 20:02:13.742665 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.742640 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca54001a-b870-4f69-a748-8e71e28c8241-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql\" (UID: \"ca54001a-b870-4f69-a748-8e71e28c8241\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" Oct 30 20:02:13.742919 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.742896 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgxmn\" (UniqueName: \"kubernetes.io/projected/ca54001a-b870-4f69-a748-8e71e28c8241-kube-api-access-qgxmn\") pod \"payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql\" (UID: \"ca54001a-b870-4f69-a748-8e71e28c8241\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" Oct 30 20:02:13.743216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.743192 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" Oct 30 20:02:13.743568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.743547 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca54001a-b870-4f69-a748-8e71e28c8241-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql\" (UID: \"ca54001a-b870-4f69-a748-8e71e28c8241\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" Oct 30 20:02:13.768231 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.767905 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-222480-dockercfg-7tx9c\"" Oct 30 20:02:13.801546 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.795323 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:13.894171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.872959 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgxmn\" (UniqueName: \"kubernetes.io/projected/ca54001a-b870-4f69-a748-8e71e28c8241-kube-api-access-qgxmn\") pod \"payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql\" (UID: \"ca54001a-b870-4f69-a748-8e71e28c8241\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" Oct 30 20:02:13.894171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:13.877195 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx"] Oct 30 20:02:14.150655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:14.150563 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:14.174756 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:14.174722 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" Oct 30 20:02:14.254257 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:14.190812 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b423dc9e-e093-4107-b703-b94fa7c96911-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx\" (UID: \"b423dc9e-e093-4107-b703-b94fa7c96911\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" Oct 30 20:02:14.254257 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:14.190956 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg8ft\" (UniqueName: \"kubernetes.io/projected/b423dc9e-e093-4107-b703-b94fa7c96911-kube-api-access-tg8ft\") pod \"payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx\" (UID: \"b423dc9e-e093-4107-b703-b94fa7c96911\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" Oct 30 20:02:14.310723 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:14.310668 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b423dc9e-e093-4107-b703-b94fa7c96911-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx\" (UID: \"b423dc9e-e093-4107-b703-b94fa7c96911\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" Oct 30 20:02:14.311323 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:14.311175 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg8ft\" (UniqueName: \"kubernetes.io/projected/b423dc9e-e093-4107-b703-b94fa7c96911-kube-api-access-tg8ft\") pod \"payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx\" (UID: \"b423dc9e-e093-4107-b703-b94fa7c96911\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" Oct 30 20:02:14.336202 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:14.336166 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b423dc9e-e093-4107-b703-b94fa7c96911-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx\" (UID: \"b423dc9e-e093-4107-b703-b94fa7c96911\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" Oct 30 20:02:14.443279 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:14.438364 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg8ft\" (UniqueName: \"kubernetes.io/projected/b423dc9e-e093-4107-b703-b94fa7c96911-kube-api-access-tg8ft\") pod \"payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx\" (UID: \"b423dc9e-e093-4107-b703-b94fa7c96911\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" Oct 30 20:02:14.535307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:14.520421 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" Oct 30 20:02:14.936434 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:14.931258 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:15.123022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.122980 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql"] Oct 30 20:02:15.483422 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.483386 5091 generic.go:356] "Generic (PLEG): container finished" podID="34968af2-47bb-4c0a-ba8b-974ae9b70f30" containerID="b0fb368a71550075b1bbe6b65a5a95525b37a2aa52a484d49b50fed08ed127e8" exitCode=0 Oct 30 20:02:15.484650 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.484525 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" event={"ID":"34968af2-47bb-4c0a-ba8b-974ae9b70f30","Type":"ContainerDied","Data":"b0fb368a71550075b1bbe6b65a5a95525b37a2aa52a484d49b50fed08ed127e8"} Oct 30 20:02:15.490120 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.490095 5091 generic.go:356] "Generic (PLEG): container finished" podID="f28281f9-b9a0-4522-9336-ea91aa6a4ce0" containerID="3bc3fad560e5f440d0e6b88b03e49522ea90e03df21cc41927de3c8ac0880e27" exitCode=0 Oct 30 20:02:15.495277 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.490398 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" event={"ID":"f28281f9-b9a0-4522-9336-ea91aa6a4ce0","Type":"ContainerDied","Data":"3bc3fad560e5f440d0e6b88b03e49522ea90e03df21cc41927de3c8ac0880e27"} Oct 30 20:02:15.499817 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.499792 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" event={"ID":"ca54001a-b870-4f69-a748-8e71e28c8241","Type":"ContainerStarted","Data":"a738eb61632e2d0b290fa2d2ae638c250e1df02376dd4563894a55ed7e176262"} Oct 30 20:02:15.620553 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.619406 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx"] Oct 30 20:02:15.627690 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.627667 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 20:02:15.737845 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.736058 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-workload-socket\") pod \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " Oct 30 20:02:15.737845 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.736112 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-data\") pod \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " Oct 30 20:02:15.737845 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.736196 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-kube-api-access-xx9c9\" (UniqueName: \"kubernetes.io/projected/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-kube-api-access-xx9c9\") pod \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " Oct 30 20:02:15.737845 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.736266 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-workload-certs\") pod \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " Oct 30 20:02:15.737845 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.736467 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-token\" (UniqueName: \"kubernetes.io/projected/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-token\") pod \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " Oct 30 20:02:15.737845 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.736518 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istiod-ca-cert\") pod \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " Oct 30 20:02:15.760250 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.738790 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istiod-ca-cert" pod "f28281f9-b9a0-4522-9336-ea91aa6a4ce0" (UID: "f28281f9-b9a0-4522-9336-ea91aa6a4ce0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:15.763795 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.763588 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-envoy\") pod \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " Oct 30 20:02:15.763795 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.763707 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-credential-socket\") pod \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " Oct 30 20:02:15.763795 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.763751 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-podinfo\") pod \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\" (UID: \"f28281f9-b9a0-4522-9336-ea91aa6a4ce0\") " Oct 30 20:02:15.789849 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.788969 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-workload-socket" pod "f28281f9-b9a0-4522-9336-ea91aa6a4ce0" (UID: "f28281f9-b9a0-4522-9336-ea91aa6a4ce0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:15.789849 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.789396 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-workload-certs" pod "f28281f9-b9a0-4522-9336-ea91aa6a4ce0" (UID: "f28281f9-b9a0-4522-9336-ea91aa6a4ce0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:15.789849 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.789447 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:15.789849 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.789474 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:15.789849 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.789766 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-data" pod "f28281f9-b9a0-4522-9336-ea91aa6a4ce0" (UID: "f28281f9-b9a0-4522-9336-ea91aa6a4ce0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:15.792939 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.790458 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-envoy" pod "f28281f9-b9a0-4522-9336-ea91aa6a4ce0" (UID: "f28281f9-b9a0-4522-9336-ea91aa6a4ce0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:15.792939 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.792052 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-podinfo" pod "f28281f9-b9a0-4522-9336-ea91aa6a4ce0" (UID: "f28281f9-b9a0-4522-9336-ea91aa6a4ce0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:15.792939 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.792274 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-token" pod "f28281f9-b9a0-4522-9336-ea91aa6a4ce0" (UID: "f28281f9-b9a0-4522-9336-ea91aa6a4ce0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:15.793345 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.793270 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-credential-socket" pod "f28281f9-b9a0-4522-9336-ea91aa6a4ce0" (UID: "f28281f9-b9a0-4522-9336-ea91aa6a4ce0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:15.795611 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.795548 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-kube-api-access-xx9c9" pod "f28281f9-b9a0-4522-9336-ea91aa6a4ce0" (UID: "f28281f9-b9a0-4522-9336-ea91aa6a4ce0"). InnerVolumeSpecName "kube-api-access-xx9c9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:15.803800 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.801235 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 20:02:15.903643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.897412 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:15.903643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.898385 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:15.903643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.898417 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:15.903643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.898438 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:15.903643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.898459 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:15.903643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.898475 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:15.903643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.898519 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx9c9\" (UniqueName: \"kubernetes.io/projected/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-kube-api-access-xx9c9\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:15.903643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.898538 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f28281f9-b9a0-4522-9336-ea91aa6a4ce0-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:15.999826 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.999791 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34968af2-47bb-4c0a-ba8b-974ae9b70f30-kube-api-access-2j7mk\" (UniqueName: \"kubernetes.io/projected/34968af2-47bb-4c0a-ba8b-974ae9b70f30-kube-api-access-2j7mk\") pod \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " Oct 30 20:02:16.000006 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.999869 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-workload-socket\") pod \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " Oct 30 20:02:16.000006 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:15.999951 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-token\" (UniqueName: \"kubernetes.io/projected/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-token\") pod \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " Oct 30 20:02:16.000006 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.000001 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-envoy\") pod \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " Oct 30 20:02:16.000596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.000106 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-data\") pod \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " Oct 30 20:02:16.000596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.000247 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-podinfo\") pod \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " Oct 30 20:02:16.000596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.000362 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-credential-socket\") pod \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " Oct 30 20:02:16.000596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.000428 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-workload-certs\") pod \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " Oct 30 20:02:16.000596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.000472 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istiod-ca-cert\") pod \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\" (UID: \"34968af2-47bb-4c0a-ba8b-974ae9b70f30\") " Oct 30 20:02:16.001562 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.001537 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istiod-ca-cert" pod "34968af2-47bb-4c0a-ba8b-974ae9b70f30" (UID: "34968af2-47bb-4c0a-ba8b-974ae9b70f30"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:16.002000 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.001981 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-data" pod "34968af2-47bb-4c0a-ba8b-974ae9b70f30" (UID: "34968af2-47bb-4c0a-ba8b-974ae9b70f30"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:16.003943 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.003721 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-workload-socket" pod "34968af2-47bb-4c0a-ba8b-974ae9b70f30" (UID: "34968af2-47bb-4c0a-ba8b-974ae9b70f30"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:16.004407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.004116 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-credential-socket" pod "34968af2-47bb-4c0a-ba8b-974ae9b70f30" (UID: "34968af2-47bb-4c0a-ba8b-974ae9b70f30"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:16.004842 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.004821 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-workload-certs" pod "34968af2-47bb-4c0a-ba8b-974ae9b70f30" (UID: "34968af2-47bb-4c0a-ba8b-974ae9b70f30"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:16.007574 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.007535 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34968af2-47bb-4c0a-ba8b-974ae9b70f30-kube-api-access-2j7mk" pod "34968af2-47bb-4c0a-ba8b-974ae9b70f30" (UID: "34968af2-47bb-4c0a-ba8b-974ae9b70f30"). InnerVolumeSpecName "kube-api-access-2j7mk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:16.008852 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.008822 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-token" pod "34968af2-47bb-4c0a-ba8b-974ae9b70f30" (UID: "34968af2-47bb-4c0a-ba8b-974ae9b70f30"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:16.014991 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.014930 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-podinfo" pod "34968af2-47bb-4c0a-ba8b-974ae9b70f30" (UID: "34968af2-47bb-4c0a-ba8b-974ae9b70f30"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:16.015282 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.015238 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-envoy" pod "34968af2-47bb-4c0a-ba8b-974ae9b70f30" (UID: "34968af2-47bb-4c0a-ba8b-974ae9b70f30"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:16.028787 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:02:16.028690 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:02:16.102759 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.102718 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:16.102759 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.102762 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:16.102953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.102780 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:16.102953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.102796 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:16.102953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.102811 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j7mk\" (UniqueName: \"kubernetes.io/projected/34968af2-47bb-4c0a-ba8b-974ae9b70f30-kube-api-access-2j7mk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:16.102953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.102828 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:16.102953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.102842 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:16.102953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.102858 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:16.102953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.102872 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34968af2-47bb-4c0a-ba8b-974ae9b70f30-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:16.515296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.515258 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" event={"ID":"ca54001a-b870-4f69-a748-8e71e28c8241","Type":"ContainerStarted","Data":"7cfb65325a0de0326e94225f9eadd6338d2f77edd215ac52c2496a3554b97771"} Oct 30 20:02:16.515916 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.515607 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" Oct 30 20:02:16.517545 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.517512 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" event={"ID":"b423dc9e-e093-4107-b703-b94fa7c96911","Type":"ContainerStarted","Data":"c0f59e92b80b57445f885ec82a976b7f2e98e7e5e42feecc15fa1ba42380e70b"} Oct 30 20:02:16.517717 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.517568 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" event={"ID":"b423dc9e-e093-4107-b703-b94fa7c96911","Type":"ContainerStarted","Data":"73ca098dc064be2d6e2da82143358bc48b16bb65a9886f422a2449614a8bb124"} Oct 30 20:02:16.517717 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.517600 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" Oct 30 20:02:16.519345 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.519322 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" Oct 30 20:02:16.519485 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.519353 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5" event={"ID":"34968af2-47bb-4c0a-ba8b-974ae9b70f30","Type":"ContainerDied","Data":"82e43ff5f02bcb5974edde2c35216eb7e1241613d264197fceb3d09b1265d09d"} Oct 30 20:02:16.519485 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.519408 5091 scope.go:122] "RemoveContainer" containerID="b0fb368a71550075b1bbe6b65a5a95525b37a2aa52a484d49b50fed08ed127e8" Oct 30 20:02:16.521677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.521654 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" Oct 30 20:02:16.521794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.521651 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj" event={"ID":"f28281f9-b9a0-4522-9336-ea91aa6a4ce0","Type":"ContainerDied","Data":"2970ee9bf48a50b715734ea68a0d10b0c0deebf1e979bc3c86697640a80a3a1e"} Oct 30 20:02:16.535080 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.535051 5091 scope.go:122] "RemoveContainer" containerID="3bc3fad560e5f440d0e6b88b03e49522ea90e03df21cc41927de3c8ac0880e27" Oct 30 20:02:16.544248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.544193 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" podStartSLOduration=3.54417615 podStartE2EDuration="3.54417615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:16.540095222 +0000 UTC m=+1856.235303181" watchObservedRunningTime="2026-10-30 20:02:16.54417615 +0000 UTC m=+1856.239384106" Oct 30 20:02:16.579439 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.579408 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj"] Oct 30 20:02:16.586435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.586406 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-zz7sj"] Oct 30 20:02:16.623459 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.623405 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" podStartSLOduration=3.623388501 podStartE2EDuration="3.623388501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:16.601100267 +0000 UTC m=+1856.296308225" watchObservedRunningTime="2026-10-30 20:02:16.623388501 +0000 UTC m=+1856.318596460" Oct 30 20:02:16.628999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.628956 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5"] Oct 30 20:02:16.634380 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:16.634358 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-t6nl5"] Oct 30 20:02:17.025214 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:17.024017 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:17.982003 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:17.981736 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:18.456167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:18.451065 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:18.576428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:18.539133 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34968af2-47bb-4c0a-ba8b-974ae9b70f30" path="/var/lib/kubelet/pods/34968af2-47bb-4c0a-ba8b-974ae9b70f30/volumes" Oct 30 20:02:18.576428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:18.572864 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28281f9-b9a0-4522-9336-ea91aa6a4ce0" path="/var/lib/kubelet/pods/f28281f9-b9a0-4522-9336-ea91aa6a4ce0/volumes" Oct 30 20:02:19.230564 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.230528 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:19.480461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.480411 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2"] Oct 30 20:02:19.489042 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.488957 5091 cpu_manager.go:414] "containerMap: removing container" podUID="34968af2-47bb-4c0a-ba8b-974ae9b70f30" containerName="istio-proxy" Oct 30 20:02:19.489267 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.489237 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34968af2-47bb-4c0a-ba8b-974ae9b70f30" containerName="istio-proxy" Oct 30 20:02:19.489412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.489400 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f28281f9-b9a0-4522-9336-ea91aa6a4ce0" containerName="istio-proxy" Oct 30 20:02:19.489518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.489505 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28281f9-b9a0-4522-9336-ea91aa6a4ce0" containerName="istio-proxy" Oct 30 20:02:19.489865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.489832 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28281f9-b9a0-4522-9336-ea91aa6a4ce0" containerName="istio-proxy" Oct 30 20:02:19.490028 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.490014 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="34968af2-47bb-4c0a-ba8b-974ae9b70f30" containerName="istio-proxy" Oct 30 20:02:19.505106 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.499883 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:19.523587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.523552 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2"] Oct 30 20:02:19.523742 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.523696 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.536106 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.536081 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-836d67-openshift-default-dockercfg-bcv7n\"" Oct 30 20:02:19.685215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.653520 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-istio-envoy\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.796871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.796829 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632cf328-468e-4865-a6ef-7358d679cb6d-istio-token\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.816350 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.816297 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-istio-data\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.816777 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.816742 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-workload-certs\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.834167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.832622 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632cf328-468e-4865-a6ef-7358d679cb6d-istio-podinfo\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.834167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.832698 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57m6l\" (UniqueName: \"kubernetes.io/projected/632cf328-468e-4865-a6ef-7358d679cb6d-kube-api-access-57m6l\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.834167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.832828 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632cf328-468e-4865-a6ef-7358d679cb6d-istiod-ca-cert\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.834167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.833012 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-credential-socket\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.834167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.833060 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-workload-socket\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.885003 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.846541 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:19.946601 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.946447 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-credential-socket\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.946858 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.946835 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-workload-socket\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.947039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.947025 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-istio-envoy\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.947220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.947201 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632cf328-468e-4865-a6ef-7358d679cb6d-istio-token\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.947459 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.947434 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-istio-data\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.947673 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.947654 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-workload-certs\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.947789 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.947775 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632cf328-468e-4865-a6ef-7358d679cb6d-istio-podinfo\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.947890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.947877 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57m6l\" (UniqueName: \"kubernetes.io/projected/632cf328-468e-4865-a6ef-7358d679cb6d-kube-api-access-57m6l\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.948037 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.948024 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632cf328-468e-4865-a6ef-7358d679cb6d-istiod-ca-cert\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.948684 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.948661 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-credential-socket\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.949065 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.949047 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-workload-socket\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:19.951268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.950631 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-workload-certs\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:20.001494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.953230 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632cf328-468e-4865-a6ef-7358d679cb6d-istiod-ca-cert\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:20.001494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:19.958364 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-istio-data\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:20.059210 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:20.059124 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632cf328-468e-4865-a6ef-7358d679cb6d-istio-token\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:20.059500 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:20.059445 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-istio-envoy\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:20.059879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:20.059826 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632cf328-468e-4865-a6ef-7358d679cb6d-istio-podinfo\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:20.062300 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:20.062255 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57m6l\" (UniqueName: \"kubernetes.io/projected/632cf328-468e-4865-a6ef-7358d679cb6d-kube-api-access-57m6l\") pod \"e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:20.092956 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:20.092851 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:20.097069 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:20.095838 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" Oct 30 20:02:20.156463 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:20.156426 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:20.504820 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:20.504726 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:20.532870 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:20.532493 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2"] Oct 30 20:02:20.537133 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:02:20.537092 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod632cf328_468e_4865_a6ef_7358d679cb6d.slice/crio-3ddf74a906a61965e00269df83bfdb2ebe7c70a787ea2d349b8703928adf9f51 WatchSource:0}: Error finding container 3ddf74a906a61965e00269df83bfdb2ebe7c70a787ea2d349b8703928adf9f51: Status 404 returned error can't find the container with id 3ddf74a906a61965e00269df83bfdb2ebe7c70a787ea2d349b8703928adf9f51 Oct 30 20:02:20.546415 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:20.541251 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:20.546415 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:20.541323 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:20.546415 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:20.541363 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:20.761743 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:20.761644 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" event={"ID":"632cf328-468e-4865-a6ef-7358d679cb6d","Type":"ContainerStarted","Data":"05cc5cc5c831bc8c6a3ee2bf75528e70eb90d375b85397a00f6aec2e88cfd762"} Oct 30 20:02:20.761743 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:20.761721 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" event={"ID":"632cf328-468e-4865-a6ef-7358d679cb6d","Type":"ContainerStarted","Data":"3ddf74a906a61965e00269df83bfdb2ebe7c70a787ea2d349b8703928adf9f51"} Oct 30 20:02:20.802347 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:20.802283 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" podStartSLOduration=1.80226257 podStartE2EDuration="1.80226257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:20.791676354 +0000 UTC m=+1860.486884339" watchObservedRunningTime="2026-10-30 20:02:20.80226257 +0000 UTC m=+1860.497470542" Oct 30 20:02:21.158688 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.158659 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:21.164238 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.164213 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:21.703167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.682435 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:21.775244 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.775211 5091 generic.go:356] "Generic (PLEG): container finished" podID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerID="f22c1b8878712b749f290f1e15da4c381305985e674b688ad3285f6b18ec360a" exitCode=0 Oct 30 20:02:21.775441 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.775284 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" event={"ID":"374daf82-137e-4a07-ad5f-b205bcb571fc","Type":"ContainerDied","Data":"f22c1b8878712b749f290f1e15da4c381305985e674b688ad3285f6b18ec360a"} Oct 30 20:02:21.775887 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.775864 5091 scope.go:122] "RemoveContainer" containerID="f22c1b8878712b749f290f1e15da4c381305985e674b688ad3285f6b18ec360a" Oct 30 20:02:21.777316 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.777295 5091 generic.go:356] "Generic (PLEG): container finished" podID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerID="b6ee1b25d608772915087d0d56222d6449e4450181ecb3053876f93e3f314c99" exitCode=0 Oct 30 20:02:21.777422 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.777362 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" event={"ID":"2bcb265a-c599-404a-b848-b62b6e632f2a","Type":"ContainerDied","Data":"b6ee1b25d608772915087d0d56222d6449e4450181ecb3053876f93e3f314c99"} Oct 30 20:02:21.777652 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.777619 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:21.777855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.777835 5091 scope.go:122] "RemoveContainer" containerID="b6ee1b25d608772915087d0d56222d6449e4450181ecb3053876f93e3f314c99" Oct 30 20:02:21.777981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.777953 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:21.778048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.778021 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:21.778107 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.778058 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:21.778777 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.778759 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:02:21.779885 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.779856 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:21.779953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.779935 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:21.780013 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:21.779998 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:22.043886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:22.043795 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:02:22.820179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:22.820116 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" event={"ID":"374daf82-137e-4a07-ad5f-b205bcb571fc","Type":"ContainerStarted","Data":"01e6ec8e203e65d17a3c0b1202e25b0a44ec2b522c011cf7b84a2bd1c7503c29"} Oct 30 20:02:22.823923 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:22.823896 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" event={"ID":"2bcb265a-c599-404a-b848-b62b6e632f2a","Type":"ContainerStarted","Data":"cd47aab2566f815246ded1d16b7fff7e4122935c0ce49879eaa31dd943fc0d0d"} Oct 30 20:02:22.863173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:22.857135 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:23.064690 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.064655 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:02:23.066395 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.066364 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 20:02:23.066622 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.066598 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 20:02:23.159446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.159349 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:02:23.199997 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.199965 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:02:23.201744 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.201706 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 20:02:23.201984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.201950 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 20:02:23.323682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.323208 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:23.503179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.503108 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:23.606181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.604891 5091 scope.go:122] "RemoveContainer" containerID="47f9eab5d3887ce0e970a380ba6d0dd44fe9b583b5c1356255d4f64e3842baed" Oct 30 20:02:23.623367 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.623344 5091 scope.go:122] "RemoveContainer" containerID="7d3ea2d146498184719c24765b00ea817d004f3a1e135f2b7a10de86bad4c427" Oct 30 20:02:23.633749 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.633727 5091 scope.go:122] "RemoveContainer" containerID="7ca5363585d104d67cec2b6cba60ccad600b69e3c5a551a5e1c8b657eb4b44d0" Oct 30 20:02:23.644036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.644015 5091 scope.go:122] "RemoveContainer" containerID="d1df54649cb655f0323f9124dbb981b0348aefd0b3a7470cc81e0e8a39419663" Oct 30 20:02:23.707860 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.707703 5091 scope.go:122] "RemoveContainer" containerID="9af42e62d6297b68c7754ae35f3e8d12d84b2b25cf8cadbc48b1b5ccec1e9142" Oct 30 20:02:23.723478 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.723453 5091 scope.go:122] "RemoveContainer" containerID="17f1494ef3449916735b5e3dc13b9c209bbc7741866d0d493357bb9bd39d4a9e" Oct 30 20:02:23.734672 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.734644 5091 scope.go:122] "RemoveContainer" containerID="081107cf61caf6fd577131076a96bb92b0d3c87adf1f36bcb9f8aa0b8b556cf3" Oct 30 20:02:23.773182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.770463 5091 scope.go:122] "RemoveContainer" containerID="00ca306477bdb853ca39453ad41cf69a27b85162f3c4159aa0bd9326f3fb6f78" Oct 30 20:02:23.822996 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.822687 5091 scope.go:122] "RemoveContainer" containerID="4983fdc4b1de371a23c3c6442be191c3c8e9915b02f353c1aae890ba52acff53" Oct 30 20:02:23.835017 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:23.834995 5091 scope.go:122] "RemoveContainer" containerID="7dd0c666a99b8e2d36669e541fab5639b80a8710d1e3f81621ac988ad65ab123" Oct 30 20:02:24.073179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:24.067621 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 20:02:24.073179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:24.067729 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 20:02:24.113501 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:24.110417 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:24.199986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:24.199542 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 20:02:24.199986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:24.199605 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 20:02:24.941237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:24.941199 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:25.056808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:25.056765 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 20:02:25.057019 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:25.056836 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 20:02:25.246410 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:25.246322 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 20:02:25.247313 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:25.247280 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 20:02:25.297172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:25.275213 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:25.624315 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:25.602261 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:26.065769 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:26.065285 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 20:02:26.065769 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:26.065370 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 20:02:26.202180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:26.198539 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 20:02:26.202180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:26.198612 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 20:02:26.880242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:26.874866 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:27.068762 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:27.068724 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 20:02:27.069446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:27.069409 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 20:02:27.121966 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:27.121928 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:27.208696 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:27.208520 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 20:02:27.208963 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:27.208930 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 20:02:27.581201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:27.577340 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" Oct 30 20:02:27.581201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:27.578343 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" Oct 30 20:02:27.924223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:27.918601 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:28.113407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:28.113374 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 20:02:28.125616 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:28.125575 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 20:02:28.212562 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:28.212468 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 20:02:28.212562 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:28.212547 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 20:02:28.297734 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:28.297696 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:29.065605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:29.065569 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body= Oct 30 20:02:29.065897 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:29.065868 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" Oct 30 20:02:29.209902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:29.209431 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" start-of-body= Oct 30 20:02:29.209902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:29.209459 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:29.209902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:29.209488 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.255:15021/healthz/ready\": dial tcp 10.132.0.255:15021: connect: connection refused" Oct 30 20:02:30.060284 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:30.060251 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:02:30.061508 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:30.061483 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:02:30.114578 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:30.114544 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:30.205394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:30.204952 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:02:30.207945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:30.206648 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:02:30.207945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:30.207460 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:02:31.017903 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:31.017865 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:32.005171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:32.000388 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:32.960735 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:32.960699 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:33.903842 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:33.903802 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:34.082832 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:34.082783 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:35.214022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:35.213986 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:36.604411 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:36.604361 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:37.908722 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:37.908689 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc"] Oct 30 20:02:38.063168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:38.063116 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:38.348104 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:38.348065 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:39.369779 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:39.369742 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:39.680850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:39.680745 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:40.041995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.041958 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph"] Oct 30 20:02:40.096006 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.064441 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.096006 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.067275 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:40.096006 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.084996 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph"] Oct 30 20:02:40.096006 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.095220 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-969878-openshift-default-dockercfg-4jjpz\"" Oct 30 20:02:40.137434 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.137392 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-podinfo\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.137639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.137527 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-workload-certs\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.137639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.137566 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-token\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.137785 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.137714 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-envoy\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.137785 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.137741 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-credential-socket\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.137785 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.137768 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-workload-socket\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.137957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.137863 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4twgv\" (UniqueName: \"kubernetes.io/projected/f717fa94-89fa-438f-8a36-8d1a985d7454-kube-api-access-4twgv\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.137957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.137904 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f717fa94-89fa-438f-8a36-8d1a985d7454-istiod-ca-cert\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.138074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.137988 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-data\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.239373 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.239327 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-envoy\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.239568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.239394 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-credential-socket\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.239568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.239418 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-workload-socket\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.239568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.239443 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4twgv\" (UniqueName: \"kubernetes.io/projected/f717fa94-89fa-438f-8a36-8d1a985d7454-kube-api-access-4twgv\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.239568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.239466 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f717fa94-89fa-438f-8a36-8d1a985d7454-istiod-ca-cert\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.239568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.239554 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-data\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.239848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.239639 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-podinfo\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.239848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.239752 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-workload-certs\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.239848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.239824 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-token\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.239848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.239836 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-workload-socket\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.240116 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.240084 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-data\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.240394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.240371 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-workload-certs\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.240524 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.240498 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-credential-socket\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.240711 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.240674 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f717fa94-89fa-438f-8a36-8d1a985d7454-istiod-ca-cert\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.242528 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.242502 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-envoy\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.244449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.244427 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-podinfo\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.297449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.296567 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-token\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.297621 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.297572 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4twgv\" (UniqueName: \"kubernetes.io/projected/f717fa94-89fa-438f-8a36-8d1a985d7454-kube-api-access-4twgv\") pod \"e2e-extmod-b-969878-openshift-default-74f689c897-bzsph\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.515192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.486715 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:40.540664 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.540624 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:40.975227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:40.972574 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:41.204291 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:41.204252 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4"] Oct 30 20:02:41.227837 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:41.227754 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a905328-zmjp4"] Oct 30 20:02:41.290099 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:02:41.290065 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf717fa94_89fa_438f_8a36_8d1a985d7454.slice/crio-35bf66b69697e1c2ce65bbb989f08a2633e74bfa8a6d493db38a98c662f33eb2 WatchSource:0}: Error finding container 35bf66b69697e1c2ce65bbb989f08a2633e74bfa8a6d493db38a98c662f33eb2: Status 404 returned error can't find the container with id 35bf66b69697e1c2ce65bbb989f08a2633e74bfa8a6d493db38a98c662f33eb2 Oct 30 20:02:41.293132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:41.293097 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:41.293294 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:41.293198 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:41.293294 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:41.293247 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:41.294382 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:41.293978 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph"] Oct 30 20:02:41.566128 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:41.566095 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:41.587169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:41.581992 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" event={"ID":"f717fa94-89fa-438f-8a36-8d1a985d7454","Type":"ContainerStarted","Data":"e3f42e072853f68097efe15c2d0fafb0d3431177a806b5fa2d3740ef1e5201ea"} Oct 30 20:02:41.587169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:41.582057 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" event={"ID":"f717fa94-89fa-438f-8a36-8d1a985d7454","Type":"ContainerStarted","Data":"35bf66b69697e1c2ce65bbb989f08a2633e74bfa8a6d493db38a98c662f33eb2"} Oct 30 20:02:41.882889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:41.882748 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:42.504479 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:42.504429 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:42.515898 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:42.515849 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28374e9f-8887-431c-a94b-8ced74f433c6" path="/var/lib/kubelet/pods/28374e9f-8887-431c-a94b-8ced74f433c6/volumes" Oct 30 20:02:42.516697 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:42.516673 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:42.523087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:42.523067 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:42.564240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:42.564175 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" podStartSLOduration=3.564140532 podStartE2EDuration="3.564140532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:02:41.686383233 +0000 UTC m=+1881.381591195" watchObservedRunningTime="2026-10-30 20:02:42.564140532 +0000 UTC m=+1882.259348495" Oct 30 20:02:42.595699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:42.595658 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:42.596976 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:42.596949 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:02:43.802932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:43.798070 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:44.297216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:44.297123 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:44.535015 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:44.534893 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:44.778704 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:44.778663 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:45.310636 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:45.309813 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:45.592256 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:45.592177 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:45.871226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:45.868541 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:46.095947 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:02:46.095889 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:02:46.426203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:46.426132 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:46.821167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:46.821002 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:47.616245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:47.616199 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:47.825122 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:47.825031 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:47.828394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:47.828316 5091 generic.go:356] "Generic (PLEG): container finished" podID="18e11004-d729-4eec-8626-28eba4b01e90" containerID="b832152b92057244bcc5e2d9da58874ecd2cee48a3954953a6a67e39f106d133" exitCode=0 Oct 30 20:02:47.828540 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:47.828482 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" event={"ID":"18e11004-d729-4eec-8626-28eba4b01e90","Type":"ContainerDied","Data":"b832152b92057244bcc5e2d9da58874ecd2cee48a3954953a6a67e39f106d133"} Oct 30 20:02:47.829193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:47.829166 5091 scope.go:122] "RemoveContainer" containerID="b832152b92057244bcc5e2d9da58874ecd2cee48a3954953a6a67e39f106d133" Oct 30 20:02:47.831438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:47.831403 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:47.831551 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:47.831492 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:47.831551 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:47.831542 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:02:48.678171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:48.676246 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:48.839640 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:48.839602 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" event={"ID":"18e11004-d729-4eec-8626-28eba4b01e90","Type":"ContainerStarted","Data":"f2c891affe98aa1c7fdad0779c85e054d7b9c3c4044839d33366360f7a45190e"} Oct 30 20:02:49.041703 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:49.041667 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9"] Oct 30 20:02:49.041917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:49.041896 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" podUID="1babb797-d60b-4d9c-8589-27a6877d4478" containerName="istio-proxy" containerID="cri-o://51fb4e46ca7fd7c32d738320b767ac0c17e51b6d2f7b4f61a64b84d94b92b6ea" gracePeriod=30 Oct 30 20:02:49.725168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:49.723167 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:49.772170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:49.771169 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:02:49.772170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:49.771309 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 20:02:49.772170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:49.771350 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" podUID="18e11004-d729-4eec-8626-28eba4b01e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 20:02:50.771489 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:50.767483 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 20:02:50.771489 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:50.767545 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" podUID="18e11004-d729-4eec-8626-28eba4b01e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 20:02:50.924324 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:50.923122 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:51.754289 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:51.754250 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:02:51.765615 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:51.765573 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 20:02:51.765978 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:51.765945 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" podUID="18e11004-d729-4eec-8626-28eba4b01e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 20:02:51.793434 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:51.793396 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:52.064346 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:52.064242 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx"] Oct 30 20:02:52.080440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:52.080397 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cddc9f7b-qd4zx"] Oct 30 20:02:52.513341 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:52.513305 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4014c09e-3d4f-4a8d-afde-97c482579676" path="/var/lib/kubelet/pods/4014c09e-3d4f-4a8d-afde-97c482579676/volumes" Oct 30 20:02:52.711045 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:52.711008 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:52.765674 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:52.765578 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 20:02:52.766013 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:52.765983 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" podUID="18e11004-d729-4eec-8626-28eba4b01e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 20:02:52.932464 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:52.932413 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" podUID="480630c3-cfff-45e5-939d-caf7a5eaace6" containerName="main" containerID="cri-o://d9bb6bd17b2e838c7bfa6699f956bef0015c007f81c84d6e70277911d056cbff" gracePeriod=45 Oct 30 20:02:53.338501 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.338475 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 20:02:53.483249 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.482750 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-tmp-dir\") pod \"480630c3-cfff-45e5-939d-caf7a5eaace6\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " Oct 30 20:02:53.483249 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.482905 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-model-cache\") pod \"480630c3-cfff-45e5-939d-caf7a5eaace6\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " Oct 30 20:02:53.483249 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.483007 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-dshm\") pod \"480630c3-cfff-45e5-939d-caf7a5eaace6\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " Oct 30 20:02:53.483249 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.483059 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/480630c3-cfff-45e5-939d-caf7a5eaace6-kube-api-access-x8c4z\" (UniqueName: \"kubernetes.io/projected/480630c3-cfff-45e5-939d-caf7a5eaace6-kube-api-access-x8c4z\") pod \"480630c3-cfff-45e5-939d-caf7a5eaace6\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " Oct 30 20:02:53.483633 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.483527 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-model-cache" pod "480630c3-cfff-45e5-939d-caf7a5eaace6" (UID: "480630c3-cfff-45e5-939d-caf7a5eaace6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:53.483724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.483697 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-tmp-dir" pod "480630c3-cfff-45e5-939d-caf7a5eaace6" (UID: "480630c3-cfff-45e5-939d-caf7a5eaace6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:53.483793 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.483702 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-home\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-home\") pod \"480630c3-cfff-45e5-939d-caf7a5eaace6\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " Oct 30 20:02:53.483793 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.483775 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/480630c3-cfff-45e5-939d-caf7a5eaace6-tls-certs\" (UniqueName: \"kubernetes.io/secret/480630c3-cfff-45e5-939d-caf7a5eaace6-tls-certs\") pod \"480630c3-cfff-45e5-939d-caf7a5eaace6\" (UID: \"480630c3-cfff-45e5-939d-caf7a5eaace6\") " Oct 30 20:02:53.483975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.483954 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-home" pod "480630c3-cfff-45e5-939d-caf7a5eaace6" (UID: "480630c3-cfff-45e5-939d-caf7a5eaace6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:53.484678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.484652 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-home\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:53.484846 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.484831 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-tmp-dir\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:53.484944 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.484933 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-model-cache\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:53.486382 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.486359 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480630c3-cfff-45e5-939d-caf7a5eaace6-kube-api-access-x8c4z" pod "480630c3-cfff-45e5-939d-caf7a5eaace6" (UID: "480630c3-cfff-45e5-939d-caf7a5eaace6"). InnerVolumeSpecName "kube-api-access-x8c4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:53.486502 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.486475 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-dshm" pod "480630c3-cfff-45e5-939d-caf7a5eaace6" (UID: "480630c3-cfff-45e5-939d-caf7a5eaace6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:53.486776 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.486751 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/480630c3-cfff-45e5-939d-caf7a5eaace6-tls-certs" pod "480630c3-cfff-45e5-939d-caf7a5eaace6" (UID: "480630c3-cfff-45e5-939d-caf7a5eaace6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:02:53.587399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.587315 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/480630c3-cfff-45e5-939d-caf7a5eaace6-dshm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:53.587399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.587347 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8c4z\" (UniqueName: \"kubernetes.io/projected/480630c3-cfff-45e5-939d-caf7a5eaace6-kube-api-access-x8c4z\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:53.587399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.587363 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/480630c3-cfff-45e5-939d-caf7a5eaace6-tls-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:53.702170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.701288 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:53.775629 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.775576 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" start-of-body= Oct 30 20:02:53.775829 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.775717 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" podUID="18e11004-d729-4eec-8626-28eba4b01e90" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.4:15021/healthz/ready\": dial tcp 10.132.1.4:15021: connect: connection refused" Oct 30 20:02:53.918377 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.918279 5091 generic.go:356] "Generic (PLEG): container finished" podID="480630c3-cfff-45e5-939d-caf7a5eaace6" containerID="d9bb6bd17b2e838c7bfa6699f956bef0015c007f81c84d6e70277911d056cbff" exitCode=0 Oct 30 20:02:53.918377 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.918333 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" Oct 30 20:02:53.918377 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.918344 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" event={"ID":"480630c3-cfff-45e5-939d-caf7a5eaace6","Type":"ContainerDied","Data":"d9bb6bd17b2e838c7bfa6699f956bef0015c007f81c84d6e70277911d056cbff"} Oct 30 20:02:53.918669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.918390 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc" event={"ID":"480630c3-cfff-45e5-939d-caf7a5eaace6","Type":"ContainerDied","Data":"2e0458354479178082018d37a237192d98d54fedc65cfaf6238d37b8d80d1108"} Oct 30 20:02:53.918669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.918411 5091 scope.go:122] "RemoveContainer" containerID="d9bb6bd17b2e838c7bfa6699f956bef0015c007f81c84d6e70277911d056cbff" Oct 30 20:02:53.930556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.930536 5091 scope.go:122] "RemoveContainer" containerID="d9bb6bd17b2e838c7bfa6699f956bef0015c007f81c84d6e70277911d056cbff" Oct 30 20:02:53.930815 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:02:53.930800 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9bb6bd17b2e838c7bfa6699f956bef0015c007f81c84d6e70277911d056cbff\": container with ID starting with d9bb6bd17b2e838c7bfa6699f956bef0015c007f81c84d6e70277911d056cbff not found: ID does not exist" containerID="d9bb6bd17b2e838c7bfa6699f956bef0015c007f81c84d6e70277911d056cbff" Oct 30 20:02:53.930856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.930823 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9bb6bd17b2e838c7bfa6699f956bef0015c007f81c84d6e70277911d056cbff"} err="failed to get container status \"d9bb6bd17b2e838c7bfa6699f956bef0015c007f81c84d6e70277911d056cbff\": rpc error: code = NotFound desc = could not find container \"d9bb6bd17b2e838c7bfa6699f956bef0015c007f81c84d6e70277911d056cbff\": container with ID starting with d9bb6bd17b2e838c7bfa6699f956bef0015c007f81c84d6e70277911d056cbff not found: ID does not exist" Oct 30 20:02:53.977226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.973191 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc"] Oct 30 20:02:53.982301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:53.982270 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7cbcbb/auth-test-model-7cbcbb-kserve-5498db68df-hcsrc"] Oct 30 20:02:54.400508 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.400478 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 20:02:54.445926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.445850 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="480630c3-cfff-45e5-939d-caf7a5eaace6" path="/var/lib/kubelet/pods/480630c3-cfff-45e5-939d-caf7a5eaace6/volumes" Oct 30 20:02:54.505268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.505236 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-credential-socket\") pod \"1babb797-d60b-4d9c-8589-27a6877d4478\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " Oct 30 20:02:54.505268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.505272 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1babb797-d60b-4d9c-8589-27a6877d4478-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1babb797-d60b-4d9c-8589-27a6877d4478-istiod-ca-cert\") pod \"1babb797-d60b-4d9c-8589-27a6877d4478\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " Oct 30 20:02:54.505555 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.505300 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1babb797-d60b-4d9c-8589-27a6877d4478-kube-api-access-scf9h\" (UniqueName: \"kubernetes.io/projected/1babb797-d60b-4d9c-8589-27a6877d4478-kube-api-access-scf9h\") pod \"1babb797-d60b-4d9c-8589-27a6877d4478\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " Oct 30 20:02:54.505555 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.505366 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-workload-certs\") pod \"1babb797-d60b-4d9c-8589-27a6877d4478\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " Oct 30 20:02:54.505555 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.505387 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1babb797-d60b-4d9c-8589-27a6877d4478-istio-token\" (UniqueName: \"kubernetes.io/projected/1babb797-d60b-4d9c-8589-27a6877d4478-istio-token\") pod \"1babb797-d60b-4d9c-8589-27a6877d4478\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " Oct 30 20:02:54.505555 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.505437 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-istio-data\") pod \"1babb797-d60b-4d9c-8589-27a6877d4478\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " Oct 30 20:02:54.505555 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.505493 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1babb797-d60b-4d9c-8589-27a6877d4478-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1babb797-d60b-4d9c-8589-27a6877d4478-istio-podinfo\") pod \"1babb797-d60b-4d9c-8589-27a6877d4478\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " Oct 30 20:02:54.505555 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.505549 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-istio-envoy\") pod \"1babb797-d60b-4d9c-8589-27a6877d4478\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " Oct 30 20:02:54.505915 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.505629 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-credential-socket" pod "1babb797-d60b-4d9c-8589-27a6877d4478" (UID: "1babb797-d60b-4d9c-8589-27a6877d4478"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:54.505915 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.505747 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-workload-socket\") pod \"1babb797-d60b-4d9c-8589-27a6877d4478\" (UID: \"1babb797-d60b-4d9c-8589-27a6877d4478\") " Oct 30 20:02:54.505915 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.505768 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1babb797-d60b-4d9c-8589-27a6877d4478-istiod-ca-cert" pod "1babb797-d60b-4d9c-8589-27a6877d4478" (UID: "1babb797-d60b-4d9c-8589-27a6877d4478"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:02:54.506508 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.506417 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-workload-certs" pod "1babb797-d60b-4d9c-8589-27a6877d4478" (UID: "1babb797-d60b-4d9c-8589-27a6877d4478"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:54.506508 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.506477 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-istio-data" pod "1babb797-d60b-4d9c-8589-27a6877d4478" (UID: "1babb797-d60b-4d9c-8589-27a6877d4478"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:54.506773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.506567 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-workload-socket" pod "1babb797-d60b-4d9c-8589-27a6877d4478" (UID: "1babb797-d60b-4d9c-8589-27a6877d4478"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:54.506773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.506659 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:54.506773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.506673 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1babb797-d60b-4d9c-8589-27a6877d4478-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:54.506773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.506682 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:54.506773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.506691 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:54.509051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.509028 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-istio-envoy" pod "1babb797-d60b-4d9c-8589-27a6877d4478" (UID: "1babb797-d60b-4d9c-8589-27a6877d4478"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:02:54.509433 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.509411 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1babb797-d60b-4d9c-8589-27a6877d4478-istio-token" pod "1babb797-d60b-4d9c-8589-27a6877d4478" (UID: "1babb797-d60b-4d9c-8589-27a6877d4478"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:54.509667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.509650 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1babb797-d60b-4d9c-8589-27a6877d4478-istio-podinfo" pod "1babb797-d60b-4d9c-8589-27a6877d4478" (UID: "1babb797-d60b-4d9c-8589-27a6877d4478"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:02:54.509889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.509874 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1babb797-d60b-4d9c-8589-27a6877d4478-kube-api-access-scf9h" pod "1babb797-d60b-4d9c-8589-27a6877d4478" (UID: "1babb797-d60b-4d9c-8589-27a6877d4478"). InnerVolumeSpecName "kube-api-access-scf9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:02:54.620225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.597478 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:54.620225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.614831 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:54.620225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.614863 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scf9h\" (UniqueName: \"kubernetes.io/projected/1babb797-d60b-4d9c-8589-27a6877d4478-kube-api-access-scf9h\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:54.620225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.614883 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1babb797-d60b-4d9c-8589-27a6877d4478-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:54.620225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.614897 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1babb797-d60b-4d9c-8589-27a6877d4478-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:54.620225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.614912 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1babb797-d60b-4d9c-8589-27a6877d4478-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:02:54.771464 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.771434 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:02:54.772664 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.772640 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:02:54.794060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.794018 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:54.930550 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.930501 5091 generic.go:356] "Generic (PLEG): container finished" podID="1babb797-d60b-4d9c-8589-27a6877d4478" containerID="51fb4e46ca7fd7c32d738320b767ac0c17e51b6d2f7b4f61a64b84d94b92b6ea" exitCode=0 Oct 30 20:02:54.930550 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.930550 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" Oct 30 20:02:54.930819 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.930559 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" event={"ID":"1babb797-d60b-4d9c-8589-27a6877d4478","Type":"ContainerDied","Data":"51fb4e46ca7fd7c32d738320b767ac0c17e51b6d2f7b4f61a64b84d94b92b6ea"} Oct 30 20:02:54.930819 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.930609 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9" event={"ID":"1babb797-d60b-4d9c-8589-27a6877d4478","Type":"ContainerDied","Data":"2eab06bbd24c9edd8cd139919ec456c61dbad57d14745753143f2a2dc86745a4"} Oct 30 20:02:54.930819 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.930637 5091 scope.go:122] "RemoveContainer" containerID="51fb4e46ca7fd7c32d738320b767ac0c17e51b6d2f7b4f61a64b84d94b92b6ea" Oct 30 20:02:54.943028 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.943009 5091 scope.go:122] "RemoveContainer" containerID="51fb4e46ca7fd7c32d738320b767ac0c17e51b6d2f7b4f61a64b84d94b92b6ea" Oct 30 20:02:54.943331 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:02:54.943308 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51fb4e46ca7fd7c32d738320b767ac0c17e51b6d2f7b4f61a64b84d94b92b6ea\": container with ID starting with 51fb4e46ca7fd7c32d738320b767ac0c17e51b6d2f7b4f61a64b84d94b92b6ea not found: ID does not exist" containerID="51fb4e46ca7fd7c32d738320b767ac0c17e51b6d2f7b4f61a64b84d94b92b6ea" Oct 30 20:02:54.943424 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.943335 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51fb4e46ca7fd7c32d738320b767ac0c17e51b6d2f7b4f61a64b84d94b92b6ea"} err="failed to get container status \"51fb4e46ca7fd7c32d738320b767ac0c17e51b6d2f7b4f61a64b84d94b92b6ea\": rpc error: code = NotFound desc = could not find container \"51fb4e46ca7fd7c32d738320b767ac0c17e51b6d2f7b4f61a64b84d94b92b6ea\": container with ID starting with 51fb4e46ca7fd7c32d738320b767ac0c17e51b6d2f7b4f61a64b84d94b92b6ea not found: ID does not exist" Oct 30 20:02:54.970504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.970471 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9"] Oct 30 20:02:54.976867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:54.976841 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-4pzf9"] Oct 30 20:02:55.599172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:55.598210 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:55.785849 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:55.785817 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:56.443823 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:56.443795 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1babb797-d60b-4d9c-8589-27a6877d4478" path="/var/lib/kubelet/pods/1babb797-d60b-4d9c-8589-27a6877d4478/volumes" Oct 30 20:02:56.807172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:56.805888 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:57.039168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:57.034226 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:57.216787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:57.216691 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:57.414976 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:57.414923 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:57.625085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:57.625030 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:58.088808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:58.088769 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:58.296042 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:58.296005 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:58.917171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:58.907652 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:58.984172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:58.967204 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq"] Oct 30 20:02:59.005170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:58.999456 5091 cpu_manager.go:414] "containerMap: removing container" podUID="480630c3-cfff-45e5-939d-caf7a5eaace6" containerName="main" Oct 30 20:02:59.005170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:58.999487 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="480630c3-cfff-45e5-939d-caf7a5eaace6" containerName="main" Oct 30 20:02:59.005170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:58.999558 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1babb797-d60b-4d9c-8589-27a6877d4478" containerName="istio-proxy" Oct 30 20:02:59.005170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:58.999568 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1babb797-d60b-4d9c-8589-27a6877d4478" containerName="istio-proxy" Oct 30 20:02:59.005170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:58.999776 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="480630c3-cfff-45e5-939d-caf7a5eaace6" containerName="main" Oct 30 20:02:59.005170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:58.999800 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1babb797-d60b-4d9c-8589-27a6877d4478" containerName="istio-proxy" Oct 30 20:02:59.005170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.004069 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq"] Oct 30 20:02:59.005170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.004201 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:02:59.029974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.029453 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"agc-cred-11a237dee94d\"" Oct 30 20:02:59.029974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.029547 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d97cc8\"" Oct 30 20:02:59.029974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.029650 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"kube-root-ca.crt\"" Oct 30 20:02:59.029974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.029736 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"openshift-service-ca.crt\"" Oct 30 20:02:59.029974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.029790 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"routing-overlay\"" Oct 30 20:02:59.029974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.029905 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"payload-processing-external-model-e2e--151804b4-dockercfg-zvjcn\"" Oct 30 20:02:59.137837 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.137805 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99dbd70-1fbe-48e7-ba22-011eed682112-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:02:59.138322 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.137976 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a99dbd70-1fbe-48e7-ba22-011eed682112-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:02:59.138322 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.138186 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a99dbd70-1fbe-48e7-ba22-011eed682112-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:02:59.138322 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.138268 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hd2w\" (UniqueName: \"kubernetes.io/projected/a99dbd70-1fbe-48e7-ba22-011eed682112-kube-api-access-5hd2w\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:02:59.240200 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.240073 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a99dbd70-1fbe-48e7-ba22-011eed682112-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:02:59.240358 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.240335 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a99dbd70-1fbe-48e7-ba22-011eed682112-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:02:59.240421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.240404 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hd2w\" (UniqueName: \"kubernetes.io/projected/a99dbd70-1fbe-48e7-ba22-011eed682112-kube-api-access-5hd2w\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:02:59.240507 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.240491 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99dbd70-1fbe-48e7-ba22-011eed682112-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:02:59.240781 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.240756 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a99dbd70-1fbe-48e7-ba22-011eed682112-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:02:59.241231 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.241210 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99dbd70-1fbe-48e7-ba22-011eed682112-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:02:59.243046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.243017 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a99dbd70-1fbe-48e7-ba22-011eed682112-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:02:59.257523 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.257501 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hd2w\" (UniqueName: \"kubernetes.io/projected/a99dbd70-1fbe-48e7-ba22-011eed682112-kube-api-access-5hd2w\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:02:59.364173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.355065 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:02:59.422993 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.422956 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:02:59.905486 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:02:59.905449 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda99dbd70_1fbe_48e7_ba22_011eed682112.slice/crio-28d483f3a1b83f6c888bf750e9be3a84faedd922e95a8b770c1d39f46a452fb7 WatchSource:0}: Error finding container 28d483f3a1b83f6c888bf750e9be3a84faedd922e95a8b770c1d39f46a452fb7: Status 404 returned error can't find the container with id 28d483f3a1b83f6c888bf750e9be3a84faedd922e95a8b770c1d39f46a452fb7 Oct 30 20:02:59.905846 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:02:59.905821 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq"] Oct 30 20:03:00.055793 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:00.055754 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" event={"ID":"a99dbd70-1fbe-48e7-ba22-011eed682112","Type":"ContainerStarted","Data":"96c413cc2f5da02dd5c231ebbf0c0b55be5106486fdf62efebfdfeaf8148c99a"} Oct 30 20:03:00.055979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:00.055818 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" event={"ID":"a99dbd70-1fbe-48e7-ba22-011eed682112","Type":"ContainerStarted","Data":"28d483f3a1b83f6c888bf750e9be3a84faedd922e95a8b770c1d39f46a452fb7"} Oct 30 20:03:00.055979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:00.055846 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:03:00.082707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:00.082660 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" podStartSLOduration=2.082646443 podStartE2EDuration="2.082646443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:00.079900455 +0000 UTC m=+1899.775108415" watchObservedRunningTime="2026-10-30 20:03:00.082646443 +0000 UTC m=+1899.777854400" Oct 30 20:03:00.537181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:00.525084 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:01.394818 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:01.394781 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:02.184161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:02.184107 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:03.111316 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:03.111271 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:03.310511 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:03.310473 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:04.049952 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:04.049870 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:04.727546 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:04.727505 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:05.066790 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.066753 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:05.156286 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.156250 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx"] Oct 30 20:03:05.161487 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.161464 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.184228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.184198 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-4d5758-openshift-default-dockercfg-tmq2l\"" Oct 30 20:03:05.214435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.214391 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx"] Oct 30 20:03:05.385520 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.345056 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-envoy\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.406194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.400331 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-workload-certs\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.406194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.400438 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-token\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.415352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.415311 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef039607-7c00-4a64-a63b-5e4a709ae26c-istiod-ca-cert\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.423165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.415645 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-credential-socket\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.429240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.423529 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-data\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.444212 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.429519 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmzzl\" (UniqueName: \"kubernetes.io/projected/ef039607-7c00-4a64-a63b-5e4a709ae26c-kube-api-access-xmzzl\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.444555 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.444478 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-workload-socket\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.460245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.444817 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-podinfo\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.531303 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.529986 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:05.562240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.562180 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmzzl\" (UniqueName: \"kubernetes.io/projected/ef039607-7c00-4a64-a63b-5e4a709ae26c-kube-api-access-xmzzl\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.562451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.562244 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-workload-socket\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.562451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.562331 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-podinfo\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.562451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.562394 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-envoy\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.562632 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.562499 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-workload-certs\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.562632 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.562569 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-token\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.562632 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.562622 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef039607-7c00-4a64-a63b-5e4a709ae26c-istiod-ca-cert\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.562805 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.562671 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-credential-socket\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.562864 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.562836 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-data\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.563394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.563364 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-data\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.588180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.564915 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-workload-certs\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.588421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.565463 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-workload-socket\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.588530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.566288 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef039607-7c00-4a64-a63b-5e4a709ae26c-istiod-ca-cert\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.588644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.566689 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-credential-socket\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.595218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.592483 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmzzl\" (UniqueName: \"kubernetes.io/projected/ef039607-7c00-4a64-a63b-5e4a709ae26c-kube-api-access-xmzzl\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.642341 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.642275 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-token\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.643172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.643119 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-podinfo\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.644665 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.644639 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-envoy\") pod \"e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:05.771857 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.771809 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:05.776645 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:05.776611 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:06.081701 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:06.081672 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx"] Oct 30 20:03:06.082699 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:03:06.082673 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef039607_7c00_4a64_a63b_5e4a709ae26c.slice/crio-3a13f9385940f2543ac6b56bf34f009dddffa6bac706238e5ed756281f245b36 WatchSource:0}: Error finding container 3a13f9385940f2543ac6b56bf34f009dddffa6bac706238e5ed756281f245b36: Status 404 returned error can't find the container with id 3a13f9385940f2543ac6b56bf34f009dddffa6bac706238e5ed756281f245b36 Oct 30 20:03:06.086038 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:06.086006 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:03:06.086129 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:06.086070 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:03:06.086129 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:06.086099 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:03:06.188041 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:06.188010 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" event={"ID":"ef039607-7c00-4a64-a63b-5e4a709ae26c","Type":"ContainerStarted","Data":"3a13f9385940f2543ac6b56bf34f009dddffa6bac706238e5ed756281f245b36"} Oct 30 20:03:06.250776 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:06.250741 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:06.413071 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:06.412950 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:06.658850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:06.658786 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:07.303314 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:07.303235 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" event={"ID":"ef039607-7c00-4a64-a63b-5e4a709ae26c","Type":"ContainerStarted","Data":"9a9a889a72a06c51fdd605d87b56053127dfc54a5538c1125c7d684012e9b7fe"} Oct 30 20:03:07.365179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:07.362547 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" podStartSLOduration=2.362527832 podStartE2EDuration="2.362527832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:07.361414023 +0000 UTC m=+1907.056621981" watchObservedRunningTime="2026-10-30 20:03:07.362527832 +0000 UTC m=+1907.057735790" Oct 30 20:03:07.491974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:07.491926 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:07.675466 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:07.675376 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:07.791224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:07.791193 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:07.902724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:07.902684 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:08.066170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:08.064167 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:08.472710 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:08.471579 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:08.472710 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:08.471693 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:03:08.695883 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:08.695830 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:09.806158 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:09.806104 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:10.069020 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:10.068921 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:10.308819 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:10.308784 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:11.079172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:11.074237 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:11.097172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:11.096754 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:03:11.380947 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:11.380849 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:11.490378 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:11.487270 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6"] Oct 30 20:03:11.494616 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:11.494589 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" Oct 30 20:03:11.498954 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:11.497454 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6"] Oct 30 20:03:11.535572 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:11.535542 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-4d5758-dockercfg-jdrbp\"" Oct 30 20:03:11.535890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:11.535867 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-4d5758\"" Oct 30 20:03:11.658060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:11.657962 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:11.670498 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:11.670463 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4pcd\" (UniqueName: \"kubernetes.io/projected/a60177f4-e29b-47ea-b5cd-3068808eb97f-kube-api-access-p4pcd\") pod \"payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6\" (UID: \"a60177f4-e29b-47ea-b5cd-3068808eb97f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" Oct 30 20:03:11.671011 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:11.670983 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60177f4-e29b-47ea-b5cd-3068808eb97f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6\" (UID: \"a60177f4-e29b-47ea-b5cd-3068808eb97f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" Oct 30 20:03:11.781700 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:11.781653 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60177f4-e29b-47ea-b5cd-3068808eb97f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6\" (UID: \"a60177f4-e29b-47ea-b5cd-3068808eb97f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" Oct 30 20:03:11.782995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:11.782862 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60177f4-e29b-47ea-b5cd-3068808eb97f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6\" (UID: \"a60177f4-e29b-47ea-b5cd-3068808eb97f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" Oct 30 20:03:11.783355 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:11.783332 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4pcd\" (UniqueName: \"kubernetes.io/projected/a60177f4-e29b-47ea-b5cd-3068808eb97f-kube-api-access-p4pcd\") pod \"payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6\" (UID: \"a60177f4-e29b-47ea-b5cd-3068808eb97f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" Oct 30 20:03:11.847397 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:11.845485 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4pcd\" (UniqueName: \"kubernetes.io/projected/a60177f4-e29b-47ea-b5cd-3068808eb97f-kube-api-access-p4pcd\") pod \"payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6\" (UID: \"a60177f4-e29b-47ea-b5cd-3068808eb97f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" Oct 30 20:03:12.121353 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:12.121320 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" Oct 30 20:03:12.416230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:12.408223 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:12.511553 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:03:12.511525 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda60177f4_e29b_47ea_b5cd_3068808eb97f.slice/crio-e0891d4f44aa9b716011bbe727463f0d6cf2f2b625d2fcd24ceb7af433dc9b05 WatchSource:0}: Error finding container e0891d4f44aa9b716011bbe727463f0d6cf2f2b625d2fcd24ceb7af433dc9b05: Status 404 returned error can't find the container with id e0891d4f44aa9b716011bbe727463f0d6cf2f2b625d2fcd24ceb7af433dc9b05 Oct 30 20:03:12.513122 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:12.512912 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6"] Oct 30 20:03:13.536032 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:13.535996 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" event={"ID":"a60177f4-e29b-47ea-b5cd-3068808eb97f","Type":"ContainerStarted","Data":"a592671d365bea5105d7ffbe294a02675cf26c6c717bdf0eac13025f7254dd91"} Oct 30 20:03:13.536633 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:13.536597 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" event={"ID":"a60177f4-e29b-47ea-b5cd-3068808eb97f","Type":"ContainerStarted","Data":"e0891d4f44aa9b716011bbe727463f0d6cf2f2b625d2fcd24ceb7af433dc9b05"} Oct 30 20:03:13.536794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:13.536780 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" Oct 30 20:03:13.607169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:13.599344 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" podStartSLOduration=2.599320133 podStartE2EDuration="2.599320133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:13.574627927 +0000 UTC m=+1913.269835890" watchObservedRunningTime="2026-10-30 20:03:13.599320133 +0000 UTC m=+1913.294528092" Oct 30 20:03:13.634218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:13.631016 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:13.816451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:13.816370 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:14.406652 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:14.406603 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:14.749260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:14.749223 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:15.832220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:15.832180 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:15.967260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:15.967129 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg"] Oct 30 20:03:15.974372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:15.974341 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:15.996220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:15.995221 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-836d67\"/\"openshift-service-ca.crt\"" Oct 30 20:03:15.996220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:15.995386 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-836d67\"/\"kube-root-ca.crt\"" Oct 30 20:03:15.996220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:15.995484 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-836d67\"/\"default-dockercfg-crbxx\"" Oct 30 20:03:15.997343 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:15.997321 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-836d67\"/\"autores-model-836d67-kserve-self-signed-certs\"" Oct 30 20:03:15.999819 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:03:15.999788 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:03:16.004886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.001900 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg"] Oct 30 20:03:16.078220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.078137 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhltm\" (UniqueName: \"kubernetes.io/projected/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-kube-api-access-rhltm\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.078418 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.078324 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-model-cache\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.078418 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.078368 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-tls-certs\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.078586 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.078559 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-tmp-dir\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.078644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.078603 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-home\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.078727 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.078708 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-dshm\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.086350 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.086267 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:16.207165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.192184 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-home\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.207165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.192312 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-dshm\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.207165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.192443 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhltm\" (UniqueName: \"kubernetes.io/projected/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-kube-api-access-rhltm\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.207165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.192541 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-model-cache\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.207165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.192570 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-tls-certs\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.207165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.192747 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-tmp-dir\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.207165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.193457 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-tmp-dir\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.207165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.193740 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-home\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.207165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.194337 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-model-cache\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.217173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.210161 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-tls-certs\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.217173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.210451 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-dshm\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.244171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.233838 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhltm\" (UniqueName: \"kubernetes.io/projected/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-kube-api-access-rhltm\") pod \"autores-model-836d67-kserve-744fb9cf49-4w7dg\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.303167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.301405 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:16.644848 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:03:16.644811 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod962d02d2_4b11_4cee_b02e_6e0920ebaa7b.slice/crio-b491c97bb30957a5b1a0c63503caad94bcae1e244a23b5f04c869945be239fb3 WatchSource:0}: Error finding container b491c97bb30957a5b1a0c63503caad94bcae1e244a23b5f04c869945be239fb3: Status 404 returned error can't find the container with id b491c97bb30957a5b1a0c63503caad94bcae1e244a23b5f04c869945be239fb3 Oct 30 20:03:16.647690 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.647663 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg"] Oct 30 20:03:16.938403 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:16.938207 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:17.577041 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:17.576998 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" event={"ID":"962d02d2-4b11-4cee-b02e-6e0920ebaa7b","Type":"ContainerStarted","Data":"038c6df61e44589e387a48da8daaf7ea834340170aebd512a75482ca361631c7"} Oct 30 20:03:17.577041 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:17.577045 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" event={"ID":"962d02d2-4b11-4cee-b02e-6e0920ebaa7b","Type":"ContainerStarted","Data":"b491c97bb30957a5b1a0c63503caad94bcae1e244a23b5f04c869945be239fb3"} Oct 30 20:03:17.606937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:17.606878 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" podStartSLOduration=2.606865925 podStartE2EDuration="2.606865925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:17.602798783 +0000 UTC m=+1917.298006741" watchObservedRunningTime="2026-10-30 20:03:17.606865925 +0000 UTC m=+1917.302073882" Oct 30 20:03:17.964274 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:17.963954 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:18.302472 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:18.302438 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:18.969613 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:18.969571 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:19.600753 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:19.600665 5091 generic.go:356] "Generic (PLEG): container finished" podID="632cf328-468e-4865-a6ef-7358d679cb6d" containerID="05cc5cc5c831bc8c6a3ee2bf75528e70eb90d375b85397a00f6aec2e88cfd762" exitCode=0 Oct 30 20:03:19.600983 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:19.600740 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" event={"ID":"632cf328-468e-4865-a6ef-7358d679cb6d","Type":"ContainerDied","Data":"05cc5cc5c831bc8c6a3ee2bf75528e70eb90d375b85397a00f6aec2e88cfd762"} Oct 30 20:03:19.601310 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:19.601290 5091 scope.go:122] "RemoveContainer" containerID="05cc5cc5c831bc8c6a3ee2bf75528e70eb90d375b85397a00f6aec2e88cfd762" Oct 30 20:03:19.603038 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:19.603011 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:03:19.603098 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:19.603081 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:03:19.603138 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:19.603116 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:03:20.121341 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:20.121300 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:20.611458 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:20.611420 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" event={"ID":"632cf328-468e-4865-a6ef-7358d679cb6d","Type":"ContainerStarted","Data":"53d6769622928252d297348fee8c82c8081b9621cdecfe411dd7a386330132ad"} Oct 30 20:03:21.107169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:21.106844 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:21.164193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:21.160560 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:03:21.164193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:21.160733 5091 patch_prober.go:29] interesting pod/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 20:03:21.164193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:21.160786 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" podUID="632cf328-468e-4865-a6ef-7358d679cb6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 20:03:21.778580 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:21.778544 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:03:22.169239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:22.160550 5091 patch_prober.go:29] interesting pod/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 20:03:22.169239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:22.160642 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" podUID="632cf328-468e-4865-a6ef-7358d679cb6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 20:03:22.180221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:22.171114 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:23.162575 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:23.162533 5091 patch_prober.go:29] interesting pod/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 20:03:23.162898 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:23.162868 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" podUID="632cf328-468e-4865-a6ef-7358d679cb6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 20:03:23.202073 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:23.202035 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:24.147175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:24.146608 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:24.175170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:24.172706 5091 patch_prober.go:29] interesting pod/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 20:03:24.175170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:24.172771 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" podUID="632cf328-468e-4865-a6ef-7358d679cb6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 20:03:24.257169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:24.247304 5091 scope.go:122] "RemoveContainer" containerID="e7b03ad774680fa3712dbd4480b3a4fac8f5d21c8191ca2470b971f08a1b1fe3" Oct 30 20:03:24.298022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:24.297989 5091 scope.go:122] "RemoveContainer" containerID="79b5de0ceafbe42a7641e12bf794886aad7b4adfc4fa3f87a8853838f54cef2a" Oct 30 20:03:24.308643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:24.308608 5091 scope.go:122] "RemoveContainer" containerID="43d6f39613b6a59b4bfaaf9d734ef10d483c144ec20cb5b0d204c1f75a7ec63f" Oct 30 20:03:24.582438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:24.582403 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" Oct 30 20:03:25.162440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:25.161890 5091 patch_prober.go:29] interesting pod/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 20:03:25.162440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:25.161966 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" podUID="632cf328-468e-4865-a6ef-7358d679cb6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 20:03:25.270937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:25.270900 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:26.166122 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:26.166083 5091 patch_prober.go:29] interesting pod/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Oct 30 20:03:26.166475 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:26.166440 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" podUID="632cf328-468e-4865-a6ef-7358d679cb6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Oct 30 20:03:26.291001 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:26.290961 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:26.313169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:26.312687 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:26.370356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:26.370322 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:26.709652 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:26.709621 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:27.171301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:27.170241 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:27.203678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:27.203646 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:03:27.206596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:27.206569 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:03:28.102757 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:28.102718 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:28.971168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:28.966765 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:29.905172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:29.898637 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:30.913854 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:30.913632 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:32.072729 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.072693 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:32.386998 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.386905 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg"] Oct 30 20:03:32.415509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.415120 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:32.535603 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.535567 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s"] Oct 30 20:03:32.627377 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.627330 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s"] Oct 30 20:03:32.627539 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.627467 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" Oct 30 20:03:32.638616 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.638553 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:03:32.638790 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.638581 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:03:32.828814 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.828767 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klwdr\" (UniqueName: \"kubernetes.io/projected/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-kube-api-access-klwdr\") pod \"maas-api-revoke-keys-e2e-autores-836d67-q4r8s\" (UID: \"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" Oct 30 20:03:32.829007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.828981 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-836d67-q4r8s\" (UID: \"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" Oct 30 20:03:32.829202 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.829172 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-836d67-q4r8s\" (UID: \"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" Oct 30 20:03:32.931303 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.931197 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klwdr\" (UniqueName: \"kubernetes.io/projected/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-kube-api-access-klwdr\") pod \"maas-api-revoke-keys-e2e-autores-836d67-q4r8s\" (UID: \"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" Oct 30 20:03:32.931494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.931346 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-836d67-q4r8s\" (UID: \"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" Oct 30 20:03:32.931494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.931473 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-836d67-q4r8s\" (UID: \"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" Oct 30 20:03:32.932591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.932566 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-836d67-q4r8s\" (UID: \"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" Oct 30 20:03:32.937083 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.937059 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-836d67-q4r8s\" (UID: \"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" Oct 30 20:03:32.979171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:32.968249 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klwdr\" (UniqueName: \"kubernetes.io/projected/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-kube-api-access-klwdr\") pod \"maas-api-revoke-keys-e2e-autores-836d67-q4r8s\" (UID: \"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" Oct 30 20:03:33.243541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:33.243454 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" Oct 30 20:03:33.353726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:33.353624 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:33.919807 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:33.919780 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s"] Oct 30 20:03:34.695614 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:34.695522 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:34.923167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:34.922424 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" event={"ID":"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8","Type":"ContainerStarted","Data":"7ba9a28061d1b343d1ac6f301b21f7757e94553749a6ad7b389e038fca168473"} Oct 30 20:03:34.923167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:34.922522 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" event={"ID":"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8","Type":"ContainerStarted","Data":"96ef6d7c4a4d05e3dfa33b59ab9321e8d5c28631f96668fb4d3646a78371573c"} Oct 30 20:03:35.020174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:35.019998 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" podStartSLOduration=3.019978379 podStartE2EDuration="3.019978379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:34.970384459 +0000 UTC m=+1934.665592417" watchObservedRunningTime="2026-10-30 20:03:35.019978379 +0000 UTC m=+1934.715186338" Oct 30 20:03:35.033900 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:35.033866 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:35.218962 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:35.218925 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:35.289430 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:35.288922 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z"] Oct 30 20:03:35.289430 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:35.289177 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" podUID="892edc88-bb54-46e1-acd1-701313b9ae0a" containerName="istio-proxy" containerID="cri-o://2a197d207774acfbab864b95083dd2da8774ad0887239e372d2438fdbdb68b77" gracePeriod=30 Oct 30 20:03:35.437074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:35.437026 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:35.628283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:35.628194 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:35.932875 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:35.932780 5091 generic.go:356] "Generic (PLEG): container finished" podID="8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8" containerID="7ba9a28061d1b343d1ac6f301b21f7757e94553749a6ad7b389e038fca168473" exitCode=0 Oct 30 20:03:35.933286 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:35.932867 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" event={"ID":"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8","Type":"ContainerDied","Data":"7ba9a28061d1b343d1ac6f301b21f7757e94553749a6ad7b389e038fca168473"} Oct 30 20:03:36.040306 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:36.031275 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:36.708112 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:36.708073 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:36.905260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:36.905223 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:37.142296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:37.142200 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:37.466223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:37.462807 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:37.857272 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:37.845182 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn"] Oct 30 20:03:37.919738 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:37.910692 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:37.988168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:37.971455 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-2a2286-openshift-default-dockercfg-grjl9\"" Oct 30 20:03:38.108223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.065873 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" Oct 30 20:03:38.108537 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.106751 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:38.204977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.204925 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-maas-api-xks-ca\") pod \"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8\" (UID: \"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8\") " Oct 30 20:03:38.205942 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.205915 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-maas-api-openshift-ca\") pod \"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8\" (UID: \"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8\") " Oct 30 20:03:38.206132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.206113 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-kube-api-access-klwdr\" (UniqueName: \"kubernetes.io/projected/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-kube-api-access-klwdr\") pod \"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8\" (UID: \"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8\") " Oct 30 20:03:38.206962 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.206933 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-podinfo\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.219169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.209104 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-workload-certs\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.219169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.209238 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1888b3fa-2409-4051-99ff-22d37072bd4f-istiod-ca-cert\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.219169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.209407 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-token\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.219169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.209488 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-envoy\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.219169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.209552 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-workload-socket\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.219169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.209579 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-credential-socket\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.219169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.209619 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-data\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.219169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.209750 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcxdm\" (UniqueName: \"kubernetes.io/projected/1888b3fa-2409-4051-99ff-22d37072bd4f-kube-api-access-fcxdm\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.219169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.206621 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-maas-api-openshift-ca" pod "8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8" (UID: "8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:38.229174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.221386 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-kube-api-access-klwdr" pod "8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8" (UID: "8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8"). InnerVolumeSpecName "kube-api-access-klwdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:38.229174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.221541 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-maas-api-xks-ca" pod "8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8" (UID: "8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:38.264171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.244545 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn"] Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.313435 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcxdm\" (UniqueName: \"kubernetes.io/projected/1888b3fa-2409-4051-99ff-22d37072bd4f-kube-api-access-fcxdm\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.313521 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-podinfo\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.313572 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-workload-certs\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.313653 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1888b3fa-2409-4051-99ff-22d37072bd4f-istiod-ca-cert\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.313791 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-token\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.313857 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-envoy\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.313916 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-workload-socket\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.313941 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-credential-socket\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.313974 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-data\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.314091 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.314107 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klwdr\" (UniqueName: \"kubernetes.io/projected/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-kube-api-access-klwdr\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.314124 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.314607 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-data\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.317738 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-envoy\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.318000 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-workload-socket\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.318269 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-credential-socket\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.319260 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-workload-certs\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.320044 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1888b3fa-2409-4051-99ff-22d37072bd4f-istiod-ca-cert\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.323069 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-podinfo\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.323317 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" event={"ID":"8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8","Type":"ContainerDied","Data":"96ef6d7c4a4d05e3dfa33b59ab9321e8d5c28631f96668fb4d3646a78371573c"} Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.323372 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="96ef6d7c4a4d05e3dfa33b59ab9321e8d5c28631f96668fb4d3646a78371573c" Oct 30 20:03:38.327169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.323439 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s" Oct 30 20:03:38.337013 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.336986 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-token\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.337218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.337184 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcxdm\" (UniqueName: \"kubernetes.io/projected/1888b3fa-2409-4051-99ff-22d37072bd4f-kube-api-access-fcxdm\") pod \"e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:38.416441 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.416325 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:38.566168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:38.565505 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:39.838348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:39.838307 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:39.862296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:39.845236 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:03:39.862296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:39.845318 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:03:39.862296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:39.845357 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:03:39.898181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:39.893078 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn"] Oct 30 20:03:40.171017 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:40.160552 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:40.706445 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:40.706328 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:40.767740 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:40.767700 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-hc22z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" start-of-body= Oct 30 20:03:40.767936 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:40.767757 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" podUID="892edc88-bb54-46e1-acd1-701313b9ae0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.51:15021/healthz/ready\": dial tcp 10.132.0.51:15021: connect: connection refused" Oct 30 20:03:40.871951 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:40.871913 5091 generic.go:356] "Generic (PLEG): container finished" podID="892edc88-bb54-46e1-acd1-701313b9ae0a" containerID="2a197d207774acfbab864b95083dd2da8774ad0887239e372d2438fdbdb68b77" exitCode=0 Oct 30 20:03:40.872461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:40.872098 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" event={"ID":"892edc88-bb54-46e1-acd1-701313b9ae0a","Type":"ContainerDied","Data":"2a197d207774acfbab864b95083dd2da8774ad0887239e372d2438fdbdb68b77"} Oct 30 20:03:40.875013 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:40.874932 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" event={"ID":"1888b3fa-2409-4051-99ff-22d37072bd4f","Type":"ContainerStarted","Data":"6a89dcf2a18e9415269f153f1b7a5ed5f61c7ce6c2408cbbd43281481255cfd6"} Oct 30 20:03:40.875013 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:40.874990 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" event={"ID":"1888b3fa-2409-4051-99ff-22d37072bd4f","Type":"ContainerStarted","Data":"72bf252784400930fba67854bd575eabbef340e7eb69007f496dbf8c02f860d7"} Oct 30 20:03:40.926042 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:40.925978 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" podStartSLOduration=3.9259590490000003 podStartE2EDuration="3.925959049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:40.922736435 +0000 UTC m=+1940.617944420" watchObservedRunningTime="2026-10-30 20:03:40.925959049 +0000 UTC m=+1940.621167008" Oct 30 20:03:41.146377 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.145954 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 20:03:41.271236 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.271181 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-workload-certs\") pod \"892edc88-bb54-46e1-acd1-701313b9ae0a\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " Oct 30 20:03:41.271435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.271288 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-workload-socket\") pod \"892edc88-bb54-46e1-acd1-701313b9ae0a\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " Oct 30 20:03:41.271435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.271352 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892edc88-bb54-46e1-acd1-701313b9ae0a-kube-api-access-5shq8\" (UniqueName: \"kubernetes.io/projected/892edc88-bb54-46e1-acd1-701313b9ae0a-kube-api-access-5shq8\") pod \"892edc88-bb54-46e1-acd1-701313b9ae0a\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " Oct 30 20:03:41.271435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.271413 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-token\" (UniqueName: \"kubernetes.io/projected/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-token\") pod \"892edc88-bb54-46e1-acd1-701313b9ae0a\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " Oct 30 20:03:41.271639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.271443 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-credential-socket\") pod \"892edc88-bb54-46e1-acd1-701313b9ae0a\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " Oct 30 20:03:41.271639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.271542 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-data\") pod \"892edc88-bb54-46e1-acd1-701313b9ae0a\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " Oct 30 20:03:41.271639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.271608 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-envoy\") pod \"892edc88-bb54-46e1-acd1-701313b9ae0a\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " Oct 30 20:03:41.271816 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.271686 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/892edc88-bb54-46e1-acd1-701313b9ae0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892edc88-bb54-46e1-acd1-701313b9ae0a-istiod-ca-cert\") pod \"892edc88-bb54-46e1-acd1-701313b9ae0a\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " Oct 30 20:03:41.271816 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.271729 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-podinfo\") pod \"892edc88-bb54-46e1-acd1-701313b9ae0a\" (UID: \"892edc88-bb54-46e1-acd1-701313b9ae0a\") " Oct 30 20:03:41.273439 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.273409 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-credential-socket" pod "892edc88-bb54-46e1-acd1-701313b9ae0a" (UID: "892edc88-bb54-46e1-acd1-701313b9ae0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:41.273640 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.273601 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-workload-certs" pod "892edc88-bb54-46e1-acd1-701313b9ae0a" (UID: "892edc88-bb54-46e1-acd1-701313b9ae0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:41.273875 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.273844 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-data" pod "892edc88-bb54-46e1-acd1-701313b9ae0a" (UID: "892edc88-bb54-46e1-acd1-701313b9ae0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:41.274103 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.274080 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-workload-socket" pod "892edc88-bb54-46e1-acd1-701313b9ae0a" (UID: "892edc88-bb54-46e1-acd1-701313b9ae0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:41.274265 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.274244 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/892edc88-bb54-46e1-acd1-701313b9ae0a-istiod-ca-cert" pod "892edc88-bb54-46e1-acd1-701313b9ae0a" (UID: "892edc88-bb54-46e1-acd1-701313b9ae0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:41.275502 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.275467 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-podinfo" pod "892edc88-bb54-46e1-acd1-701313b9ae0a" (UID: "892edc88-bb54-46e1-acd1-701313b9ae0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:41.277099 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.277065 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-envoy" pod "892edc88-bb54-46e1-acd1-701313b9ae0a" (UID: "892edc88-bb54-46e1-acd1-701313b9ae0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:41.277259 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.277225 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-token" pod "892edc88-bb54-46e1-acd1-701313b9ae0a" (UID: "892edc88-bb54-46e1-acd1-701313b9ae0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:41.277259 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.277252 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/892edc88-bb54-46e1-acd1-701313b9ae0a-kube-api-access-5shq8" pod "892edc88-bb54-46e1-acd1-701313b9ae0a" (UID: "892edc88-bb54-46e1-acd1-701313b9ae0a"). InnerVolumeSpecName "kube-api-access-5shq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:41.374898 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.374860 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:41.374898 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.374898 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:41.375188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.374916 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:41.375188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.374930 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:41.375188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.374943 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/892edc88-bb54-46e1-acd1-701313b9ae0a-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:41.375188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.374955 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/892edc88-bb54-46e1-acd1-701313b9ae0a-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:41.375188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.374968 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:41.375188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.374981 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/892edc88-bb54-46e1-acd1-701313b9ae0a-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:41.375188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.374993 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5shq8\" (UniqueName: \"kubernetes.io/projected/892edc88-bb54-46e1-acd1-701313b9ae0a-kube-api-access-5shq8\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:41.566561 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.566465 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:41.575262 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.574771 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:41.757867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.757270 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:41.912260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.909639 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" Oct 30 20:03:41.916841 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.914511 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z" event={"ID":"892edc88-bb54-46e1-acd1-701313b9ae0a","Type":"ContainerDied","Data":"417efd96100f34b231adb1433aabb138108472373065ea6ede7a091dd1ca9d7e"} Oct 30 20:03:41.916841 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.914571 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:41.916841 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.914595 5091 scope.go:122] "RemoveContainer" containerID="2a197d207774acfbab864b95083dd2da8774ad0887239e372d2438fdbdb68b77" Oct 30 20:03:41.916841 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:41.915560 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:03:42.025201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:42.023469 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z"] Oct 30 20:03:42.032294 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:42.032257 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-hc22z"] Oct 30 20:03:42.445551 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:42.445505 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="892edc88-bb54-46e1-acd1-701313b9ae0a" path="/var/lib/kubelet/pods/892edc88-bb54-46e1-acd1-701313b9ae0a/volumes" Oct 30 20:03:42.688123 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:42.688081 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:42.933168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:42.929876 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:43.364167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:43.363807 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:43.639025 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:43.638931 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:03:43.639025 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:43.639008 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:03:43.654209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:43.651529 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:44.246669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.237257 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:44.272173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.267950 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm"] Oct 30 20:03:44.272173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.268251 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" containerID="cri-o://68832aa66e8f93e216fc70eaba91df2186e8add251452beab56420b434911714" gracePeriod=30 Oct 30 20:03:44.283419 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.277677 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb"] Oct 30 20:03:44.283419 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.277905 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" containerID="cri-o://ab47c816ecf73eaca11a1d6a055b56ebc135ebbbb19b71ea81407d71998ff790" gracePeriod=30 Oct 30 20:03:44.295178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.288193 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz"] Oct 30 20:03:44.295178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.290506 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" podUID="a95d6ded-89c1-4aee-b4e2-da19030afca8" containerName="istio-proxy" containerID="cri-o://22cd906037041447865cbba1675dbbf1cc763480e22e5f811a55961209058013" gracePeriod=30 Oct 30 20:03:44.865598 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.864353 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t"] Oct 30 20:03:44.865828 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.865712 5091 cpu_manager.go:414] "containerMap: removing container" podUID="892edc88-bb54-46e1-acd1-701313b9ae0a" containerName="istio-proxy" Oct 30 20:03:44.865828 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.865733 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="892edc88-bb54-46e1-acd1-701313b9ae0a" containerName="istio-proxy" Oct 30 20:03:44.865828 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.865807 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8" containerName="revoke-keys" Oct 30 20:03:44.865828 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.865816 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8" containerName="revoke-keys" Oct 30 20:03:44.866594 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.866027 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="892edc88-bb54-46e1-acd1-701313b9ae0a" containerName="istio-proxy" Oct 30 20:03:44.866594 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.866059 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8" containerName="revoke-keys" Oct 30 20:03:44.896184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.894912 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t"] Oct 30 20:03:44.896184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.895031 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" Oct 30 20:03:44.906396 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.904576 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-2a2286-dockercfg-c9t7r\"" Oct 30 20:03:44.906396 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.905720 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-2a2286\"" Oct 30 20:03:44.915528 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.915497 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:44.960288 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.959753 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t\" (UID: \"8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" Oct 30 20:03:44.960288 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:44.959891 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7p4t\" (UniqueName: \"kubernetes.io/projected/8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8-kube-api-access-v7p4t\") pod \"payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t\" (UID: \"8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" Oct 30 20:03:45.097169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:45.094830 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t\" (UID: \"8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" Oct 30 20:03:45.097169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:45.094966 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7p4t\" (UniqueName: \"kubernetes.io/projected/8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8-kube-api-access-v7p4t\") pod \"payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t\" (UID: \"8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" Oct 30 20:03:45.097169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:45.096339 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t\" (UID: \"8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" Oct 30 20:03:45.168218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:45.156025 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:45.313949 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:45.313858 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7p4t\" (UniqueName: \"kubernetes.io/projected/8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8-kube-api-access-v7p4t\") pod \"payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t\" (UID: \"8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" Oct 30 20:03:45.529566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:45.529536 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" Oct 30 20:03:45.711628 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:45.711463 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:45.903629 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:03:45.903600 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c1d2799_2a52_4d5e_9ddc_a4e84b4e26e8.slice/crio-5df2cd3b813371a3d8a401235adc4a60ba31708341187f8627630b0633aa706d WatchSource:0}: Error finding container 5df2cd3b813371a3d8a401235adc4a60ba31708341187f8627630b0633aa706d: Status 404 returned error can't find the container with id 5df2cd3b813371a3d8a401235adc4a60ba31708341187f8627630b0633aa706d Oct 30 20:03:45.906944 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:45.906895 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t"] Oct 30 20:03:45.979074 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:03:45.979038 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:03:46.166450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.166319 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" event={"ID":"8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8","Type":"ContainerStarted","Data":"d939db0cf44c4331b56e4046a78ab5b8905ccf9cdc9ed0403b3b71339fdf8045"} Oct 30 20:03:46.166450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.166384 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" event={"ID":"8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8","Type":"ContainerStarted","Data":"5df2cd3b813371a3d8a401235adc4a60ba31708341187f8627630b0633aa706d"} Oct 30 20:03:46.166450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.166417 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" Oct 30 20:03:46.169014 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.168990 5091 generic.go:356] "Generic (PLEG): container finished" podID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerID="e3f42e072853f68097efe15c2d0fafb0d3431177a806b5fa2d3740ef1e5201ea" exitCode=0 Oct 30 20:03:46.169186 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.169060 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" event={"ID":"f717fa94-89fa-438f-8a36-8d1a985d7454","Type":"ContainerDied","Data":"e3f42e072853f68097efe15c2d0fafb0d3431177a806b5fa2d3740ef1e5201ea"} Oct 30 20:03:46.169521 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.169505 5091 scope.go:122] "RemoveContainer" containerID="e3f42e072853f68097efe15c2d0fafb0d3431177a806b5fa2d3740ef1e5201ea" Oct 30 20:03:46.171347 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.171327 5091 generic.go:356] "Generic (PLEG): container finished" podID="071ef2ea-f7f5-4e12-9287-73c4f832f361" containerID="f2f279b992166019e79e7a0bfa6990bdbd4044a1aa3acfcf808ae8cbb196ec17" exitCode=0 Oct 30 20:03:46.171461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.171412 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" event={"ID":"071ef2ea-f7f5-4e12-9287-73c4f832f361","Type":"ContainerDied","Data":"f2f279b992166019e79e7a0bfa6990bdbd4044a1aa3acfcf808ae8cbb196ec17"} Oct 30 20:03:46.171737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.171712 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:03:46.171829 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.171795 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:03:46.171900 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.171837 5091 scope.go:122] "RemoveContainer" containerID="f2f279b992166019e79e7a0bfa6990bdbd4044a1aa3acfcf808ae8cbb196ec17" Oct 30 20:03:46.171900 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.171845 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:03:46.173662 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.173636 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:03:46.173777 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.173693 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:03:46.173777 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.173726 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:03:46.200655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.200590 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" podStartSLOduration=2.200577677 podStartE2EDuration="2.200577677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:03:46.19456179 +0000 UTC m=+1945.889769749" watchObservedRunningTime="2026-10-30 20:03:46.200577677 +0000 UTC m=+1945.895785634" Oct 30 20:03:46.844137 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:46.844090 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:47.212352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:47.212239 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" event={"ID":"f717fa94-89fa-438f-8a36-8d1a985d7454","Type":"ContainerStarted","Data":"e9fbd8ce7aaa5f7b1f3ce5a3ad477d68b5c35005d69c74a8e978c5198cd79ed2"} Oct 30 20:03:47.214612 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:47.214582 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" event={"ID":"071ef2ea-f7f5-4e12-9287-73c4f832f361","Type":"ContainerStarted","Data":"20f0990421ab107f3462fb007c9b5566560bf8a94baf2f6b82c2fc88e9d97838"} Oct 30 20:03:47.385029 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:47.384994 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:03:47.400322 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:47.400285 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 20:03:47.402826 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:47.402787 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" podUID="071ef2ea-f7f5-4e12-9287-73c4f832f361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 20:03:47.507968 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:47.507938 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:03:47.508373 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:47.508349 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 20:03:47.508823 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:47.508799 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 20:03:47.532412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:47.532380 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" podUID="962d02d2-4b11-4cee-b02e-6e0920ebaa7b" containerName="main" containerID="cri-o://038c6df61e44589e387a48da8daaf7ea834340170aebd512a75482ca361631c7" gracePeriod=45 Oct 30 20:03:47.701169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:47.699166 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" podUID="962d02d2-4b11-4cee-b02e-6e0920ebaa7b" containerName="main" probeResult="failure" output="Get \"https://10.132.1.19:8000/ready\": dial tcp 10.132.1.19:8000: connect: connection refused" Oct 30 20:03:48.037488 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.037456 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:48.240271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.236808 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-tmp-dir\") pod \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " Oct 30 20:03:48.245211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.244386 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-dshm\") pod \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " Oct 30 20:03:48.245211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.244457 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-model-cache\") pod \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " Oct 30 20:03:48.245211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.244507 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-kube-api-access-rhltm\" (UniqueName: \"kubernetes.io/projected/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-kube-api-access-rhltm\") pod \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " Oct 30 20:03:48.245211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.244569 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-home\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-home\") pod \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " Oct 30 20:03:48.245211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.244629 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-tls-certs\" (UniqueName: \"kubernetes.io/secret/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-tls-certs\") pod \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\" (UID: \"962d02d2-4b11-4cee-b02e-6e0920ebaa7b\") " Oct 30 20:03:48.296275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.261487 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-tmp-dir" pod "962d02d2-4b11-4cee-b02e-6e0920ebaa7b" (UID: "962d02d2-4b11-4cee-b02e-6e0920ebaa7b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:48.296275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.261842 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:48.296275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.274789 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-model-cache" pod "962d02d2-4b11-4cee-b02e-6e0920ebaa7b" (UID: "962d02d2-4b11-4cee-b02e-6e0920ebaa7b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:48.296275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.274870 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-dshm" pod "962d02d2-4b11-4cee-b02e-6e0920ebaa7b" (UID: "962d02d2-4b11-4cee-b02e-6e0920ebaa7b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:48.296275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.275086 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-home" pod "962d02d2-4b11-4cee-b02e-6e0920ebaa7b" (UID: "962d02d2-4b11-4cee-b02e-6e0920ebaa7b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:48.296275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.276222 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:03:48.307733 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.306934 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-tls-certs" pod "962d02d2-4b11-4cee-b02e-6e0920ebaa7b" (UID: "962d02d2-4b11-4cee-b02e-6e0920ebaa7b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:03:48.317650 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.317615 5091 generic.go:356] "Generic (PLEG): container finished" podID="962d02d2-4b11-4cee-b02e-6e0920ebaa7b" containerID="038c6df61e44589e387a48da8daaf7ea834340170aebd512a75482ca361631c7" exitCode=0 Oct 30 20:03:48.318359 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.318332 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" Oct 30 20:03:48.318779 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.318734 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" event={"ID":"962d02d2-4b11-4cee-b02e-6e0920ebaa7b","Type":"ContainerDied","Data":"038c6df61e44589e387a48da8daaf7ea834340170aebd512a75482ca361631c7"} Oct 30 20:03:48.318852 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.318779 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg" event={"ID":"962d02d2-4b11-4cee-b02e-6e0920ebaa7b","Type":"ContainerDied","Data":"b491c97bb30957a5b1a0c63503caad94bcae1e244a23b5f04c869945be239fb3"} Oct 30 20:03:48.318852 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.318803 5091 scope.go:122] "RemoveContainer" containerID="038c6df61e44589e387a48da8daaf7ea834340170aebd512a75482ca361631c7" Oct 30 20:03:48.319902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.319865 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-kube-api-access-rhltm" pod "962d02d2-4b11-4cee-b02e-6e0920ebaa7b" (UID: "962d02d2-4b11-4cee-b02e-6e0920ebaa7b"). InnerVolumeSpecName "kube-api-access-rhltm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:48.338027 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.338003 5091 scope.go:122] "RemoveContainer" containerID="038c6df61e44589e387a48da8daaf7ea834340170aebd512a75482ca361631c7" Oct 30 20:03:48.338415 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:03:48.338389 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"038c6df61e44589e387a48da8daaf7ea834340170aebd512a75482ca361631c7\": container with ID starting with 038c6df61e44589e387a48da8daaf7ea834340170aebd512a75482ca361631c7 not found: ID does not exist" containerID="038c6df61e44589e387a48da8daaf7ea834340170aebd512a75482ca361631c7" Oct 30 20:03:48.338503 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.338423 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038c6df61e44589e387a48da8daaf7ea834340170aebd512a75482ca361631c7"} err="failed to get container status \"038c6df61e44589e387a48da8daaf7ea834340170aebd512a75482ca361631c7\": rpc error: code = NotFound desc = could not find container \"038c6df61e44589e387a48da8daaf7ea834340170aebd512a75482ca361631c7\": container with ID starting with 038c6df61e44589e387a48da8daaf7ea834340170aebd512a75482ca361631c7 not found: ID does not exist" Oct 30 20:03:48.400182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.391887 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-tmp-dir\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:48.400182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.391931 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-dshm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:48.400182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.391947 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-model-cache\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:48.400182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.391969 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhltm\" (UniqueName: \"kubernetes.io/projected/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-kube-api-access-rhltm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:48.400182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.391991 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-home\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:48.400182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.392008 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/962d02d2-4b11-4cee-b02e-6e0920ebaa7b-tls-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:48.400182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.394247 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 20:03:48.400182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.394299 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" podUID="071ef2ea-f7f5-4e12-9287-73c4f832f361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 20:03:48.502605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.502565 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 20:03:48.502803 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.502626 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 20:03:48.691519 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.691484 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg"] Oct 30 20:03:48.703793 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:48.703755 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-836d67/autores-model-836d67-kserve-744fb9cf49-4w7dg"] Oct 30 20:03:49.224517 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.224470 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:49.369553 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.369512 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 20:03:49.370055 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.370021 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" podUID="071ef2ea-f7f5-4e12-9287-73c4f832f361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 20:03:49.509050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.508599 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 20:03:49.509050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.508670 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 20:03:49.810217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.809918 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 20:03:49.841258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.840736 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-data\") pod \"a95d6ded-89c1-4aee-b4e2-da19030afca8\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " Oct 30 20:03:49.841258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.840788 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95d6ded-89c1-4aee-b4e2-da19030afca8-kube-api-access-d77vb\" (UniqueName: \"kubernetes.io/projected/a95d6ded-89c1-4aee-b4e2-da19030afca8-kube-api-access-d77vb\") pod \"a95d6ded-89c1-4aee-b4e2-da19030afca8\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " Oct 30 20:03:49.841258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.840940 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-credential-socket\") pod \"a95d6ded-89c1-4aee-b4e2-da19030afca8\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " Oct 30 20:03:49.841258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.841046 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-workload-certs\") pod \"a95d6ded-89c1-4aee-b4e2-da19030afca8\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " Oct 30 20:03:49.841258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.841082 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a95d6ded-89c1-4aee-b4e2-da19030afca8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95d6ded-89c1-4aee-b4e2-da19030afca8-istiod-ca-cert\") pod \"a95d6ded-89c1-4aee-b4e2-da19030afca8\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " Oct 30 20:03:49.842936 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.841633 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-envoy\") pod \"a95d6ded-89c1-4aee-b4e2-da19030afca8\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " Oct 30 20:03:49.842936 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.841728 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-token\" (UniqueName: \"kubernetes.io/projected/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-token\") pod \"a95d6ded-89c1-4aee-b4e2-da19030afca8\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " Oct 30 20:03:49.842936 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.841771 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-workload-socket\") pod \"a95d6ded-89c1-4aee-b4e2-da19030afca8\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " Oct 30 20:03:49.842936 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.841850 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-podinfo\") pod \"a95d6ded-89c1-4aee-b4e2-da19030afca8\" (UID: \"a95d6ded-89c1-4aee-b4e2-da19030afca8\") " Oct 30 20:03:49.844060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.843866 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-data" pod "a95d6ded-89c1-4aee-b4e2-da19030afca8" (UID: "a95d6ded-89c1-4aee-b4e2-da19030afca8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:49.888170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.881239 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-workload-socket" pod "a95d6ded-89c1-4aee-b4e2-da19030afca8" (UID: "a95d6ded-89c1-4aee-b4e2-da19030afca8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:49.888170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.883244 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-credential-socket" pod "a95d6ded-89c1-4aee-b4e2-da19030afca8" (UID: "a95d6ded-89c1-4aee-b4e2-da19030afca8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:49.890278 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.889605 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-workload-certs" pod "a95d6ded-89c1-4aee-b4e2-da19030afca8" (UID: "a95d6ded-89c1-4aee-b4e2-da19030afca8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:49.918273 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.898341 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95d6ded-89c1-4aee-b4e2-da19030afca8-istiod-ca-cert" pod "a95d6ded-89c1-4aee-b4e2-da19030afca8" (UID: "a95d6ded-89c1-4aee-b4e2-da19030afca8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:49.942324 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.942265 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95d6ded-89c1-4aee-b4e2-da19030afca8-kube-api-access-d77vb" pod "a95d6ded-89c1-4aee-b4e2-da19030afca8" (UID: "a95d6ded-89c1-4aee-b4e2-da19030afca8"). InnerVolumeSpecName "kube-api-access-d77vb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:49.945315 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.945284 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a95d6ded-89c1-4aee-b4e2-da19030afca8-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:49.945423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.945324 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:49.945423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.945344 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:49.945423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.945358 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d77vb\" (UniqueName: \"kubernetes.io/projected/a95d6ded-89c1-4aee-b4e2-da19030afca8-kube-api-access-d77vb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:49.945423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.945372 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:49.945423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.945385 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:49.959173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:49.958547 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-envoy" pod "a95d6ded-89c1-4aee-b4e2-da19030afca8" (UID: "a95d6ded-89c1-4aee-b4e2-da19030afca8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:50.018558 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.006304 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-token" pod "a95d6ded-89c1-4aee-b4e2-da19030afca8" (UID: "a95d6ded-89c1-4aee-b4e2-da19030afca8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:50.027128 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.027001 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-podinfo" pod "a95d6ded-89c1-4aee-b4e2-da19030afca8" (UID: "a95d6ded-89c1-4aee-b4e2-da19030afca8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:50.033605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.033511 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 20:03:50.076242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.057627 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.124299 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.122845 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.124299 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.122880 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a95d6ded-89c1-4aee-b4e2-da19030afca8-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.225165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.224931 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-credential-socket\") pod \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " Oct 30 20:03:50.225165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.225046 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-kube-api-access-fhnv5\" (UniqueName: \"kubernetes.io/projected/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-kube-api-access-fhnv5\") pod \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " Oct 30 20:03:50.225727 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.225187 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-token\" (UniqueName: \"kubernetes.io/projected/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-token\") pod \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " Oct 30 20:03:50.225727 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.225264 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-podinfo\") pod \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " Oct 30 20:03:50.225727 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.225315 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-data\") pod \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " Oct 30 20:03:50.225727 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.225356 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-workload-socket\") pod \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " Oct 30 20:03:50.225727 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.225500 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-workload-certs\") pod \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " Oct 30 20:03:50.225727 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.225690 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istiod-ca-cert\") pod \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " Oct 30 20:03:50.226097 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.225759 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-envoy\") pod \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\" (UID: \"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8\") " Oct 30 20:03:50.234197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.231653 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-workload-socket" pod "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" (UID: "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:50.234197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.231873 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-data" pod "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" (UID: "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:50.268688 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.268646 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istiod-ca-cert" pod "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" (UID: "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:50.271404 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.271363 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-credential-socket" pod "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" (UID: "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:50.331069 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.330861 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.331328 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.331305 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.331456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.331441 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.331572 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.331559 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.332342 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.332316 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-workload-certs" pod "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" (UID: "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:50.426099 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.426056 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-token" pod "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" (UID: "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:50.431271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.426600 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:50.434699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.434675 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.434895 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.434879 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.443077 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.443047 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 20:03:50.443257 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.443205 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" podUID="071ef2ea-f7f5-4e12-9287-73c4f832f361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 20:03:50.518167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.505615 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-envoy" pod "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" (UID: "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:50.523021 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.522989 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 20:03:50.523176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.523057 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 20:03:50.525322 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.524509 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-podinfo" pod "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" (UID: "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:50.525322 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.525071 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-kube-api-access-fhnv5" pod "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" (UID: "1d48d1bc-06e7-4deb-8d61-d8e86c303aa8"). InnerVolumeSpecName "kube-api-access-fhnv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:50.529224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.528060 5091 generic.go:356] "Generic (PLEG): container finished" podID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerID="ab47c816ecf73eaca11a1d6a055b56ebc135ebbbb19b71ea81407d71998ff790" exitCode=0 Oct 30 20:03:50.529224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.528217 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" Oct 30 20:03:50.530779 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.530754 5091 generic.go:356] "Generic (PLEG): container finished" podID="a95d6ded-89c1-4aee-b4e2-da19030afca8" containerID="22cd906037041447865cbba1675dbbf1cc763480e22e5f811a55961209058013" exitCode=0 Oct 30 20:03:50.532504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.530782 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="962d02d2-4b11-4cee-b02e-6e0920ebaa7b" path="/var/lib/kubelet/pods/962d02d2-4b11-4cee-b02e-6e0920ebaa7b/volumes" Oct 30 20:03:50.532504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.530841 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" Oct 30 20:03:50.532504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.531735 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" event={"ID":"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8","Type":"ContainerDied","Data":"ab47c816ecf73eaca11a1d6a055b56ebc135ebbbb19b71ea81407d71998ff790"} Oct 30 20:03:50.532504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.531771 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb" event={"ID":"1d48d1bc-06e7-4deb-8d61-d8e86c303aa8","Type":"ContainerDied","Data":"4888f04372e69b28a165c87c94f4f320346b8e9067f425196d39c4f5fcd119d2"} Oct 30 20:03:50.532504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.531786 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" event={"ID":"a95d6ded-89c1-4aee-b4e2-da19030afca8","Type":"ContainerDied","Data":"22cd906037041447865cbba1675dbbf1cc763480e22e5f811a55961209058013"} Oct 30 20:03:50.532504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.531802 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz" event={"ID":"a95d6ded-89c1-4aee-b4e2-da19030afca8","Type":"ContainerDied","Data":"c5d3aeb52b242df38980ce1368a5c91959b2c9b9ab3fd523fd9991174837dec6"} Oct 30 20:03:50.532504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.531823 5091 scope.go:122] "RemoveContainer" containerID="ab47c816ecf73eaca11a1d6a055b56ebc135ebbbb19b71ea81407d71998ff790" Oct 30 20:03:50.536685 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.536661 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.536890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.536858 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.537024 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.537009 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhnv5\" (UniqueName: \"kubernetes.io/projected/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8-kube-api-access-fhnv5\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.541016 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.540994 5091 generic.go:356] "Generic (PLEG): container finished" podID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerID="68832aa66e8f93e216fc70eaba91df2186e8add251452beab56420b434911714" exitCode=0 Oct 30 20:03:50.541379 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.541359 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" event={"ID":"ff97dc89-e1e0-457c-8ef2-8a4684f36e31","Type":"ContainerDied","Data":"68832aa66e8f93e216fc70eaba91df2186e8add251452beab56420b434911714"} Oct 30 20:03:50.559168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.550329 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 20:03:50.617420 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.617392 5091 scope.go:122] "RemoveContainer" containerID="ab47c816ecf73eaca11a1d6a055b56ebc135ebbbb19b71ea81407d71998ff790" Oct 30 20:03:50.617527 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.617512 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb"] Oct 30 20:03:50.617844 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:03:50.617821 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab47c816ecf73eaca11a1d6a055b56ebc135ebbbb19b71ea81407d71998ff790\": container with ID starting with ab47c816ecf73eaca11a1d6a055b56ebc135ebbbb19b71ea81407d71998ff790 not found: ID does not exist" containerID="ab47c816ecf73eaca11a1d6a055b56ebc135ebbbb19b71ea81407d71998ff790" Oct 30 20:03:50.617957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.617851 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab47c816ecf73eaca11a1d6a055b56ebc135ebbbb19b71ea81407d71998ff790"} err="failed to get container status \"ab47c816ecf73eaca11a1d6a055b56ebc135ebbbb19b71ea81407d71998ff790\": rpc error: code = NotFound desc = could not find container \"ab47c816ecf73eaca11a1d6a055b56ebc135ebbbb19b71ea81407d71998ff790\": container with ID starting with ab47c816ecf73eaca11a1d6a055b56ebc135ebbbb19b71ea81407d71998ff790 not found: ID does not exist" Oct 30 20:03:50.617957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.617875 5091 scope.go:122] "RemoveContainer" containerID="22cd906037041447865cbba1675dbbf1cc763480e22e5f811a55961209058013" Oct 30 20:03:50.621366 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.621342 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-rszxb"] Oct 30 20:03:50.630073 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.630053 5091 scope.go:122] "RemoveContainer" containerID="22cd906037041447865cbba1675dbbf1cc763480e22e5f811a55961209058013" Oct 30 20:03:50.630418 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:03:50.630396 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22cd906037041447865cbba1675dbbf1cc763480e22e5f811a55961209058013\": container with ID starting with 22cd906037041447865cbba1675dbbf1cc763480e22e5f811a55961209058013 not found: ID does not exist" containerID="22cd906037041447865cbba1675dbbf1cc763480e22e5f811a55961209058013" Oct 30 20:03:50.630473 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.630425 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22cd906037041447865cbba1675dbbf1cc763480e22e5f811a55961209058013"} err="failed to get container status \"22cd906037041447865cbba1675dbbf1cc763480e22e5f811a55961209058013\": rpc error: code = NotFound desc = could not find container \"22cd906037041447865cbba1675dbbf1cc763480e22e5f811a55961209058013\": container with ID starting with 22cd906037041447865cbba1675dbbf1cc763480e22e5f811a55961209058013 not found: ID does not exist" Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.637925 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-workload-certs\") pod \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.638023 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-kube-api-access-chsdr\" (UniqueName: \"kubernetes.io/projected/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-kube-api-access-chsdr\") pod \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.638159 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-data\") pod \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.638208 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-workload-socket\") pod \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.638362 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istiod-ca-cert\") pod \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.638425 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-workload-certs" pod "ff97dc89-e1e0-457c-8ef2-8a4684f36e31" (UID: "ff97dc89-e1e0-457c-8ef2-8a4684f36e31"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.638470 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-podinfo\") pod \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.638520 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-envoy\") pod \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.638573 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-credential-socket\") pod \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.638610 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-token\" (UniqueName: \"kubernetes.io/projected/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-token\") pod \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\" (UID: \"ff97dc89-e1e0-457c-8ef2-8a4684f36e31\") " Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.638622 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-data" pod "ff97dc89-e1e0-457c-8ef2-8a4684f36e31" (UID: "ff97dc89-e1e0-457c-8ef2-8a4684f36e31"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.639283 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-workload-socket" pod "ff97dc89-e1e0-457c-8ef2-8a4684f36e31" (UID: "ff97dc89-e1e0-457c-8ef2-8a4684f36e31"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.639392 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.639412 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.639427 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.639560 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istiod-ca-cert" pod "ff97dc89-e1e0-457c-8ef2-8a4684f36e31" (UID: "ff97dc89-e1e0-457c-8ef2-8a4684f36e31"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.640710 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-credential-socket" pod "ff97dc89-e1e0-457c-8ef2-8a4684f36e31" (UID: "ff97dc89-e1e0-457c-8ef2-8a4684f36e31"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:50.643050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.641775 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-kube-api-access-chsdr" pod "ff97dc89-e1e0-457c-8ef2-8a4684f36e31" (UID: "ff97dc89-e1e0-457c-8ef2-8a4684f36e31"). InnerVolumeSpecName "kube-api-access-chsdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:50.644377 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.644346 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-token" pod "ff97dc89-e1e0-457c-8ef2-8a4684f36e31" (UID: "ff97dc89-e1e0-457c-8ef2-8a4684f36e31"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:03:50.645282 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.645257 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-envoy" pod "ff97dc89-e1e0-457c-8ef2-8a4684f36e31" (UID: "ff97dc89-e1e0-457c-8ef2-8a4684f36e31"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:03:50.645574 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.645554 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-podinfo" pod "ff97dc89-e1e0-457c-8ef2-8a4684f36e31" (UID: "ff97dc89-e1e0-457c-8ef2-8a4684f36e31"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:03:50.713197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.710648 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz"] Oct 30 20:03:50.720542 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.720513 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-pl6kz"] Oct 30 20:03:50.740982 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.740951 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.740982 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.740982 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.741205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.740995 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.741205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.741005 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.741205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.741013 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:50.741205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:50.741022 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chsdr\" (UniqueName: \"kubernetes.io/projected/ff97dc89-e1e0-457c-8ef2-8a4684f36e31-kube-api-access-chsdr\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:03:51.339978 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:51.339897 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:51.400175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:51.390295 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 20:03:51.400175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:51.390359 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" podUID="071ef2ea-f7f5-4e12-9287-73c4f832f361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 20:03:51.503572 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:51.503533 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 20:03:51.503933 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:51.503902 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 20:03:51.594051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:51.593956 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" event={"ID":"ff97dc89-e1e0-457c-8ef2-8a4684f36e31","Type":"ContainerDied","Data":"1bf5863cee8c84c3f77d5853b64ade8ce45325264db383447f4e4ce2b6e4fb4e"} Oct 30 20:03:51.594322 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:51.594302 5091 scope.go:122] "RemoveContainer" containerID="68832aa66e8f93e216fc70eaba91df2186e8add251452beab56420b434911714" Oct 30 20:03:51.594538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:51.594523 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm" Oct 30 20:03:51.665172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:51.655205 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm"] Oct 30 20:03:51.695171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:51.694758 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-fbnrm"] Oct 30 20:03:52.299177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:52.298129 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:52.372046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:52.372007 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" start-of-body= Oct 30 20:03:52.372697 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:52.372663 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" podUID="071ef2ea-f7f5-4e12-9287-73c4f832f361" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.8:15021/healthz/ready\": dial tcp 10.132.1.8:15021: connect: connection refused" Oct 30 20:03:52.443604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:52.443568 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" path="/var/lib/kubelet/pods/1d48d1bc-06e7-4deb-8d61-d8e86c303aa8/volumes" Oct 30 20:03:52.444197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:52.444174 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a95d6ded-89c1-4aee-b4e2-da19030afca8" path="/var/lib/kubelet/pods/a95d6ded-89c1-4aee-b4e2-da19030afca8/volumes" Oct 30 20:03:52.444752 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:52.444729 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" path="/var/lib/kubelet/pods/ff97dc89-e1e0-457c-8ef2-8a4684f36e31/volumes" Oct 30 20:03:52.505171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:52.501875 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" start-of-body= Oct 30 20:03:52.505171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:52.501939 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.15:15021/healthz/ready\": dial tcp 10.132.1.15:15021: connect: connection refused" Oct 30 20:03:53.158184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:53.157326 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:53.375118 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:53.375082 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:03:53.376367 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:53.376338 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:03:53.506642 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:53.506610 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:03:53.507192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:53.507133 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:03:53.508193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:53.508137 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:03:54.011311 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:53.989338 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:54.936707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:54.936671 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:55.782172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:55.775912 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:56.632309 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:56.628839 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:57.217592 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:57.217564 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" Oct 30 20:03:57.635937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:57.635902 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:58.483171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:58.480109 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:03:59.453172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:03:59.452805 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:00.293709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:00.293661 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:01.190986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:01.190167 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:02.186967 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:02.184932 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:03.202271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:03.202232 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:03.917489 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:03.917453 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:04.141169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:04.137729 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:04.294165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:04.285195 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s"] Oct 30 20:04:04.313832 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:04.309513 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-836d67-q4r8s"] Oct 30 20:04:04.387438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:04.387400 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:04.487800 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:04.487767 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8" path="/var/lib/kubelet/pods/8c1b809f-82cd-4b9f-9e3a-ee9554ec96a8/volumes" Oct 30 20:04:04.563396 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:04.563313 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:04.948809 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:04.948712 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:05.694455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:05.694423 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:06.235240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:06.235199 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:06.681656 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:06.681613 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:06.880181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:06.877080 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:07.447800 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:07.447755 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:07.679764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:07.679375 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:07.861038 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:07.860996 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:08.030203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:08.030165 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:08.239713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:08.239625 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:08.395587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:08.395546 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:08.895934 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:08.895896 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:09.135596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.135562 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:09.538825 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.538787 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:09.639646 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.639613 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh"] Oct 30 20:04:09.641051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.641017 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a95d6ded-89c1-4aee-b4e2-da19030afca8" containerName="istio-proxy" Oct 30 20:04:09.641271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.641253 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a95d6ded-89c1-4aee-b4e2-da19030afca8" containerName="istio-proxy" Oct 30 20:04:09.641485 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.641471 5091 cpu_manager.go:414] "containerMap: removing container" podUID="962d02d2-4b11-4cee-b02e-6e0920ebaa7b" containerName="main" Oct 30 20:04:09.641594 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.641583 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="962d02d2-4b11-4cee-b02e-6e0920ebaa7b" containerName="main" Oct 30 20:04:09.641679 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.641668 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" Oct 30 20:04:09.641753 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.641744 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" Oct 30 20:04:09.641831 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.641821 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" Oct 30 20:04:09.641910 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.641901 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" Oct 30 20:04:09.649212 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.642161 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="962d02d2-4b11-4cee-b02e-6e0920ebaa7b" containerName="main" Oct 30 20:04:09.649403 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.649385 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff97dc89-e1e0-457c-8ef2-8a4684f36e31" containerName="istio-proxy" Oct 30 20:04:09.649516 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.649506 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d48d1bc-06e7-4deb-8d61-d8e86c303aa8" containerName="istio-proxy" Oct 30 20:04:09.649618 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.649609 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a95d6ded-89c1-4aee-b4e2-da19030afca8" containerName="istio-proxy" Oct 30 20:04:09.838217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.809532 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:09.903235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.900738 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh"] Oct 30 20:04:09.903235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.900861 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:09.947213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.946385 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4d5758\"" Oct 30 20:04:09.947213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.946580 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"kube-root-ca.crt\"" Oct 30 20:04:09.947213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.947177 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"openshift-service-ca.crt\"" Oct 30 20:04:09.949208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.948687 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"routing-overlay\"" Oct 30 20:04:09.949774 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.949608 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"agc-cred-14ae83b19b59\"" Oct 30 20:04:09.949774 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:09.949747 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"payload-processing-external-model-e2e--632181a7-dockercfg-lhmxz\"" Oct 30 20:04:10.103215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:10.101830 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:10.103215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:10.101921 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:10.103215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:10.102034 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjgps\" (UniqueName: \"kubernetes.io/projected/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-kube-api-access-tjgps\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:10.103215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:10.102084 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:10.203251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:10.203200 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjgps\" (UniqueName: \"kubernetes.io/projected/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-kube-api-access-tjgps\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:10.203251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:10.203253 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:10.203826 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:10.203420 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:10.203826 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:10.203463 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:10.204230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:10.204198 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:10.204320 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:10.204264 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:10.206005 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:10.205973 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:10.218205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:10.218138 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjgps\" (UniqueName: \"kubernetes.io/projected/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-kube-api-access-tjgps\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:10.412838 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:10.412755 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:10.535544 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:10.535414 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:10.844324 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:10.844295 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh"] Oct 30 20:04:10.844483 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:04:10.844420 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8b5abf_46bc_4f75_b8a5_0f3d69b55aea.slice/crio-e359cd96e52d8d81674f678b5296d8a85a379338866976f539418d48a0e2a838 WatchSource:0}: Error finding container e359cd96e52d8d81674f678b5296d8a85a379338866976f539418d48a0e2a838: Status 404 returned error can't find the container with id e359cd96e52d8d81674f678b5296d8a85a379338866976f539418d48a0e2a838 Oct 30 20:04:11.085535 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:11.085500 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" event={"ID":"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea","Type":"ContainerStarted","Data":"4a432706dd586ae865512848302fcd70174a787feb6ea253b261017cc24f9509"} Oct 30 20:04:11.085716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:11.085562 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" event={"ID":"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea","Type":"ContainerStarted","Data":"e359cd96e52d8d81674f678b5296d8a85a379338866976f539418d48a0e2a838"} Oct 30 20:04:11.085716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:11.085594 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:11.113520 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:11.113432 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" podStartSLOduration=2.113418653 podStartE2EDuration="2.113418653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:11.109261866 +0000 UTC m=+1970.804469825" watchObservedRunningTime="2026-10-30 20:04:11.113418653 +0000 UTC m=+1970.808626640" Oct 30 20:04:11.579411 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:11.579319 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:12.006439 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:12.005225 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:12.194133 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:12.194097 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:12.315205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:12.315102 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2"] Oct 30 20:04:12.315661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:12.315627 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" podUID="632cf328-468e-4865-a6ef-7358d679cb6d" containerName="istio-proxy" containerID="cri-o://53d6769622928252d297348fee8c82c8081b9621cdecfe411dd7a386330132ad" gracePeriod=30 Oct 30 20:04:12.489675 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:12.489635 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:12.682354 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:12.682268 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:12.917588 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:12.917555 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:13.525177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:13.522935 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:13.872296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:13.872188 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:14.332591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.332557 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:14.525347 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.525263 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:14.750447 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.750412 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq"] Oct 30 20:04:14.757656 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.756834 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:14.773344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.773308 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq"] Oct 30 20:04:14.774693 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.774568 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-517027-openshift-default-dockercfg-292ww\"" Oct 30 20:04:14.853094 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.853058 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-workload-socket\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:14.853248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.853135 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7zfr\" (UniqueName: \"kubernetes.io/projected/43e01282-cbf5-43e3-9815-32c7d4439800-kube-api-access-g7zfr\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:14.853308 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.853293 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-workload-certs\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:14.853352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.853316 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e01282-cbf5-43e3-9815-32c7d4439800-istiod-ca-cert\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:14.853498 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.853469 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e01282-cbf5-43e3-9815-32c7d4439800-istio-podinfo\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:14.853584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.853527 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e01282-cbf5-43e3-9815-32c7d4439800-istio-token\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:14.853681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.853654 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-istio-envoy\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:14.853729 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.853714 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-credential-socket\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:14.853791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.853773 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-istio-data\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:14.904672 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.902083 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:14.960671 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.960634 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-workload-socket\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:14.964704 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.964581 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-workload-socket\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:14.968304 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.968270 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7zfr\" (UniqueName: \"kubernetes.io/projected/43e01282-cbf5-43e3-9815-32c7d4439800-kube-api-access-g7zfr\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:14.985540 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:14.985511 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-workload-certs\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.008851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.008819 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7zfr\" (UniqueName: \"kubernetes.io/projected/43e01282-cbf5-43e3-9815-32c7d4439800-kube-api-access-g7zfr\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.041225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.009075 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-workload-certs\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.042955 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.042920 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e01282-cbf5-43e3-9815-32c7d4439800-istiod-ca-cert\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.043281 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.043260 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e01282-cbf5-43e3-9815-32c7d4439800-istio-podinfo\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.043495 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.043475 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e01282-cbf5-43e3-9815-32c7d4439800-istio-token\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.043670 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.043656 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-istio-envoy\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.047274 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.047242 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-credential-socket\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.047478 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.047459 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-istio-data\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.058449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.048010 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e01282-cbf5-43e3-9815-32c7d4439800-istiod-ca-cert\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.058621 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.048727 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-credential-socket\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.058707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.051480 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-istio-data\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.123168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.114826 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e01282-cbf5-43e3-9815-32c7d4439800-istio-podinfo\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.127699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.127658 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e01282-cbf5-43e3-9815-32c7d4439800-istio-token\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.145250 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.142438 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-istio-envoy\") pod \"e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.224185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.222778 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:15.387040 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.386961 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:15.703219 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.702026 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:15.768545 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:04:15.768500 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e01282_cbf5_43e3_9815_32c7d4439800.slice/crio-cf8a8481bda03aec270ddfafede3d046ed2a242f8aa77f7d76a5d10d1888d254 WatchSource:0}: Error finding container cf8a8481bda03aec270ddfafede3d046ed2a242f8aa77f7d76a5d10d1888d254: Status 404 returned error can't find the container with id cf8a8481bda03aec270ddfafede3d046ed2a242f8aa77f7d76a5d10d1888d254 Oct 30 20:04:15.769111 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.769073 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq"] Oct 30 20:04:15.772569 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.772522 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:04:15.772677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.772618 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:04:15.772677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:15.772664 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:04:15.985994 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:04:15.985893 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:04:16.208940 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:16.208902 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" event={"ID":"43e01282-cbf5-43e3-9815-32c7d4439800","Type":"ContainerStarted","Data":"51dd32bbba52c2223ad6dbbffa3b462b1068e4d77a4c2b50b458cd5b7a6ed92a"} Oct 30 20:04:16.209126 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:16.208966 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" event={"ID":"43e01282-cbf5-43e3-9815-32c7d4439800","Type":"ContainerStarted","Data":"cf8a8481bda03aec270ddfafede3d046ed2a242f8aa77f7d76a5d10d1888d254"} Oct 30 20:04:16.241221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:16.241070 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" podStartSLOduration=2.241051719 podStartE2EDuration="2.241051719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:16.23863346 +0000 UTC m=+1975.933841417" watchObservedRunningTime="2026-10-30 20:04:16.241051719 +0000 UTC m=+1975.936259680" Oct 30 20:04:16.398197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:16.391418 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:16.451320 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:16.451292 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:16.530787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:16.530745 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:17.222497 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.222463 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:17.223533 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.223504 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:04:17.460716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.460676 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:17.668863 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.668833 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:04:17.733843 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.733754 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-credential-socket\") pod \"632cf328-468e-4865-a6ef-7358d679cb6d\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " Oct 30 20:04:17.734479 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.734358 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-credential-socket" pod "632cf328-468e-4865-a6ef-7358d679cb6d" (UID: "632cf328-468e-4865-a6ef-7358d679cb6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:17.734597 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.734501 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-istio-data\") pod \"632cf328-468e-4865-a6ef-7358d679cb6d\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " Oct 30 20:04:17.734597 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.734539 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/632cf328-468e-4865-a6ef-7358d679cb6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632cf328-468e-4865-a6ef-7358d679cb6d-istiod-ca-cert\") pod \"632cf328-468e-4865-a6ef-7358d679cb6d\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " Oct 30 20:04:17.734713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.734611 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/632cf328-468e-4865-a6ef-7358d679cb6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632cf328-468e-4865-a6ef-7358d679cb6d-istio-podinfo\") pod \"632cf328-468e-4865-a6ef-7358d679cb6d\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " Oct 30 20:04:17.734713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.734704 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/632cf328-468e-4865-a6ef-7358d679cb6d-istio-token\" (UniqueName: \"kubernetes.io/projected/632cf328-468e-4865-a6ef-7358d679cb6d-istio-token\") pod \"632cf328-468e-4865-a6ef-7358d679cb6d\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " Oct 30 20:04:17.734847 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.734821 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-istio-envoy\") pod \"632cf328-468e-4865-a6ef-7358d679cb6d\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " Oct 30 20:04:17.734916 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.734899 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-workload-certs\") pod \"632cf328-468e-4865-a6ef-7358d679cb6d\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " Oct 30 20:04:17.734975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.734952 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/632cf328-468e-4865-a6ef-7358d679cb6d-kube-api-access-57m6l\" (UniqueName: \"kubernetes.io/projected/632cf328-468e-4865-a6ef-7358d679cb6d-kube-api-access-57m6l\") pod \"632cf328-468e-4865-a6ef-7358d679cb6d\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " Oct 30 20:04:17.735042 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.735015 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/632cf328-468e-4865-a6ef-7358d679cb6d-istiod-ca-cert" pod "632cf328-468e-4865-a6ef-7358d679cb6d" (UID: "632cf328-468e-4865-a6ef-7358d679cb6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:17.735133 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.735112 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-workload-socket\") pod \"632cf328-468e-4865-a6ef-7358d679cb6d\" (UID: \"632cf328-468e-4865-a6ef-7358d679cb6d\") " Oct 30 20:04:17.736405 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.735873 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-workload-certs" pod "632cf328-468e-4865-a6ef-7358d679cb6d" (UID: "632cf328-468e-4865-a6ef-7358d679cb6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:17.736405 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.736185 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-workload-socket" pod "632cf328-468e-4865-a6ef-7358d679cb6d" (UID: "632cf328-468e-4865-a6ef-7358d679cb6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:17.736405 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.736342 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:17.736405 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.736366 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/632cf328-468e-4865-a6ef-7358d679cb6d-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:17.736405 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.736381 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:17.736836 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.736814 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-istio-data" pod "632cf328-468e-4865-a6ef-7358d679cb6d" (UID: "632cf328-468e-4865-a6ef-7358d679cb6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:17.738882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.738860 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-istio-envoy" pod "632cf328-468e-4865-a6ef-7358d679cb6d" (UID: "632cf328-468e-4865-a6ef-7358d679cb6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:04:17.739668 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.739643 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632cf328-468e-4865-a6ef-7358d679cb6d-istio-token" pod "632cf328-468e-4865-a6ef-7358d679cb6d" (UID: "632cf328-468e-4865-a6ef-7358d679cb6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:17.741566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.741529 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/632cf328-468e-4865-a6ef-7358d679cb6d-kube-api-access-57m6l" pod "632cf328-468e-4865-a6ef-7358d679cb6d" (UID: "632cf328-468e-4865-a6ef-7358d679cb6d"). InnerVolumeSpecName "kube-api-access-57m6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:17.742030 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.741998 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/632cf328-468e-4865-a6ef-7358d679cb6d-istio-podinfo" pod "632cf328-468e-4865-a6ef-7358d679cb6d" (UID: "632cf328-468e-4865-a6ef-7358d679cb6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:04:17.837916 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.837875 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/632cf328-468e-4865-a6ef-7358d679cb6d-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:17.837916 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.837918 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:17.838124 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.837936 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57m6l\" (UniqueName: \"kubernetes.io/projected/632cf328-468e-4865-a6ef-7358d679cb6d-kube-api-access-57m6l\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:17.838124 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.837954 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:17.838124 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.837968 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/632cf328-468e-4865-a6ef-7358d679cb6d-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:17.838124 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:17.837982 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/632cf328-468e-4865-a6ef-7358d679cb6d-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:18.244729 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.244686 5091 generic.go:356] "Generic (PLEG): container finished" podID="632cf328-468e-4865-a6ef-7358d679cb6d" containerID="53d6769622928252d297348fee8c82c8081b9621cdecfe411dd7a386330132ad" exitCode=0 Oct 30 20:04:18.245509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.245178 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" Oct 30 20:04:18.245769 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.245200 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" event={"ID":"632cf328-468e-4865-a6ef-7358d679cb6d","Type":"ContainerDied","Data":"53d6769622928252d297348fee8c82c8081b9621cdecfe411dd7a386330132ad"} Oct 30 20:04:18.245992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.245965 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2" event={"ID":"632cf328-468e-4865-a6ef-7358d679cb6d","Type":"ContainerDied","Data":"3ddf74a906a61965e00269df83bfdb2ebe7c70a787ea2d349b8703928adf9f51"} Oct 30 20:04:18.246113 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.246070 5091 scope.go:122] "RemoveContainer" containerID="53d6769622928252d297348fee8c82c8081b9621cdecfe411dd7a386330132ad" Oct 30 20:04:18.262529 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.262261 5091 scope.go:122] "RemoveContainer" containerID="05cc5cc5c831bc8c6a3ee2bf75528e70eb90d375b85397a00f6aec2e88cfd762" Oct 30 20:04:18.278492 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.278461 5091 scope.go:122] "RemoveContainer" containerID="53d6769622928252d297348fee8c82c8081b9621cdecfe411dd7a386330132ad" Oct 30 20:04:18.278924 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:04:18.278893 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53d6769622928252d297348fee8c82c8081b9621cdecfe411dd7a386330132ad\": container with ID starting with 53d6769622928252d297348fee8c82c8081b9621cdecfe411dd7a386330132ad not found: ID does not exist" containerID="53d6769622928252d297348fee8c82c8081b9621cdecfe411dd7a386330132ad" Oct 30 20:04:18.279033 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.278936 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53d6769622928252d297348fee8c82c8081b9621cdecfe411dd7a386330132ad"} err="failed to get container status \"53d6769622928252d297348fee8c82c8081b9621cdecfe411dd7a386330132ad\": rpc error: code = NotFound desc = could not find container \"53d6769622928252d297348fee8c82c8081b9621cdecfe411dd7a386330132ad\": container with ID starting with 53d6769622928252d297348fee8c82c8081b9621cdecfe411dd7a386330132ad not found: ID does not exist" Oct 30 20:04:18.279033 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.278961 5091 scope.go:122] "RemoveContainer" containerID="05cc5cc5c831bc8c6a3ee2bf75528e70eb90d375b85397a00f6aec2e88cfd762" Oct 30 20:04:18.279432 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:04:18.279391 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05cc5cc5c831bc8c6a3ee2bf75528e70eb90d375b85397a00f6aec2e88cfd762\": container with ID starting with 05cc5cc5c831bc8c6a3ee2bf75528e70eb90d375b85397a00f6aec2e88cfd762 not found: ID does not exist" containerID="05cc5cc5c831bc8c6a3ee2bf75528e70eb90d375b85397a00f6aec2e88cfd762" Oct 30 20:04:18.279502 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.279441 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05cc5cc5c831bc8c6a3ee2bf75528e70eb90d375b85397a00f6aec2e88cfd762"} err="failed to get container status \"05cc5cc5c831bc8c6a3ee2bf75528e70eb90d375b85397a00f6aec2e88cfd762\": rpc error: code = NotFound desc = could not find container \"05cc5cc5c831bc8c6a3ee2bf75528e70eb90d375b85397a00f6aec2e88cfd762\": container with ID starting with 05cc5cc5c831bc8c6a3ee2bf75528e70eb90d375b85397a00f6aec2e88cfd762 not found: ID does not exist" Oct 30 20:04:18.287057 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.287031 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2"] Oct 30 20:04:18.292593 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.292569 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-836d67-openshift-default-ff47b8b4c-rqvr2"] Oct 30 20:04:18.419174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.407039 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:18.449373 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.449333 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="632cf328-468e-4865-a6ef-7358d679cb6d" path="/var/lib/kubelet/pods/632cf328-468e-4865-a6ef-7358d679cb6d/volumes" Oct 30 20:04:18.612598 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.612553 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:18.755953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:18.755917 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:19.261568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.260116 5091 generic.go:356] "Generic (PLEG): container finished" podID="ef039607-7c00-4a64-a63b-5e4a709ae26c" containerID="9a9a889a72a06c51fdd605d87b56053127dfc54a5538c1125c7d684012e9b7fe" exitCode=0 Oct 30 20:04:19.261568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.260180 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" event={"ID":"ef039607-7c00-4a64-a63b-5e4a709ae26c","Type":"ContainerDied","Data":"9a9a889a72a06c51fdd605d87b56053127dfc54a5538c1125c7d684012e9b7fe"} Oct 30 20:04:19.262696 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.261890 5091 scope.go:122] "RemoveContainer" containerID="9a9a889a72a06c51fdd605d87b56053127dfc54a5538c1125c7d684012e9b7fe" Oct 30 20:04:19.264297 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.264264 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:04:19.264432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.264363 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:04:19.264432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.264411 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:04:19.481174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.480292 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:19.833379 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.833312 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:19.861413 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.861370 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs"] Oct 30 20:04:19.862569 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.862543 5091 cpu_manager.go:414] "containerMap: removing container" podUID="632cf328-468e-4865-a6ef-7358d679cb6d" containerName="istio-proxy" Oct 30 20:04:19.862686 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.862570 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="632cf328-468e-4865-a6ef-7358d679cb6d" containerName="istio-proxy" Oct 30 20:04:19.862686 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.862618 5091 cpu_manager.go:414] "containerMap: removing container" podUID="632cf328-468e-4865-a6ef-7358d679cb6d" containerName="istio-proxy" Oct 30 20:04:19.862686 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.862627 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="632cf328-468e-4865-a6ef-7358d679cb6d" containerName="istio-proxy" Oct 30 20:04:19.862868 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.862825 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="632cf328-468e-4865-a6ef-7358d679cb6d" containerName="istio-proxy" Oct 30 20:04:19.867509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.867483 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" Oct 30 20:04:19.879655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.878133 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-517027-dockercfg-dkv8l\"" Oct 30 20:04:19.879655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.878355 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-517027\"" Oct 30 20:04:19.894094 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.894060 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs"] Oct 30 20:04:19.920950 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.920919 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p"] Oct 30 20:04:19.922442 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.922414 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="632cf328-468e-4865-a6ef-7358d679cb6d" containerName="istio-proxy" Oct 30 20:04:19.927042 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.927020 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" Oct 30 20:04:19.944788 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:19.944762 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p"] Oct 30 20:04:20.047210 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.029137 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h86dr\" (UniqueName: \"kubernetes.io/projected/fa98f754-341a-426e-91cf-1c914b3a2433-kube-api-access-h86dr\") pod \"payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs\" (UID: \"fa98f754-341a-426e-91cf-1c914b3a2433\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" Oct 30 20:04:20.242102 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.047515 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa98f754-341a-426e-91cf-1c914b3a2433-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs\" (UID: \"fa98f754-341a-426e-91cf-1c914b3a2433\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" Oct 30 20:04:20.243219 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.088892 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:20.296010 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.295967 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" event={"ID":"ef039607-7c00-4a64-a63b-5e4a709ae26c","Type":"ContainerStarted","Data":"ffa299f2cbebfda0637d31301684a8a6fdfbacaf38f60ed498fa29be758b22ab"} Oct 30 20:04:20.345760 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.344552 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h86dr\" (UniqueName: \"kubernetes.io/projected/fa98f754-341a-426e-91cf-1c914b3a2433-kube-api-access-h86dr\") pod \"payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs\" (UID: \"fa98f754-341a-426e-91cf-1c914b3a2433\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" Oct 30 20:04:20.345760 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.344647 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa98f754-341a-426e-91cf-1c914b3a2433-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs\" (UID: \"fa98f754-341a-426e-91cf-1c914b3a2433\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" Oct 30 20:04:20.345760 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.344885 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1b31d4-eafb-4f36-922a-c7fdcc8c007f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p\" (UID: \"0b1b31d4-eafb-4f36-922a-c7fdcc8c007f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" Oct 30 20:04:20.345760 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.344919 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh8ld\" (UniqueName: \"kubernetes.io/projected/0b1b31d4-eafb-4f36-922a-c7fdcc8c007f-kube-api-access-kh8ld\") pod \"payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p\" (UID: \"0b1b31d4-eafb-4f36-922a-c7fdcc8c007f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" Oct 30 20:04:20.346140 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.345928 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa98f754-341a-426e-91cf-1c914b3a2433-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs\" (UID: \"fa98f754-341a-426e-91cf-1c914b3a2433\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" Oct 30 20:04:20.359887 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.359858 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h86dr\" (UniqueName: \"kubernetes.io/projected/fa98f754-341a-426e-91cf-1c914b3a2433-kube-api-access-h86dr\") pod \"payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs\" (UID: \"fa98f754-341a-426e-91cf-1c914b3a2433\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" Oct 30 20:04:20.446407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.446379 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1b31d4-eafb-4f36-922a-c7fdcc8c007f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p\" (UID: \"0b1b31d4-eafb-4f36-922a-c7fdcc8c007f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" Oct 30 20:04:20.446616 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.446412 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh8ld\" (UniqueName: \"kubernetes.io/projected/0b1b31d4-eafb-4f36-922a-c7fdcc8c007f-kube-api-access-kh8ld\") pod \"payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p\" (UID: \"0b1b31d4-eafb-4f36-922a-c7fdcc8c007f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" Oct 30 20:04:20.447312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.447280 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1b31d4-eafb-4f36-922a-c7fdcc8c007f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p\" (UID: \"0b1b31d4-eafb-4f36-922a-c7fdcc8c007f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" Oct 30 20:04:20.476908 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.476879 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh8ld\" (UniqueName: \"kubernetes.io/projected/0b1b31d4-eafb-4f36-922a-c7fdcc8c007f-kube-api-access-kh8ld\") pod \"payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p\" (UID: \"0b1b31d4-eafb-4f36-922a-c7fdcc8c007f\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" Oct 30 20:04:20.504783 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.504758 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" Oct 30 20:04:20.545838 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.545808 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" Oct 30 20:04:20.681791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.681754 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:20.807297 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.806750 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:04:20.809176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.808934 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 30 20:04:20.809176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.809007 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" podUID="ef039607-7c00-4a64-a63b-5e4a709ae26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 30 20:04:20.829277 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:20.828811 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs"] Oct 30 20:04:21.021902 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:04:21.021868 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b1b31d4_eafb_4f36_922a_c7fdcc8c007f.slice/crio-7a15fe607751af3f148f41c71a9b4c6358ea4dad37a476a7034591f1b38b65ae WatchSource:0}: Error finding container 7a15fe607751af3f148f41c71a9b4c6358ea4dad37a476a7034591f1b38b65ae: Status 404 returned error can't find the container with id 7a15fe607751af3f148f41c71a9b4c6358ea4dad37a476a7034591f1b38b65ae Oct 30 20:04:21.022629 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:21.022598 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p"] Oct 30 20:04:21.308493 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:21.308118 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" event={"ID":"0b1b31d4-eafb-4f36-922a-c7fdcc8c007f","Type":"ContainerStarted","Data":"8a56a4980659f1dce4e6a96661e21c43e4226e290ebdfd016b44af1132cf2ab4"} Oct 30 20:04:21.308493 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:21.308200 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" event={"ID":"0b1b31d4-eafb-4f36-922a-c7fdcc8c007f","Type":"ContainerStarted","Data":"7a15fe607751af3f148f41c71a9b4c6358ea4dad37a476a7034591f1b38b65ae"} Oct 30 20:04:21.308493 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:21.308256 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" Oct 30 20:04:21.310446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:21.310412 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" event={"ID":"fa98f754-341a-426e-91cf-1c914b3a2433","Type":"ContainerStarted","Data":"7301ad28de5127e792170f28099af85f9db7cff71f9a19a0a0ec9ddf24771a58"} Oct 30 20:04:21.310589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:21.310467 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" event={"ID":"fa98f754-341a-426e-91cf-1c914b3a2433","Type":"ContainerStarted","Data":"140fae30d542744c3f0d8332b4da592b2ea8d7bf62b0f626aa3311b307f4950c"} Oct 30 20:04:21.310589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:21.310495 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" Oct 30 20:04:21.338327 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:21.338275 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" podStartSLOduration=2.338260657 podStartE2EDuration="2.338260657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:21.332795295 +0000 UTC m=+1981.028003244" watchObservedRunningTime="2026-10-30 20:04:21.338260657 +0000 UTC m=+1981.033468614" Oct 30 20:04:21.387174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:21.381230 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" podStartSLOduration=2.362123232 podStartE2EDuration="2.362123232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:21.35895188 +0000 UTC m=+1981.054159839" watchObservedRunningTime="2026-10-30 20:04:21.362123232 +0000 UTC m=+1981.057331191" Oct 30 20:04:21.785247 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:21.782618 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 30 20:04:21.785247 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:21.782695 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" podUID="ef039607-7c00-4a64-a63b-5e4a709ae26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 30 20:04:21.785247 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:21.784447 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:22.098033 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:22.097951 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:04:22.579217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:22.579158 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:22.777674 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:22.777630 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 30 20:04:22.777957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:22.777924 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" podUID="ef039607-7c00-4a64-a63b-5e4a709ae26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 30 20:04:23.467172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:23.460233 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:04:23.598169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:23.598042 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:23.777712 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:23.777676 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 30 20:04:23.777926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:23.777735 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" podUID="ef039607-7c00-4a64-a63b-5e4a709ae26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 30 20:04:24.426165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:24.420756 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:24.440557 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:24.440529 5091 scope.go:122] "RemoveContainer" containerID="e39752ca634098647dd8be4bca1c86a9ab6e4f99d6b7e40ce96e05d3c6cd89d1" Oct 30 20:04:24.509283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:24.509257 5091 scope.go:122] "RemoveContainer" containerID="1c9cf0c7574d4ff724fc6ad7483cd81c08c3c2f7c0e0d43ed1d14d5561c6b593" Oct 30 20:04:24.777752 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:24.777710 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 30 20:04:24.778227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:24.777776 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" podUID="ef039607-7c00-4a64-a63b-5e4a709ae26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 30 20:04:25.303185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:25.294665 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:25.777817 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:25.777771 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" start-of-body= Oct 30 20:04:25.778380 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:25.777848 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" podUID="ef039607-7c00-4a64-a63b-5e4a709ae26c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.17:15021/healthz/ready\": dial tcp 10.132.1.17:15021: connect: connection refused" Oct 30 20:04:26.219239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:26.214419 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:26.782163 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:26.782115 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:04:26.783377 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:26.783343 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:04:27.196943 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:27.196849 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:28.023174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:28.019124 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:28.929739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:28.929467 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:29.922311 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:29.922275 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:30.774121 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:30.774078 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:31.833711 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:31.833662 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:32.321462 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:32.321426 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" Oct 30 20:04:32.321638 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:32.321583 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" Oct 30 20:04:32.796170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:32.791788 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:33.844416 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:33.844384 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:34.427306 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:34.427269 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:34.641163 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:34.641109 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:34.894795 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:34.894760 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:35.100992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:35.100954 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:35.311132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:35.311095 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:35.607030 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:35.606571 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:35.803169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:35.800770 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:36.053185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.052175 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:36.236487 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.236455 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh"] Oct 30 20:04:36.243029 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.242998 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:36.251401 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.250999 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"agc-cred-ec9289edc0a9\"" Oct 30 20:04:36.288035 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.287982 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh"] Oct 30 20:04:36.304966 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.304887 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:36.377327 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.377277 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d5ac221-47f7-46e1-abd4-500f1513e78d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:36.377693 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.377665 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45qnk\" (UniqueName: \"kubernetes.io/projected/3d5ac221-47f7-46e1-abd4-500f1513e78d-kube-api-access-45qnk\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:36.378011 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.377989 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d5ac221-47f7-46e1-abd4-500f1513e78d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:36.378194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.378175 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5ac221-47f7-46e1-abd4-500f1513e78d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:36.484239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.484192 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d5ac221-47f7-46e1-abd4-500f1513e78d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:36.484470 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.484310 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45qnk\" (UniqueName: \"kubernetes.io/projected/3d5ac221-47f7-46e1-abd4-500f1513e78d-kube-api-access-45qnk\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:36.484541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.484477 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d5ac221-47f7-46e1-abd4-500f1513e78d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:36.484541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.484517 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5ac221-47f7-46e1-abd4-500f1513e78d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:36.485114 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.485085 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d5ac221-47f7-46e1-abd4-500f1513e78d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:36.485452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.485428 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5ac221-47f7-46e1-abd4-500f1513e78d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:36.487705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.487670 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d5ac221-47f7-46e1-abd4-500f1513e78d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:36.498885 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.498850 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45qnk\" (UniqueName: \"kubernetes.io/projected/3d5ac221-47f7-46e1-abd4-500f1513e78d-kube-api-access-45qnk\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:36.525184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.525154 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:36.597218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.596381 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:36.697070 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.696967 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:36.880225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.879790 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:36.985403 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:04:36.985371 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d5ac221_47f7_46e1_abd4_500f1513e78d.slice/crio-455aa565fbbeaae9ac17da6f27c8fb4d1946dc65fe6bb67d509cf7aa6ccf53c0 WatchSource:0}: Error finding container 455aa565fbbeaae9ac17da6f27c8fb4d1946dc65fe6bb67d509cf7aa6ccf53c0: Status 404 returned error can't find the container with id 455aa565fbbeaae9ac17da6f27c8fb4d1946dc65fe6bb67d509cf7aa6ccf53c0 Oct 30 20:04:36.987860 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:36.987819 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh"] Oct 30 20:04:37.332419 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:37.332387 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:37.546166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:37.544703 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:37.750229 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:37.743209 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" event={"ID":"3d5ac221-47f7-46e1-abd4-500f1513e78d","Type":"ContainerStarted","Data":"429e936c9e7860d43805f089020c786a596c6182f882491aab14e72ce4d823b8"} Oct 30 20:04:37.750229 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:37.743280 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" event={"ID":"3d5ac221-47f7-46e1-abd4-500f1513e78d","Type":"ContainerStarted","Data":"455aa565fbbeaae9ac17da6f27c8fb4d1946dc65fe6bb67d509cf7aa6ccf53c0"} Oct 30 20:04:37.754184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:37.752546 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:37.854585 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:37.845774 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" podStartSLOduration=1.845756101 podStartE2EDuration="1.845756101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:37.792668093 +0000 UTC m=+1997.487876051" watchObservedRunningTime="2026-10-30 20:04:37.845756101 +0000 UTC m=+1997.540964060" Oct 30 20:04:37.925415 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:37.925378 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4"] Oct 30 20:04:37.930378 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:37.930342 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:37.938076 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:37.937844 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4"] Oct 30 20:04:37.941228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:37.941204 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-2a2286\"/\"agc-cred-184285b93bd5\"" Oct 30 20:04:37.941351 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:37.941336 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2a2286\"/\"payload-processing-external-model-plugins-e2e-extmod-b-2a2286\"" Oct 30 20:04:37.941419 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:37.941390 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2a2286\"/\"routing-overlay\"" Oct 30 20:04:37.941419 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:37.941413 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2a2286\"/\"openshift-service-ca.crt\"" Oct 30 20:04:37.941633 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:37.941611 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-2a2286\"/\"kube-root-ca.crt\"" Oct 30 20:04:37.990414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:37.990336 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:38.100313 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.098263 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63684302-4590-4e0b-b314-8b3a574ca02c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:38.100313 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.098335 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcpbd\" (UniqueName: \"kubernetes.io/projected/63684302-4590-4e0b-b314-8b3a574ca02c-kube-api-access-lcpbd\") pod \"payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:38.100313 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.098594 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63684302-4590-4e0b-b314-8b3a574ca02c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:38.100313 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.098638 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63684302-4590-4e0b-b314-8b3a574ca02c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:38.205193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.201656 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:38.205193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.202962 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63684302-4590-4e0b-b314-8b3a574ca02c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:38.205193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.203013 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63684302-4590-4e0b-b314-8b3a574ca02c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:38.205193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.203178 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63684302-4590-4e0b-b314-8b3a574ca02c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:38.205193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.203223 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcpbd\" (UniqueName: \"kubernetes.io/projected/63684302-4590-4e0b-b314-8b3a574ca02c-kube-api-access-lcpbd\") pod \"payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:38.246482 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.219860 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63684302-4590-4e0b-b314-8b3a574ca02c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:38.247990 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.247925 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63684302-4590-4e0b-b314-8b3a574ca02c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:38.248255 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.248230 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63684302-4590-4e0b-b314-8b3a574ca02c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:38.271624 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.271597 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcpbd\" (UniqueName: \"kubernetes.io/projected/63684302-4590-4e0b-b314-8b3a574ca02c-kube-api-access-lcpbd\") pod \"payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:38.302167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.295996 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:38.721173 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:04:38.721126 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63684302_4590_4e0b_b314_8b3a574ca02c.slice/crio-049dfa419470cebf185afac97f112379901486d383aefef62eb1ff24487b8d73 WatchSource:0}: Error finding container 049dfa419470cebf185afac97f112379901486d383aefef62eb1ff24487b8d73: Status 404 returned error can't find the container with id 049dfa419470cebf185afac97f112379901486d383aefef62eb1ff24487b8d73 Oct 30 20:04:38.722503 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.722475 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4"] Oct 30 20:04:38.759864 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.759833 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" event={"ID":"63684302-4590-4e0b-b314-8b3a574ca02c","Type":"ContainerStarted","Data":"049dfa419470cebf185afac97f112379901486d383aefef62eb1ff24487b8d73"} Oct 30 20:04:38.997124 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:38.997013 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:39.772216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:39.771716 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" event={"ID":"63684302-4590-4e0b-b314-8b3a574ca02c","Type":"ContainerStarted","Data":"78a1227891fffc67d4529c667bd554105a8be2fd0041001f55e46b2d7147e37c"} Oct 30 20:04:39.772216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:39.771973 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:39.798346 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:39.798284 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" podStartSLOduration=2.798268507 podStartE2EDuration="2.798268507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:04:39.796378275 +0000 UTC m=+1999.491586232" watchObservedRunningTime="2026-10-30 20:04:39.798268507 +0000 UTC m=+1999.493476466" Oct 30 20:04:39.884540 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:39.884494 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:40.856992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:40.856910 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:42.144828 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:42.144785 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:42.613728 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:42.611979 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": read tcp 10.132.0.2:52824->10.132.1.21:15021: read: connection reset by peer" start-of-body= Oct 30 20:04:42.613728 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:42.612052 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" podUID="1888b3fa-2409-4051-99ff-22d37072bd4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": read tcp 10.132.0.2:52824->10.132.1.21:15021: read: connection reset by peer" Oct 30 20:04:42.802887 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:42.802809 5091 generic.go:356] "Generic (PLEG): container finished" podID="1888b3fa-2409-4051-99ff-22d37072bd4f" containerID="6a89dcf2a18e9415269f153f1b7a5ed5f61c7ce6c2408cbbd43281481255cfd6" exitCode=0 Oct 30 20:04:42.803030 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:42.802882 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" event={"ID":"1888b3fa-2409-4051-99ff-22d37072bd4f","Type":"ContainerDied","Data":"6a89dcf2a18e9415269f153f1b7a5ed5f61c7ce6c2408cbbd43281481255cfd6"} Oct 30 20:04:42.803412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:42.803396 5091 scope.go:122] "RemoveContainer" containerID="6a89dcf2a18e9415269f153f1b7a5ed5f61c7ce6c2408cbbd43281481255cfd6" Oct 30 20:04:42.805390 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:42.805359 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:04:42.805499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:42.805429 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:04:42.805499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:42.805460 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:04:43.244059 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:43.243968 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:43.435438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:43.435401 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:43.860170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:43.847844 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" event={"ID":"1888b3fa-2409-4051-99ff-22d37072bd4f","Type":"ContainerStarted","Data":"6214aa32fc9e307705e938777fbd2c2575e52d296e1b1c41b91c5993c8df8c87"} Oct 30 20:04:44.339286 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:44.338759 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:44.576196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:44.571366 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:04:44.576196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:44.571943 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 20:04:44.576196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:44.572033 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" podUID="1888b3fa-2409-4051-99ff-22d37072bd4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 20:04:45.309900 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:45.309855 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:45.566614 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:45.566524 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 20:04:45.567193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:45.567138 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" podUID="1888b3fa-2409-4051-99ff-22d37072bd4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 20:04:46.001666 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:04:46.001626 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:04:46.442775 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:46.442685 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:46.575168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:46.574585 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 20:04:46.575168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:46.574653 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" podUID="1888b3fa-2409-4051-99ff-22d37072bd4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 20:04:47.574201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:47.562998 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:47.574201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:47.569835 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 20:04:47.574201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:47.569885 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" podUID="1888b3fa-2409-4051-99ff-22d37072bd4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 20:04:48.569172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:48.567603 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" start-of-body= Oct 30 20:04:48.569172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:48.567664 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" podUID="1888b3fa-2409-4051-99ff-22d37072bd4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.21:15021/healthz/ready\": dial tcp 10.132.1.21:15021: connect: connection refused" Oct 30 20:04:48.609981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:48.609930 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:49.532168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:49.531336 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:49.572630 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:49.572593 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:04:49.573114 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:49.573058 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:04:49.573724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:49.573701 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:04:49.796223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:49.796103 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:04:49.873126 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:49.873087 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq"] Oct 30 20:04:50.557559 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:50.557526 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:50.787321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:50.787290 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:04:51.697175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:51.688710 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:52.526073 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:52.526023 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:53.531187 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:53.530784 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:54.519953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:54.519916 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:54.874195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:54.874065 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" podUID="a99dbd70-1fbe-48e7-ba22-011eed682112" containerName="payload-processing" containerID="cri-o://96c413cc2f5da02dd5c231ebbf0c0b55be5106486fdf62efebfdfeaf8148c99a" gracePeriod=25 Oct 30 20:04:55.186823 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:55.186792 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:04:55.313483 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:55.313450 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99dbd70-1fbe-48e7-ba22-011eed682112-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a99dbd70-1fbe-48e7-ba22-011eed682112-provider-credentials\") pod \"a99dbd70-1fbe-48e7-ba22-011eed682112\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " Oct 30 20:04:55.313677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:55.313549 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99dbd70-1fbe-48e7-ba22-011eed682112-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a99dbd70-1fbe-48e7-ba22-011eed682112-routing-overlay-volume\") pod \"a99dbd70-1fbe-48e7-ba22-011eed682112\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " Oct 30 20:04:55.313677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:55.313674 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a99dbd70-1fbe-48e7-ba22-011eed682112-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99dbd70-1fbe-48e7-ba22-011eed682112-plugins-config-volume\") pod \"a99dbd70-1fbe-48e7-ba22-011eed682112\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " Oct 30 20:04:55.313772 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:55.313709 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a99dbd70-1fbe-48e7-ba22-011eed682112-kube-api-access-5hd2w\" (UniqueName: \"kubernetes.io/projected/a99dbd70-1fbe-48e7-ba22-011eed682112-kube-api-access-5hd2w\") pod \"a99dbd70-1fbe-48e7-ba22-011eed682112\" (UID: \"a99dbd70-1fbe-48e7-ba22-011eed682112\") " Oct 30 20:04:55.313993 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:55.313965 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99dbd70-1fbe-48e7-ba22-011eed682112-routing-overlay-volume" pod "a99dbd70-1fbe-48e7-ba22-011eed682112" (UID: "a99dbd70-1fbe-48e7-ba22-011eed682112"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:55.314126 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:55.314098 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a99dbd70-1fbe-48e7-ba22-011eed682112-plugins-config-volume" pod "a99dbd70-1fbe-48e7-ba22-011eed682112" (UID: "a99dbd70-1fbe-48e7-ba22-011eed682112"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:04:55.314354 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:55.314331 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a99dbd70-1fbe-48e7-ba22-011eed682112-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:55.314432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:55.314358 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a99dbd70-1fbe-48e7-ba22-011eed682112-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:55.316001 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:55.315975 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99dbd70-1fbe-48e7-ba22-011eed682112-kube-api-access-5hd2w" pod "a99dbd70-1fbe-48e7-ba22-011eed682112" (UID: "a99dbd70-1fbe-48e7-ba22-011eed682112"). InnerVolumeSpecName "kube-api-access-5hd2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:55.316106 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:55.316011 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a99dbd70-1fbe-48e7-ba22-011eed682112-provider-credentials" pod "a99dbd70-1fbe-48e7-ba22-011eed682112" (UID: "a99dbd70-1fbe-48e7-ba22-011eed682112"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:04:55.327820 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:55.327681 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:55.418708 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:55.417792 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hd2w\" (UniqueName: \"kubernetes.io/projected/a99dbd70-1fbe-48e7-ba22-011eed682112-kube-api-access-5hd2w\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:55.418708 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:55.417833 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a99dbd70-1fbe-48e7-ba22-011eed682112-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:04:56.000533 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:56.000501 5091 generic.go:356] "Generic (PLEG): container finished" podID="a99dbd70-1fbe-48e7-ba22-011eed682112" containerID="96c413cc2f5da02dd5c231ebbf0c0b55be5106486fdf62efebfdfeaf8148c99a" exitCode=0 Oct 30 20:04:56.001007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:56.000562 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" Oct 30 20:04:56.001007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:56.000586 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" event={"ID":"a99dbd70-1fbe-48e7-ba22-011eed682112","Type":"ContainerDied","Data":"96c413cc2f5da02dd5c231ebbf0c0b55be5106486fdf62efebfdfeaf8148c99a"} Oct 30 20:04:56.001007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:56.000651 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq" event={"ID":"a99dbd70-1fbe-48e7-ba22-011eed682112","Type":"ContainerDied","Data":"28d483f3a1b83f6c888bf750e9be3a84faedd922e95a8b770c1d39f46a452fb7"} Oct 30 20:04:56.001007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:56.000675 5091 scope.go:122] "RemoveContainer" containerID="96c413cc2f5da02dd5c231ebbf0c0b55be5106486fdf62efebfdfeaf8148c99a" Oct 30 20:04:56.014080 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:56.014055 5091 scope.go:122] "RemoveContainer" containerID="96c413cc2f5da02dd5c231ebbf0c0b55be5106486fdf62efebfdfeaf8148c99a" Oct 30 20:04:56.014512 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:04:56.014480 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96c413cc2f5da02dd5c231ebbf0c0b55be5106486fdf62efebfdfeaf8148c99a\": container with ID starting with 96c413cc2f5da02dd5c231ebbf0c0b55be5106486fdf62efebfdfeaf8148c99a not found: ID does not exist" containerID="96c413cc2f5da02dd5c231ebbf0c0b55be5106486fdf62efebfdfeaf8148c99a" Oct 30 20:04:56.014654 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:56.014523 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96c413cc2f5da02dd5c231ebbf0c0b55be5106486fdf62efebfdfeaf8148c99a"} err="failed to get container status \"96c413cc2f5da02dd5c231ebbf0c0b55be5106486fdf62efebfdfeaf8148c99a\": rpc error: code = NotFound desc = could not find container \"96c413cc2f5da02dd5c231ebbf0c0b55be5106486fdf62efebfdfeaf8148c99a\": container with ID starting with 96c413cc2f5da02dd5c231ebbf0c0b55be5106486fdf62efebfdfeaf8148c99a not found: ID does not exist" Oct 30 20:04:56.048742 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:56.048713 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq"] Oct 30 20:04:56.055660 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:56.055637 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58568htdq"] Oct 30 20:04:56.256231 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:56.255211 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:56.444957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:56.444926 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a99dbd70-1fbe-48e7-ba22-011eed682112" path="/var/lib/kubelet/pods/a99dbd70-1fbe-48e7-ba22-011eed682112/volumes" Oct 30 20:04:57.063174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:57.055266 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:57.927170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:57.926032 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:58.838282 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:58.836269 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:04:59.721967 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:04:59.721931 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:00.650122 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:00.650082 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:01.655103 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:01.655065 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:02.672174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:02.668359 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:03.499592 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:03.499553 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:04.736879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:04.736839 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:05.903103 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:05.903046 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:06.691806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:06.691768 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:06.864042 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:06.864003 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:07.507833 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.507803 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm"] Oct 30 20:05:07.508848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.508820 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a99dbd70-1fbe-48e7-ba22-011eed682112" containerName="payload-processing" Oct 30 20:05:07.508848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.508846 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a99dbd70-1fbe-48e7-ba22-011eed682112" containerName="payload-processing" Oct 30 20:05:07.509079 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.509066 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a99dbd70-1fbe-48e7-ba22-011eed682112" containerName="payload-processing" Oct 30 20:05:07.514388 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.514368 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:07.523863 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.523840 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"agc-cred-068ada534966\"" Oct 30 20:05:07.525543 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.525520 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm"] Oct 30 20:05:07.595821 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.595786 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ch8kb\" (UniqueName: \"kubernetes.io/projected/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-kube-api-access-ch8kb\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:07.595996 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.595975 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:07.596133 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.596114 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:07.596306 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.596285 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:07.698132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.698098 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ch8kb\" (UniqueName: \"kubernetes.io/projected/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-kube-api-access-ch8kb\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:07.698312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.698230 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:07.698357 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.698317 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:07.698404 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.698390 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:07.698932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.698902 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:07.699205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.699184 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:07.701332 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.701303 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:07.711896 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.711870 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ch8kb\" (UniqueName: \"kubernetes.io/projected/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-kube-api-access-ch8kb\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:07.833097 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.833011 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:07.879546 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:07.879509 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:08.233549 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:05:08.233518 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ab2e57_b1fb_4b5c_bbae_3f6a7c32e3dc.slice/crio-a5e9ee124da30b03a84c2df8ed44972049b8b1203369cf3b9502de5f768caa54 WatchSource:0}: Error finding container a5e9ee124da30b03a84c2df8ed44972049b8b1203369cf3b9502de5f768caa54: Status 404 returned error can't find the container with id a5e9ee124da30b03a84c2df8ed44972049b8b1203369cf3b9502de5f768caa54 Oct 30 20:05:08.233890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:08.233872 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm"] Oct 30 20:05:08.589051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:08.589012 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:08.849273 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:08.848203 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:09.041377 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:09.041328 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:09.192456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:09.192354 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" event={"ID":"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc","Type":"ContainerStarted","Data":"3260cbee6ac85fdec3db92654ebe711e720f8f47fb5bed6f28f5500de9576f45"} Oct 30 20:05:09.192456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:09.192418 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" event={"ID":"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc","Type":"ContainerStarted","Data":"a5e9ee124da30b03a84c2df8ed44972049b8b1203369cf3b9502de5f768caa54"} Oct 30 20:05:09.192726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:09.192460 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:09.219250 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:09.219191 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" podStartSLOduration=2.219134493 podStartE2EDuration="2.219134493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:09.216200411 +0000 UTC m=+2028.911408372" watchObservedRunningTime="2026-10-30 20:05:09.219134493 +0000 UTC m=+2028.914342452" Oct 30 20:05:09.585951 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:09.585908 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:09.804548 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:09.804510 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:10.280978 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.280943 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs"] Oct 30 20:05:10.285906 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.285885 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:10.297407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.297382 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-969878\"/\"payload-processing-external-model-e2e--5b5eb86f-dockercfg-tpqsc\"" Oct 30 20:05:10.297547 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.297457 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-969878\"/\"payload-processing-external-model-plugins-e2e-extmod-b-969878\"" Oct 30 20:05:10.297547 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.297532 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-969878\"/\"routing-overlay\"" Oct 30 20:05:10.297691 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.297665 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-969878\"/\"openshift-service-ca.crt\"" Oct 30 20:05:10.297761 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.297676 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-969878\"/\"agc-cred-5fdacea3cbca\"" Oct 30 20:05:10.297816 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.297803 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-969878\"/\"kube-root-ca.crt\"" Oct 30 20:05:10.297963 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.297945 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-969878\"/\"agc-cred-565825ce94ca\"" Oct 30 20:05:10.305108 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.305084 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs"] Oct 30 20:05:10.314254 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.314219 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qkj7\" (UniqueName: \"kubernetes.io/projected/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-kube-api-access-6qkj7\") pod \"payload-processing-external-model-e2e-extmod-b-969878-6888z87hs\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:10.314389 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.314270 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-969878-6888z87hs\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:10.314463 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.314413 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-969878-6888z87hs\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:10.314463 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.314451 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-969878-6888z87hs\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:10.416531 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.416493 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-969878-6888z87hs\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:10.416829 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.416803 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-969878-6888z87hs\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:10.417268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.417241 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qkj7\" (UniqueName: \"kubernetes.io/projected/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-kube-api-access-6qkj7\") pod \"payload-processing-external-model-e2e-extmod-b-969878-6888z87hs\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:10.417404 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.417291 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-969878-6888z87hs\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:10.418499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.418452 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-969878-6888z87hs\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:10.418499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.418458 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-969878-6888z87hs\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:10.421408 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.421386 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-969878-6888z87hs\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:10.470817 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.470782 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qkj7\" (UniqueName: \"kubernetes.io/projected/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-kube-api-access-6qkj7\") pod \"payload-processing-external-model-e2e-extmod-b-969878-6888z87hs\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:10.561301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.561189 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:10.606174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.606124 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:10.928555 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:05:10.928525 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7491db9_6f6b_4f3b_bddf_397c96e2ac87.slice/crio-3022ae9d1e4fa3ebcf71bc28ee1d0e09db0375e86119210676474b2e878b7cfe WatchSource:0}: Error finding container 3022ae9d1e4fa3ebcf71bc28ee1d0e09db0375e86119210676474b2e878b7cfe: Status 404 returned error can't find the container with id 3022ae9d1e4fa3ebcf71bc28ee1d0e09db0375e86119210676474b2e878b7cfe Oct 30 20:05:10.930139 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:10.930112 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs"] Oct 30 20:05:11.118653 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.118618 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:11.214081 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.213998 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" event={"ID":"e7491db9-6f6b-4f3b-bddf-397c96e2ac87","Type":"ContainerStarted","Data":"b03d119c8711a9d300a36fe7082edb461633459226ef36276c45e65a4f07e606"} Oct 30 20:05:11.214081 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.214056 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" event={"ID":"e7491db9-6f6b-4f3b-bddf-397c96e2ac87","Type":"ContainerStarted","Data":"3022ae9d1e4fa3ebcf71bc28ee1d0e09db0375e86119210676474b2e878b7cfe"} Oct 30 20:05:11.214081 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.214076 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:11.268171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.267466 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" podStartSLOduration=1.267447489 podStartE2EDuration="1.267447489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:11.256372851 +0000 UTC m=+2030.951580811" watchObservedRunningTime="2026-10-30 20:05:11.267447489 +0000 UTC m=+2030.962655449" Oct 30 20:05:11.593472 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.593431 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72"] Oct 30 20:05:11.618365 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.618298 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.625901 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.625871 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72"] Oct 30 20:05:11.634615 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.634592 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-5f9b3c-openshift-default-dockercfg-tdmlf\"" Oct 30 20:05:11.797172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.747974 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-istio-envoy\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.797172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.748055 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-credential-socket\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.797172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.748089 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w62t\" (UniqueName: \"kubernetes.io/projected/5109436f-1c85-4413-a7e7-f62dac25203f-kube-api-access-8w62t\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.797172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.794226 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:11.797172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.794627 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-workload-socket\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.797172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.794722 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5109436f-1c85-4413-a7e7-f62dac25203f-istio-token\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.797172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.794785 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5109436f-1c85-4413-a7e7-f62dac25203f-istiod-ca-cert\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.797172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.794818 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-workload-certs\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.797172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.794844 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-istio-data\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.797172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.794884 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5109436f-1c85-4413-a7e7-f62dac25203f-istio-podinfo\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.934227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.919306 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-istio-envoy\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.934875 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.934845 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-credential-socket\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.944178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.935012 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w62t\" (UniqueName: \"kubernetes.io/projected/5109436f-1c85-4413-a7e7-f62dac25203f-kube-api-access-8w62t\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.944543 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.944514 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-workload-socket\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.944753 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.944733 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5109436f-1c85-4413-a7e7-f62dac25203f-istio-token\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.944914 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.944898 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5109436f-1c85-4413-a7e7-f62dac25203f-istiod-ca-cert\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.945027 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.945013 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-workload-certs\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.956176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.945164 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-istio-data\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.956407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.956381 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5109436f-1c85-4413-a7e7-f62dac25203f-istio-podinfo\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.956752 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.945942 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-workload-socket\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.956881 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.946552 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5109436f-1c85-4413-a7e7-f62dac25203f-istiod-ca-cert\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.956978 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.946765 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-workload-certs\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.957063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.946971 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-istio-data\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.957170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.936552 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-credential-socket\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:11.993180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:11.989002 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-istio-envoy\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:12.044185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:12.008307 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w62t\" (UniqueName: \"kubernetes.io/projected/5109436f-1c85-4413-a7e7-f62dac25203f-kube-api-access-8w62t\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:12.044185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:12.024186 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:12.054197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:12.051843 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5109436f-1c85-4413-a7e7-f62dac25203f-istio-token\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:12.054197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:12.051843 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5109436f-1c85-4413-a7e7-f62dac25203f-istio-podinfo\") pod \"e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:12.243655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:12.243571 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:12.511296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:12.511100 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:12.606688 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:12.606602 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72"] Oct 30 20:05:12.607899 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:05:12.607873 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5109436f_1c85_4413_a7e7_f62dac25203f.slice/crio-c6dfc7d70ee86c32d3b0132c9b267114ee7c29683798db903bfc5fd6d6f285fc WatchSource:0}: Error finding container c6dfc7d70ee86c32d3b0132c9b267114ee7c29683798db903bfc5fd6d6f285fc: Status 404 returned error can't find the container with id c6dfc7d70ee86c32d3b0132c9b267114ee7c29683798db903bfc5fd6d6f285fc Oct 30 20:05:12.611551 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:12.611518 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:05:12.611650 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:12.611597 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:05:12.611650 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:12.611643 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:05:13.012809 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:13.012770 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:13.235799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:13.235760 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" event={"ID":"5109436f-1c85-4413-a7e7-f62dac25203f","Type":"ContainerStarted","Data":"0f4009c3de1f21e6df9055dbab35fb4077d58766c61711a5fbce24a99dcb56e7"} Oct 30 20:05:13.235974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:13.235827 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" event={"ID":"5109436f-1c85-4413-a7e7-f62dac25203f","Type":"ContainerStarted","Data":"c6dfc7d70ee86c32d3b0132c9b267114ee7c29683798db903bfc5fd6d6f285fc"} Oct 30 20:05:13.243820 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:13.243789 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:13.245130 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:13.245104 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 20:05:13.245290 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:13.245180 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 20:05:13.277855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:13.277742 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" podStartSLOduration=2.277720813 podStartE2EDuration="2.277720813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:13.269958247 +0000 UTC m=+2032.965166205" watchObservedRunningTime="2026-10-30 20:05:13.277720813 +0000 UTC m=+2032.972928772" Oct 30 20:05:14.035888 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:14.035848 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:14.212292 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:14.212258 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:14.255541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:14.255509 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:14.278284 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:14.278251 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:14.279605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:14.279585 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:05:15.180663 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:15.180628 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:15.784949 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:15.784912 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:15.977494 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:05:15.977458 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:05:16.051554 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:16.051460 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:16.328689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:16.328593 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:17.059750 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.059711 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:17.127606 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.127568 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth"] Oct 30 20:05:17.134137 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.133562 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" Oct 30 20:05:17.222183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.214561 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-5f9b3c-dockercfg-g4dlw\"" Oct 30 20:05:17.238850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.238743 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth"] Oct 30 20:05:17.253372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.246078 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-5f9b3c\"" Oct 30 20:05:17.253372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.252838 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94b5b35-f3af-48c4-a865-c278844435ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth\" (UID: \"f94b5b35-f3af-48c4-a865-c278844435ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" Oct 30 20:05:17.253372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.253078 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4cpq\" (UniqueName: \"kubernetes.io/projected/f94b5b35-f3af-48c4-a865-c278844435ed-kube-api-access-z4cpq\") pod \"payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth\" (UID: \"f94b5b35-f3af-48c4-a865-c278844435ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" Oct 30 20:05:17.308353 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.299882 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:17.360217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.359263 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4cpq\" (UniqueName: \"kubernetes.io/projected/f94b5b35-f3af-48c4-a865-c278844435ed-kube-api-access-z4cpq\") pod \"payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth\" (UID: \"f94b5b35-f3af-48c4-a865-c278844435ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" Oct 30 20:05:17.360217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.359520 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94b5b35-f3af-48c4-a865-c278844435ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth\" (UID: \"f94b5b35-f3af-48c4-a865-c278844435ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" Oct 30 20:05:17.360760 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.360673 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94b5b35-f3af-48c4-a865-c278844435ed-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth\" (UID: \"f94b5b35-f3af-48c4-a865-c278844435ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" Oct 30 20:05:17.510203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.497974 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4cpq\" (UniqueName: \"kubernetes.io/projected/f94b5b35-f3af-48c4-a865-c278844435ed-kube-api-access-z4cpq\") pod \"payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth\" (UID: \"f94b5b35-f3af-48c4-a865-c278844435ed\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" Oct 30 20:05:17.516874 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.516614 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" Oct 30 20:05:17.678510 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.678357 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:17.940658 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:17.940628 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth"] Oct 30 20:05:18.265322 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:18.265274 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:05:18.265614 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:18.265354 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" podUID="43e01282-cbf5-43e3-9815-32c7d4439800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:05:18.497062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:18.497022 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" event={"ID":"f94b5b35-f3af-48c4-a865-c278844435ed","Type":"ContainerStarted","Data":"c4f9efa69db4ee330fdb10c325532668b2864c9840c29b24faa5dc967b1dcfee"} Oct 30 20:05:18.497518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:18.497088 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" event={"ID":"f94b5b35-f3af-48c4-a865-c278844435ed","Type":"ContainerStarted","Data":"bb0417b775dfd886fabc98e992b56eebff166e4eaae52deb8eb319e8ed7a922b"} Oct 30 20:05:18.497518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:18.497116 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" Oct 30 20:05:18.524737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:18.524639 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" podStartSLOduration=1.524625453 podStartE2EDuration="1.524625453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:18.521471652 +0000 UTC m=+2038.216679610" watchObservedRunningTime="2026-10-30 20:05:18.524625453 +0000 UTC m=+2038.219833410" Oct 30 20:05:18.743176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:18.742419 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:19.726104 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:19.726057 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:20.205060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:20.204977 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:05:20.279567 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:20.279525 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh"] Oct 30 20:05:20.516505 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:20.516469 5091 generic.go:356] "Generic (PLEG): container finished" podID="43e01282-cbf5-43e3-9815-32c7d4439800" containerID="51dd32bbba52c2223ad6dbbffa3b462b1068e4d77a4c2b50b458cd5b7a6ed92a" exitCode=0 Oct 30 20:05:20.516697 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:20.516542 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" event={"ID":"43e01282-cbf5-43e3-9815-32c7d4439800","Type":"ContainerDied","Data":"51dd32bbba52c2223ad6dbbffa3b462b1068e4d77a4c2b50b458cd5b7a6ed92a"} Oct 30 20:05:20.516987 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:20.516972 5091 scope.go:122] "RemoveContainer" containerID="51dd32bbba52c2223ad6dbbffa3b462b1068e4d77a4c2b50b458cd5b7a6ed92a" Oct 30 20:05:20.518734 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:20.518706 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:05:20.518842 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:20.518772 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:05:20.518842 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:20.518801 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:05:20.597783 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:20.597741 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:21.535166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:21.532361 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" event={"ID":"43e01282-cbf5-43e3-9815-32c7d4439800","Type":"ContainerStarted","Data":"08cb1dcb463b7df7783bb8e40545b717c07a5b7956f0378b2158ee057d8967ec"} Oct 30 20:05:21.591872 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:21.577084 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:22.227982 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:22.227950 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:05:22.388016 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:22.387980 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:05:22.388288 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:22.388253 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 20:05:22.388396 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:22.388319 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" podUID="43e01282-cbf5-43e3-9815-32c7d4439800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 20:05:22.509589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:22.509550 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:23.387968 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:23.387937 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 20:05:23.388418 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:23.387986 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" podUID="43e01282-cbf5-43e3-9815-32c7d4439800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 20:05:23.458175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:23.455359 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:24.408172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:24.405355 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 20:05:24.408172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:24.405415 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" podUID="43e01282-cbf5-43e3-9815-32c7d4439800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 20:05:24.440287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:24.440187 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:24.614225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:24.614194 5091 scope.go:122] "RemoveContainer" containerID="45676de1b02051f1774b6e4c9f5ebfcd02713d209b4760b6c72f516ecbe623cc" Oct 30 20:05:25.281173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.280439 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" podUID="3d5ac221-47f7-46e1-abd4-500f1513e78d" containerName="payload-processing" containerID="cri-o://429e936c9e7860d43805f089020c786a596c6182f882491aab14e72ce4d823b8" gracePeriod=25 Oct 30 20:05:25.392547 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.392514 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 20:05:25.392806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.392770 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" podUID="43e01282-cbf5-43e3-9815-32c7d4439800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 20:05:25.690645 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.690300 5091 generic.go:356] "Generic (PLEG): container finished" podID="3d5ac221-47f7-46e1-abd4-500f1513e78d" containerID="429e936c9e7860d43805f089020c786a596c6182f882491aab14e72ce4d823b8" exitCode=0 Oct 30 20:05:25.690645 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.690529 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" event={"ID":"3d5ac221-47f7-46e1-abd4-500f1513e78d","Type":"ContainerDied","Data":"429e936c9e7860d43805f089020c786a596c6182f882491aab14e72ce4d823b8"} Oct 30 20:05:25.690645 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.690598 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" event={"ID":"3d5ac221-47f7-46e1-abd4-500f1513e78d","Type":"ContainerDied","Data":"455aa565fbbeaae9ac17da6f27c8fb4d1946dc65fe6bb67d509cf7aa6ccf53c0"} Oct 30 20:05:25.690645 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.690614 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="455aa565fbbeaae9ac17da6f27c8fb4d1946dc65fe6bb67d509cf7aa6ccf53c0" Oct 30 20:05:25.702220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.702196 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:05:25.729473 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.729442 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d5ac221-47f7-46e1-abd4-500f1513e78d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5ac221-47f7-46e1-abd4-500f1513e78d-plugins-config-volume\") pod \"3d5ac221-47f7-46e1-abd4-500f1513e78d\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " Oct 30 20:05:25.729634 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.729489 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d5ac221-47f7-46e1-abd4-500f1513e78d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d5ac221-47f7-46e1-abd4-500f1513e78d-routing-overlay-volume\") pod \"3d5ac221-47f7-46e1-abd4-500f1513e78d\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " Oct 30 20:05:25.729713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.729683 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5ac221-47f7-46e1-abd4-500f1513e78d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d5ac221-47f7-46e1-abd4-500f1513e78d-provider-credentials\") pod \"3d5ac221-47f7-46e1-abd4-500f1513e78d\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " Oct 30 20:05:25.730356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.729817 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5ac221-47f7-46e1-abd4-500f1513e78d-kube-api-access-45qnk\" (UniqueName: \"kubernetes.io/projected/3d5ac221-47f7-46e1-abd4-500f1513e78d-kube-api-access-45qnk\") pod \"3d5ac221-47f7-46e1-abd4-500f1513e78d\" (UID: \"3d5ac221-47f7-46e1-abd4-500f1513e78d\") " Oct 30 20:05:25.730356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.729818 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5ac221-47f7-46e1-abd4-500f1513e78d-routing-overlay-volume" pod "3d5ac221-47f7-46e1-abd4-500f1513e78d" (UID: "3d5ac221-47f7-46e1-abd4-500f1513e78d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:25.730356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.729908 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5ac221-47f7-46e1-abd4-500f1513e78d-plugins-config-volume" pod "3d5ac221-47f7-46e1-abd4-500f1513e78d" (UID: "3d5ac221-47f7-46e1-abd4-500f1513e78d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:25.731232 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.731208 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d5ac221-47f7-46e1-abd4-500f1513e78d-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:25.731232 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.731234 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/3d5ac221-47f7-46e1-abd4-500f1513e78d-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:25.732295 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.732267 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5ac221-47f7-46e1-abd4-500f1513e78d-kube-api-access-45qnk" pod "3d5ac221-47f7-46e1-abd4-500f1513e78d" (UID: "3d5ac221-47f7-46e1-abd4-500f1513e78d"). InnerVolumeSpecName "kube-api-access-45qnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:25.732418 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.732399 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5ac221-47f7-46e1-abd4-500f1513e78d-provider-credentials" pod "3d5ac221-47f7-46e1-abd4-500f1513e78d" (UID: "3d5ac221-47f7-46e1-abd4-500f1513e78d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:25.842044 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.841987 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/3d5ac221-47f7-46e1-abd4-500f1513e78d-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:25.842044 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:25.842021 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45qnk\" (UniqueName: \"kubernetes.io/projected/3d5ac221-47f7-46e1-abd4-500f1513e78d-kube-api-access-45qnk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:26.387731 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:26.387685 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body= Oct 30 20:05:26.387958 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:26.387754 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" podUID="43e01282-cbf5-43e3-9815-32c7d4439800" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" Oct 30 20:05:26.415920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:26.415880 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:26.700415 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:26.700328 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh" Oct 30 20:05:26.727680 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:26.727642 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh"] Oct 30 20:05:26.737945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:26.737917 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4d95kmh"] Oct 30 20:05:27.112943 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:27.112838 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:27.393808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:27.393720 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:05:27.394240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:27.394178 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:05:27.395014 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:27.394986 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:05:28.072170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:28.065818 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:28.444127 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:28.444050 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5ac221-47f7-46e1-abd4-500f1513e78d" path="/var/lib/kubelet/pods/3d5ac221-47f7-46e1-abd4-500f1513e78d/volumes" Oct 30 20:05:28.986585 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:28.986535 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:29.512196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:29.512161 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" Oct 30 20:05:29.991051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:29.990947 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:30.894644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:30.894608 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:31.065057 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.065014 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:31.748766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.748726 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7"] Oct 30 20:05:31.756613 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.756543 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3d5ac221-47f7-46e1-abd4-500f1513e78d" containerName="payload-processing" Oct 30 20:05:31.756859 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.756836 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5ac221-47f7-46e1-abd4-500f1513e78d" containerName="payload-processing" Oct 30 20:05:31.757273 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.757252 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5ac221-47f7-46e1-abd4-500f1513e78d" containerName="payload-processing" Oct 30 20:05:31.766790 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.766762 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7"] Oct 30 20:05:31.766920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.766890 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:31.774089 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.774067 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"agc-cred-4c65d9ab39c0\"" Oct 30 20:05:31.864158 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.864124 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0803362-6d1e-4965-b31d-799329a92c99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:31.864309 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.864282 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbsv7\" (UniqueName: \"kubernetes.io/projected/f0803362-6d1e-4965-b31d-799329a92c99-kube-api-access-fbsv7\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:31.864611 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.864583 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0803362-6d1e-4965-b31d-799329a92c99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:31.864655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.864625 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0803362-6d1e-4965-b31d-799329a92c99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:31.966407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.966373 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0803362-6d1e-4965-b31d-799329a92c99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:31.966805 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.966462 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbsv7\" (UniqueName: \"kubernetes.io/projected/f0803362-6d1e-4965-b31d-799329a92c99-kube-api-access-fbsv7\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:31.966805 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.966690 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0803362-6d1e-4965-b31d-799329a92c99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:31.966805 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.966722 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0803362-6d1e-4965-b31d-799329a92c99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:31.967518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.967488 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0803362-6d1e-4965-b31d-799329a92c99-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:31.967618 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.967601 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0803362-6d1e-4965-b31d-799329a92c99-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:31.969107 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.969084 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0803362-6d1e-4965-b31d-799329a92c99-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:31.981455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:31.981428 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbsv7\" (UniqueName: \"kubernetes.io/projected/f0803362-6d1e-4965-b31d-799329a92c99-kube-api-access-fbsv7\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:32.084861 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:32.084778 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:32.107101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:32.107069 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:32.344015 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:32.343910 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:32.395781 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:05:32.395744 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0803362_6d1e_4965_b31d_799329a92c99.slice/crio-a253313376b4c5ad8db8267cb72e726d505469c9e004ba2e190e7172a3a30e90 WatchSource:0}: Error finding container a253313376b4c5ad8db8267cb72e726d505469c9e004ba2e190e7172a3a30e90: Status 404 returned error can't find the container with id a253313376b4c5ad8db8267cb72e726d505469c9e004ba2e190e7172a3a30e90 Oct 30 20:05:32.396373 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:32.396297 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7"] Oct 30 20:05:32.782488 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:32.782456 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" event={"ID":"f0803362-6d1e-4965-b31d-799329a92c99","Type":"ContainerStarted","Data":"0eb062d04406bcdf766b270894d3e3a52a3cf8051deab04564f2c01773d1bdcd"} Oct 30 20:05:32.782647 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:32.782517 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" event={"ID":"f0803362-6d1e-4965-b31d-799329a92c99","Type":"ContainerStarted","Data":"a253313376b4c5ad8db8267cb72e726d505469c9e004ba2e190e7172a3a30e90"} Oct 30 20:05:32.783533 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:32.783252 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:32.812827 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:32.812768 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" podStartSLOduration=1.812754175 podStartE2EDuration="1.812754175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:05:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:05:32.808724754 +0000 UTC m=+2052.503932709" watchObservedRunningTime="2026-10-30 20:05:32.812754175 +0000 UTC m=+2052.507962168" Oct 30 20:05:33.317608 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:33.309068 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:34.279174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:34.278217 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:35.166169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:35.164802 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:36.130167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:36.127242 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:37.070955 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:37.070909 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:37.924168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:37.922680 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:38.744611 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:38.744559 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:39.149179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:39.146794 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:39.960225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:39.960126 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:40.109440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:40.109395 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:40.281257 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:40.281211 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:41.007170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:41.006398 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:41.831611 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:41.831516 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:42.632452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:42.632413 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:43.033237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:43.033204 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h"] Oct 30 20:05:43.038074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:43.038043 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8520dc94-mg76h"] Oct 30 20:05:43.578359 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:43.578320 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:43.797220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:43.797190 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:05:43.861669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:43.861583 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh"] Oct 30 20:05:44.369858 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:44.369767 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:44.445138 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:44.445102 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7efa7f1b-69d0-4bac-a180-89e102205b24" path="/var/lib/kubelet/pods/7efa7f1b-69d0-4bac-a180-89e102205b24/volumes" Oct 30 20:05:45.308347 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:45.308290 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:45.978581 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:05:45.978540 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:05:46.259950 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:46.259881 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:47.212175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:47.206174 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:48.038170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:48.036789 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:48.862773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:48.862732 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" podUID="ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea" containerName="payload-processing" containerID="cri-o://4a432706dd586ae865512848302fcd70174a787feb6ea253b261017cc24f9509" gracePeriod=25 Oct 30 20:05:48.987933 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:48.987824 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:49.162481 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:49.162366 5091 generic.go:356] "Generic (PLEG): container finished" podID="ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea" containerID="4a432706dd586ae865512848302fcd70174a787feb6ea253b261017cc24f9509" exitCode=0 Oct 30 20:05:49.162912 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:49.162882 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" event={"ID":"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea","Type":"ContainerDied","Data":"4a432706dd586ae865512848302fcd70174a787feb6ea253b261017cc24f9509"} Oct 30 20:05:49.235757 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:49.235733 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:05:49.413888 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:49.412947 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-provider-credentials\") pod \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " Oct 30 20:05:49.413888 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:49.413103 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-routing-overlay-volume\") pod \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " Oct 30 20:05:49.413888 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:49.413274 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-kube-api-access-tjgps\" (UniqueName: \"kubernetes.io/projected/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-kube-api-access-tjgps\") pod \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " Oct 30 20:05:49.413888 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:49.413378 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-plugins-config-volume\") pod \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\" (UID: \"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea\") " Oct 30 20:05:49.414278 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:49.413835 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-routing-overlay-volume" pod "ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea" (UID: "ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:49.414278 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:49.414178 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-plugins-config-volume" pod "ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea" (UID: "ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:05:49.414421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:49.414384 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:49.414421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:49.414413 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:49.416592 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:49.416566 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-kube-api-access-tjgps" pod "ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea" (UID: "ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea"). InnerVolumeSpecName "kube-api-access-tjgps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:49.416865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:49.416835 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-provider-credentials" pod "ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea" (UID: "ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:05:49.516021 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:49.515984 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:49.516021 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:49.516016 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjgps\" (UniqueName: \"kubernetes.io/projected/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea-kube-api-access-tjgps\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:05:50.105640 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:50.104711 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:50.173993 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:50.173962 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" Oct 30 20:05:50.174232 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:50.174002 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh" event={"ID":"ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea","Type":"ContainerDied","Data":"e359cd96e52d8d81674f678b5296d8a85a379338866976f539418d48a0e2a838"} Oct 30 20:05:50.174232 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:50.174063 5091 scope.go:122] "RemoveContainer" containerID="4a432706dd586ae865512848302fcd70174a787feb6ea253b261017cc24f9509" Oct 30 20:05:50.205192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:50.205127 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh"] Oct 30 20:05:50.209604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:50.209583 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dcphrsh"] Oct 30 20:05:50.443433 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:50.443357 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea" path="/var/lib/kubelet/pods/ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea/volumes" Oct 30 20:05:50.967188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:50.967126 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:51.898535 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:51.897283 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:52.901169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:52.900491 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:53.033607 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:53.033575 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4"] Oct 30 20:05:53.037720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:53.037696 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70882eb3-5dtf4"] Oct 30 20:05:53.907315 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:53.907278 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:54.444637 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:54.444592 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb0ae25-c4ad-4001-bbd9-494d7dc610ba" path="/var/lib/kubelet/pods/4bb0ae25-c4ad-4001-bbd9-494d7dc610ba/volumes" Oct 30 20:05:54.909258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:54.909219 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:55.906179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:55.904066 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:56.840167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:56.839847 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:57.693392 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:57.693355 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:58.684235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:58.684186 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:05:59.528919 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:05:59.528876 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:00.519205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:00.519162 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:01.628691 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:01.628658 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:02.761936 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:02.761895 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:02.976686 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:02.976650 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:03.714008 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:03.713951 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:03.879532 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:03.879493 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:04.222970 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.222934 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:04.587411 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.587376 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5"] Oct 30 20:06:04.588929 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.588901 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea" containerName="payload-processing" Oct 30 20:06:04.590298 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.590279 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea" containerName="payload-processing" Oct 30 20:06:04.590712 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.590692 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec8b5abf-46bc-4f75-b8a5-0f3d69b55aea" containerName="payload-processing" Oct 30 20:06:04.596984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.596967 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5"] Oct 30 20:06:04.597155 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.597099 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:04.608360 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.608332 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"agc-cred-d2f3b71bc52a\"" Oct 30 20:06:04.686713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.686675 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxs2s\" (UniqueName: \"kubernetes.io/projected/477ec777-3941-4e8b-a088-0877d7e1f5f3-kube-api-access-hxs2s\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:04.686920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.686793 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/477ec777-3941-4e8b-a088-0877d7e1f5f3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:04.686920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.686863 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477ec777-3941-4e8b-a088-0877d7e1f5f3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:04.687048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.687008 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/477ec777-3941-4e8b-a088-0877d7e1f5f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:04.788425 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.788387 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477ec777-3941-4e8b-a088-0877d7e1f5f3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:04.788643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.788504 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/477ec777-3941-4e8b-a088-0877d7e1f5f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:04.788729 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.788644 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxs2s\" (UniqueName: \"kubernetes.io/projected/477ec777-3941-4e8b-a088-0877d7e1f5f3-kube-api-access-hxs2s\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:04.788729 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.788699 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/477ec777-3941-4e8b-a088-0877d7e1f5f3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:04.789545 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.789514 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/477ec777-3941-4e8b-a088-0877d7e1f5f3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:04.790224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.790196 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477ec777-3941-4e8b-a088-0877d7e1f5f3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:04.792140 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.792095 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/477ec777-3941-4e8b-a088-0877d7e1f5f3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:04.806539 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.806511 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxs2s\" (UniqueName: \"kubernetes.io/projected/477ec777-3941-4e8b-a088-0877d7e1f5f3-kube-api-access-hxs2s\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:04.895908 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.895801 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:04.914945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:04.914912 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:05.604230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:05.593846 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:05.695028 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:05.694990 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5"] Oct 30 20:06:05.702132 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:06:05.702103 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod477ec777_3941_4e8b_a088_0877d7e1f5f3.slice/crio-29649b1263ffc7cab71370e2971e2adab3d76614c7d06b5e13d4d044d42ec0fa WatchSource:0}: Error finding container 29649b1263ffc7cab71370e2971e2adab3d76614c7d06b5e13d4d044d42ec0fa: Status 404 returned error can't find the container with id 29649b1263ffc7cab71370e2971e2adab3d76614c7d06b5e13d4d044d42ec0fa Oct 30 20:06:05.780210 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:05.779163 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:05.965326 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:05.962111 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:06.521358 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:06.521315 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" event={"ID":"477ec777-3941-4e8b-a088-0877d7e1f5f3","Type":"ContainerStarted","Data":"2aa1a4c3f4e45b15e75a291fae9c674c21aff831716a8b99da4fac264899d056"} Oct 30 20:06:06.521565 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:06.521493 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" event={"ID":"477ec777-3941-4e8b-a088-0877d7e1f5f3","Type":"ContainerStarted","Data":"29649b1263ffc7cab71370e2971e2adab3d76614c7d06b5e13d4d044d42ec0fa"} Oct 30 20:06:06.521565 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:06.521534 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:06.580635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:06.578431 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" podStartSLOduration=2.57840852 podStartE2EDuration="2.57840852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:06.543977128 +0000 UTC m=+2086.239185085" watchObservedRunningTime="2026-10-30 20:06:06.57840852 +0000 UTC m=+2086.273616479" Oct 30 20:06:07.134209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:07.134174 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:07.691464 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:07.688956 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:08.277756 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.277718 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:08.291237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.291194 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89"] Oct 30 20:06:08.334228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.333427 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:08.343195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.340736 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89"] Oct 30 20:06:08.432214 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.432175 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9294290-0590-4829-b4d5-9def2109de6b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:08.432444 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.432418 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9294290-0590-4829-b4d5-9def2109de6b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:08.432647 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.432626 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9294290-0590-4829-b4d5-9def2109de6b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:08.432719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.432669 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkmks\" (UniqueName: \"kubernetes.io/projected/b9294290-0590-4829-b4d5-9def2109de6b-kube-api-access-dkmks\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:08.533764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.533676 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkmks\" (UniqueName: \"kubernetes.io/projected/b9294290-0590-4829-b4d5-9def2109de6b-kube-api-access-dkmks\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:08.533941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.533801 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9294290-0590-4829-b4d5-9def2109de6b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:08.534022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.533974 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9294290-0590-4829-b4d5-9def2109de6b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:08.534190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.534135 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9294290-0590-4829-b4d5-9def2109de6b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:08.534629 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.534604 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9294290-0590-4829-b4d5-9def2109de6b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:08.534856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.534829 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9294290-0590-4829-b4d5-9def2109de6b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:08.536917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.536895 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9294290-0590-4829-b4d5-9def2109de6b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:08.572501 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.572471 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkmks\" (UniqueName: \"kubernetes.io/projected/b9294290-0590-4829-b4d5-9def2109de6b-kube-api-access-dkmks\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:08.700651 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:08.700613 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:09.019622 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:06:09.019592 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9294290_0590_4829_b4d5_9def2109de6b.slice/crio-00820084343ea4fc3179639898cd0c291a293e76737eb38c42f1d314231ef818 WatchSource:0}: Error finding container 00820084343ea4fc3179639898cd0c291a293e76737eb38c42f1d314231ef818: Status 404 returned error can't find the container with id 00820084343ea4fc3179639898cd0c291a293e76737eb38c42f1d314231ef818 Oct 30 20:06:09.022868 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:09.022839 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89"] Oct 30 20:06:09.343814 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:09.343724 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:09.552062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:09.552011 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" event={"ID":"b9294290-0590-4829-b4d5-9def2109de6b","Type":"ContainerStarted","Data":"e7c363af74ec467bb4620cb4290cc050eab5b870e4f43ce57abb8d6cada02a59"} Oct 30 20:06:09.552561 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:09.552322 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:09.552561 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:09.552368 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" event={"ID":"b9294290-0590-4829-b4d5-9def2109de6b","Type":"ContainerStarted","Data":"00820084343ea4fc3179639898cd0c291a293e76737eb38c42f1d314231ef818"} Oct 30 20:06:09.584936 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:09.584887 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" podStartSLOduration=1.584873833 podStartE2EDuration="1.584873833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:09.578602915 +0000 UTC m=+2089.273810877" watchObservedRunningTime="2026-10-30 20:06:09.584873833 +0000 UTC m=+2089.280081790" Oct 30 20:06:10.234283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:10.233920 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:11.137306 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:11.137272 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:11.318170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:11.317709 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:11.720857 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:11.720825 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5"] Oct 30 20:06:11.874168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:11.867739 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:11.874168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:11.870319 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:12.065639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:12.065591 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:12.364672 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:12.364565 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:12.526556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:12.526513 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:12.924757 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:12.924717 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw"] Oct 30 20:06:12.929737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:12.929717 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" Oct 30 20:06:12.939980 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:12.939951 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw"] Oct 30 20:06:12.981257 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:12.980834 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c7bc95-72c8-4b53-9a9e-c0f069783697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ea0d3b9-xw6bw\" (UID: \"c0c7bc95-72c8-4b53-9a9e-c0f069783697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" Oct 30 20:06:12.981257 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:12.980897 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zvpz\" (UniqueName: \"kubernetes.io/projected/c0c7bc95-72c8-4b53-9a9e-c0f069783697-kube-api-access-7zvpz\") pod \"maas-api-revoke-sub-9ea0d3b9-xw6bw\" (UID: \"c0c7bc95-72c8-4b53-9a9e-c0f069783697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" Oct 30 20:06:12.981257 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:12.981055 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c7bc95-72c8-4b53-9a9e-c0f069783697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ea0d3b9-xw6bw\" (UID: \"c0c7bc95-72c8-4b53-9a9e-c0f069783697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" Oct 30 20:06:13.009568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:13.007960 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:06:13.009568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:13.008084 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:06:13.082738 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:13.082702 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c7bc95-72c8-4b53-9a9e-c0f069783697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ea0d3b9-xw6bw\" (UID: \"c0c7bc95-72c8-4b53-9a9e-c0f069783697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" Oct 30 20:06:13.082942 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:13.082923 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c7bc95-72c8-4b53-9a9e-c0f069783697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ea0d3b9-xw6bw\" (UID: \"c0c7bc95-72c8-4b53-9a9e-c0f069783697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" Oct 30 20:06:13.082990 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:13.082968 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zvpz\" (UniqueName: \"kubernetes.io/projected/c0c7bc95-72c8-4b53-9a9e-c0f069783697-kube-api-access-7zvpz\") pod \"maas-api-revoke-sub-9ea0d3b9-xw6bw\" (UID: \"c0c7bc95-72c8-4b53-9a9e-c0f069783697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" Oct 30 20:06:13.083802 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:13.083774 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c7bc95-72c8-4b53-9a9e-c0f069783697-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ea0d3b9-xw6bw\" (UID: \"c0c7bc95-72c8-4b53-9a9e-c0f069783697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" Oct 30 20:06:13.085554 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:13.085525 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c7bc95-72c8-4b53-9a9e-c0f069783697-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ea0d3b9-xw6bw\" (UID: \"c0c7bc95-72c8-4b53-9a9e-c0f069783697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" Oct 30 20:06:13.099278 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:13.099246 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zvpz\" (UniqueName: \"kubernetes.io/projected/c0c7bc95-72c8-4b53-9a9e-c0f069783697-kube-api-access-7zvpz\") pod \"maas-api-revoke-sub-9ea0d3b9-xw6bw\" (UID: \"c0c7bc95-72c8-4b53-9a9e-c0f069783697\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" Oct 30 20:06:13.117253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:13.117219 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:13.256074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:13.256038 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" Oct 30 20:06:13.331168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:13.326550 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:13.551133 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:13.551036 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:13.681740 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:06:13.681702 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0c7bc95_72c8_4b53_9a9e_c0f069783697.slice/crio-799a1fc90621e25d0b7005b33fc612e50b49eaaa6bd055c4cb6e9bc98fa9bdad WatchSource:0}: Error finding container 799a1fc90621e25d0b7005b33fc612e50b49eaaa6bd055c4cb6e9bc98fa9bdad: Status 404 returned error can't find the container with id 799a1fc90621e25d0b7005b33fc612e50b49eaaa6bd055c4cb6e9bc98fa9bdad Oct 30 20:06:13.684801 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:13.684772 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw"] Oct 30 20:06:13.801673 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:13.797222 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:13.987744 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:13.987716 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:14.178586 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:14.178559 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:14.279786 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:14.279754 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 20:06:14.279932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:14.279809 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 20:06:14.554172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:14.544791 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:14.750776 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:14.750727 5091 generic.go:356] "Generic (PLEG): container finished" podID="5109436f-1c85-4413-a7e7-f62dac25203f" containerID="0f4009c3de1f21e6df9055dbab35fb4077d58766c61711a5fbce24a99dcb56e7" exitCode=0 Oct 30 20:06:14.750987 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:14.750833 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" event={"ID":"5109436f-1c85-4413-a7e7-f62dac25203f","Type":"ContainerDied","Data":"0f4009c3de1f21e6df9055dbab35fb4077d58766c61711a5fbce24a99dcb56e7"} Oct 30 20:06:14.751391 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:14.751367 5091 scope.go:122] "RemoveContainer" containerID="0f4009c3de1f21e6df9055dbab35fb4077d58766c61711a5fbce24a99dcb56e7" Oct 30 20:06:14.754031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:14.753997 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:06:14.754208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:14.754089 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:06:14.754208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:14.754157 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:06:14.757194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:14.756802 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" event={"ID":"c0c7bc95-72c8-4b53-9a9e-c0f069783697","Type":"ContainerStarted","Data":"8e58312d8d89251ef24821fedc5273fbae2bc1c830ade9e479823318e05e054e"} Oct 30 20:06:14.757194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:14.756924 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" event={"ID":"c0c7bc95-72c8-4b53-9a9e-c0f069783697","Type":"ContainerStarted","Data":"799a1fc90621e25d0b7005b33fc612e50b49eaaa6bd055c4cb6e9bc98fa9bdad"} Oct 30 20:06:14.768432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:14.768264 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:14.836174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:14.836113 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" podStartSLOduration=2.836100189 podStartE2EDuration="2.836100189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:14.83313482 +0000 UTC m=+2094.528342779" watchObservedRunningTime="2026-10-30 20:06:14.836100189 +0000 UTC m=+2094.531308146" Oct 30 20:06:15.785618 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:15.785537 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" event={"ID":"5109436f-1c85-4413-a7e7-f62dac25203f","Type":"ContainerStarted","Data":"c5db5edb55b9cbbd2fed3e99b4dd54eebbbc50d738d2bf0f882067ff447533c4"} Oct 30 20:06:15.789195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:15.789137 5091 generic.go:356] "Generic (PLEG): container finished" podID="c0c7bc95-72c8-4b53-9a9e-c0f069783697" containerID="8e58312d8d89251ef24821fedc5273fbae2bc1c830ade9e479823318e05e054e" exitCode=0 Oct 30 20:06:15.789347 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:15.789313 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" event={"ID":"c0c7bc95-72c8-4b53-9a9e-c0f069783697","Type":"ContainerDied","Data":"8e58312d8d89251ef24821fedc5273fbae2bc1c830ade9e479823318e05e054e"} Oct 30 20:06:15.946615 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:15.946573 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:16.021776 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:16.021737 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:06:16.244746 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.244660 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:06:16.244933 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.244905 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 20:06:16.244979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.244960 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 20:06:16.324804 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.324763 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f"] Oct 30 20:06:16.328840 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.328818 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" Oct 30 20:06:16.394698 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.394019 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f"] Oct 30 20:06:16.413494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.413456 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89cb5bcb-k6s5f\" (UID: \"e90e48ff-e0d8-4495-b447-ccdd7fb3935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" Oct 30 20:06:16.413666 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.413508 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89cb5bcb-k6s5f\" (UID: \"e90e48ff-e0d8-4495-b447-ccdd7fb3935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" Oct 30 20:06:16.413732 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.413707 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwv5v\" (UniqueName: \"kubernetes.io/projected/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-kube-api-access-kwv5v\") pod \"maas-api-revoke-sub-89cb5bcb-k6s5f\" (UID: \"e90e48ff-e0d8-4495-b447-ccdd7fb3935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" Oct 30 20:06:16.515689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.515653 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89cb5bcb-k6s5f\" (UID: \"e90e48ff-e0d8-4495-b447-ccdd7fb3935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" Oct 30 20:06:16.515905 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.515719 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89cb5bcb-k6s5f\" (UID: \"e90e48ff-e0d8-4495-b447-ccdd7fb3935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" Oct 30 20:06:16.515905 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.515806 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwv5v\" (UniqueName: \"kubernetes.io/projected/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-kube-api-access-kwv5v\") pod \"maas-api-revoke-sub-89cb5bcb-k6s5f\" (UID: \"e90e48ff-e0d8-4495-b447-ccdd7fb3935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" Oct 30 20:06:16.516462 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.516436 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89cb5bcb-k6s5f\" (UID: \"e90e48ff-e0d8-4495-b447-ccdd7fb3935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" Oct 30 20:06:16.518250 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.518228 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89cb5bcb-k6s5f\" (UID: \"e90e48ff-e0d8-4495-b447-ccdd7fb3935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" Oct 30 20:06:16.577169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.558552 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwv5v\" (UniqueName: \"kubernetes.io/projected/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-kube-api-access-kwv5v\") pod \"maas-api-revoke-sub-89cb5bcb-k6s5f\" (UID: \"e90e48ff-e0d8-4495-b447-ccdd7fb3935c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" Oct 30 20:06:16.686997 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.686958 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" Oct 30 20:06:16.722103 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.722037 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" podUID="477ec777-3941-4e8b-a088-0877d7e1f5f3" containerName="payload-processing" containerID="cri-o://2aa1a4c3f4e45b15e75a291fae9c674c21aff831716a8b99da4fac264899d056" gracePeriod=25 Oct 30 20:06:17.005754 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:16.948646 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:17.035466 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:06:17.035433 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90e48ff_e0d8_4495_b447_ccdd7fb3935c.slice/crio-3203feb834800aab2459957835a0ae2ab1ea01f60b724f66c102742c6e417001 WatchSource:0}: Error finding container 3203feb834800aab2459957835a0ae2ab1ea01f60b724f66c102742c6e417001: Status 404 returned error can't find the container with id 3203feb834800aab2459957835a0ae2ab1ea01f60b724f66c102742c6e417001 Oct 30 20:06:17.042686 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.042391 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f"] Oct 30 20:06:17.245036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.245006 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 20:06:17.249275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.245057 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 20:06:17.304750 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.304689 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" Oct 30 20:06:17.308323 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.308300 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:17.315285 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.315213 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0c7bc95-72c8-4b53-9a9e-c0f069783697-kube-api-access-7zvpz\" (UniqueName: \"kubernetes.io/projected/c0c7bc95-72c8-4b53-9a9e-c0f069783697-kube-api-access-7zvpz\") pod \"c0c7bc95-72c8-4b53-9a9e-c0f069783697\" (UID: \"c0c7bc95-72c8-4b53-9a9e-c0f069783697\") " Oct 30 20:06:17.317719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.315332 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/477ec777-3941-4e8b-a088-0877d7e1f5f3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/477ec777-3941-4e8b-a088-0877d7e1f5f3-provider-credentials\") pod \"477ec777-3941-4e8b-a088-0877d7e1f5f3\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " Oct 30 20:06:17.317719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.315401 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0c7bc95-72c8-4b53-9a9e-c0f069783697-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c7bc95-72c8-4b53-9a9e-c0f069783697-maas-api-openshift-ca\") pod \"c0c7bc95-72c8-4b53-9a9e-c0f069783697\" (UID: \"c0c7bc95-72c8-4b53-9a9e-c0f069783697\") " Oct 30 20:06:17.317719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.315475 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/477ec777-3941-4e8b-a088-0877d7e1f5f3-kube-api-access-hxs2s\" (UniqueName: \"kubernetes.io/projected/477ec777-3941-4e8b-a088-0877d7e1f5f3-kube-api-access-hxs2s\") pod \"477ec777-3941-4e8b-a088-0877d7e1f5f3\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " Oct 30 20:06:17.317719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.315956 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0c7bc95-72c8-4b53-9a9e-c0f069783697-maas-api-openshift-ca" pod "c0c7bc95-72c8-4b53-9a9e-c0f069783697" (UID: "c0c7bc95-72c8-4b53-9a9e-c0f069783697"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:17.317719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.316010 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0c7bc95-72c8-4b53-9a9e-c0f069783697-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c7bc95-72c8-4b53-9a9e-c0f069783697-maas-api-xks-ca\") pod \"c0c7bc95-72c8-4b53-9a9e-c0f069783697\" (UID: \"c0c7bc95-72c8-4b53-9a9e-c0f069783697\") " Oct 30 20:06:17.317719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.316108 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/477ec777-3941-4e8b-a088-0877d7e1f5f3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/477ec777-3941-4e8b-a088-0877d7e1f5f3-routing-overlay-volume\") pod \"477ec777-3941-4e8b-a088-0877d7e1f5f3\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " Oct 30 20:06:17.317719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.316200 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/477ec777-3941-4e8b-a088-0877d7e1f5f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477ec777-3941-4e8b-a088-0877d7e1f5f3-plugins-config-volume\") pod \"477ec777-3941-4e8b-a088-0877d7e1f5f3\" (UID: \"477ec777-3941-4e8b-a088-0877d7e1f5f3\") " Oct 30 20:06:17.317719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.316888 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0c7bc95-72c8-4b53-9a9e-c0f069783697-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.317719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.317184 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477ec777-3941-4e8b-a088-0877d7e1f5f3-routing-overlay-volume" pod "477ec777-3941-4e8b-a088-0877d7e1f5f3" (UID: "477ec777-3941-4e8b-a088-0877d7e1f5f3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:17.317719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.317302 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/477ec777-3941-4e8b-a088-0877d7e1f5f3-plugins-config-volume" pod "477ec777-3941-4e8b-a088-0877d7e1f5f3" (UID: "477ec777-3941-4e8b-a088-0877d7e1f5f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:17.319356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.318378 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477ec777-3941-4e8b-a088-0877d7e1f5f3-provider-credentials" pod "477ec777-3941-4e8b-a088-0877d7e1f5f3" (UID: "477ec777-3941-4e8b-a088-0877d7e1f5f3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:17.319356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.318758 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0c7bc95-72c8-4b53-9a9e-c0f069783697-maas-api-xks-ca" pod "c0c7bc95-72c8-4b53-9a9e-c0f069783697" (UID: "c0c7bc95-72c8-4b53-9a9e-c0f069783697"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:17.319675 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.319654 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0c7bc95-72c8-4b53-9a9e-c0f069783697-kube-api-access-7zvpz" pod "c0c7bc95-72c8-4b53-9a9e-c0f069783697" (UID: "c0c7bc95-72c8-4b53-9a9e-c0f069783697"). InnerVolumeSpecName "kube-api-access-7zvpz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:17.320048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.320026 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/477ec777-3941-4e8b-a088-0877d7e1f5f3-kube-api-access-hxs2s" pod "477ec777-3941-4e8b-a088-0877d7e1f5f3" (UID: "477ec777-3941-4e8b-a088-0877d7e1f5f3"). InnerVolumeSpecName "kube-api-access-hxs2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:17.417942 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.417911 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zvpz\" (UniqueName: \"kubernetes.io/projected/c0c7bc95-72c8-4b53-9a9e-c0f069783697-kube-api-access-7zvpz\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.417942 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.417939 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/477ec777-3941-4e8b-a088-0877d7e1f5f3-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.418132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.417949 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxs2s\" (UniqueName: \"kubernetes.io/projected/477ec777-3941-4e8b-a088-0877d7e1f5f3-kube-api-access-hxs2s\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.418132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.417960 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0c7bc95-72c8-4b53-9a9e-c0f069783697-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.418132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.417970 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/477ec777-3941-4e8b-a088-0877d7e1f5f3-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.418132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.417979 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/477ec777-3941-4e8b-a088-0877d7e1f5f3-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:17.825275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.825240 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" event={"ID":"e90e48ff-e0d8-4495-b447-ccdd7fb3935c","Type":"ContainerStarted","Data":"3d5f60e466b766deeb8390068f3ec4c2a81e27ede581672ab3c4a6a60a2b0774"} Oct 30 20:06:17.825564 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.825538 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" event={"ID":"e90e48ff-e0d8-4495-b447-ccdd7fb3935c","Type":"ContainerStarted","Data":"3203feb834800aab2459957835a0ae2ab1ea01f60b724f66c102742c6e417001"} Oct 30 20:06:17.826636 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.826610 5091 generic.go:356] "Generic (PLEG): container finished" podID="477ec777-3941-4e8b-a088-0877d7e1f5f3" containerID="2aa1a4c3f4e45b15e75a291fae9c674c21aff831716a8b99da4fac264899d056" exitCode=0 Oct 30 20:06:17.826740 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.826653 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" Oct 30 20:06:17.826740 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.826732 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" event={"ID":"477ec777-3941-4e8b-a088-0877d7e1f5f3","Type":"ContainerDied","Data":"2aa1a4c3f4e45b15e75a291fae9c674c21aff831716a8b99da4fac264899d056"} Oct 30 20:06:17.826867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.826761 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5" event={"ID":"477ec777-3941-4e8b-a088-0877d7e1f5f3","Type":"ContainerDied","Data":"29649b1263ffc7cab71370e2971e2adab3d76614c7d06b5e13d4d044d42ec0fa"} Oct 30 20:06:17.826867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.826782 5091 scope.go:122] "RemoveContainer" containerID="2aa1a4c3f4e45b15e75a291fae9c674c21aff831716a8b99da4fac264899d056" Oct 30 20:06:17.828416 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.828398 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" Oct 30 20:06:17.828512 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.828431 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw" event={"ID":"c0c7bc95-72c8-4b53-9a9e-c0f069783697","Type":"ContainerDied","Data":"799a1fc90621e25d0b7005b33fc612e50b49eaaa6bd055c4cb6e9bc98fa9bdad"} Oct 30 20:06:17.828512 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.828475 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="799a1fc90621e25d0b7005b33fc612e50b49eaaa6bd055c4cb6e9bc98fa9bdad" Oct 30 20:06:17.868178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.860014 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" podStartSLOduration=1.859995069 podStartE2EDuration="1.859995069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:17.857550168 +0000 UTC m=+2097.552758126" watchObservedRunningTime="2026-10-30 20:06:17.859995069 +0000 UTC m=+2097.555203028" Oct 30 20:06:17.887349 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.887094 5091 scope.go:122] "RemoveContainer" containerID="2aa1a4c3f4e45b15e75a291fae9c674c21aff831716a8b99da4fac264899d056" Oct 30 20:06:17.888622 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:17.888585 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aa1a4c3f4e45b15e75a291fae9c674c21aff831716a8b99da4fac264899d056\": container with ID starting with 2aa1a4c3f4e45b15e75a291fae9c674c21aff831716a8b99da4fac264899d056 not found: ID does not exist" containerID="2aa1a4c3f4e45b15e75a291fae9c674c21aff831716a8b99da4fac264899d056" Oct 30 20:06:17.888702 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.888634 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aa1a4c3f4e45b15e75a291fae9c674c21aff831716a8b99da4fac264899d056"} err="failed to get container status \"2aa1a4c3f4e45b15e75a291fae9c674c21aff831716a8b99da4fac264899d056\": rpc error: code = NotFound desc = could not find container \"2aa1a4c3f4e45b15e75a291fae9c674c21aff831716a8b99da4fac264899d056\": container with ID starting with 2aa1a4c3f4e45b15e75a291fae9c674c21aff831716a8b99da4fac264899d056 not found: ID does not exist" Oct 30 20:06:17.906820 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.906764 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5"] Oct 30 20:06:17.919029 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:17.919000 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-75d4fbsv5"] Oct 30 20:06:18.012257 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:18.012226 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:18.244932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:18.244851 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 20:06:18.244932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:18.244905 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 20:06:18.444867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:18.444838 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="477ec777-3941-4e8b-a088-0877d7e1f5f3" path="/var/lib/kubelet/pods/477ec777-3941-4e8b-a088-0877d7e1f5f3/volumes" Oct 30 20:06:18.840798 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:18.840758 5091 generic.go:356] "Generic (PLEG): container finished" podID="e90e48ff-e0d8-4495-b447-ccdd7fb3935c" containerID="3d5f60e466b766deeb8390068f3ec4c2a81e27ede581672ab3c4a6a60a2b0774" exitCode=0 Oct 30 20:06:18.840981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:18.840819 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" event={"ID":"e90e48ff-e0d8-4495-b447-ccdd7fb3935c","Type":"ContainerDied","Data":"3d5f60e466b766deeb8390068f3ec4c2a81e27ede581672ab3c4a6a60a2b0774"} Oct 30 20:06:18.886524 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:18.886488 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:19.084882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:19.084843 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:19.266166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:19.254853 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 20:06:19.266166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:19.254922 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 20:06:19.925451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:19.924856 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:20.214173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.214116 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" Oct 30 20:06:20.244432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.244409 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 20:06:20.244553 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.244453 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 20:06:20.337176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.337128 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-kube-api-access-kwv5v\" (UniqueName: \"kubernetes.io/projected/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-kube-api-access-kwv5v\") pod \"e90e48ff-e0d8-4495-b447-ccdd7fb3935c\" (UID: \"e90e48ff-e0d8-4495-b447-ccdd7fb3935c\") " Oct 30 20:06:20.338084 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.338052 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-maas-api-xks-ca\") pod \"e90e48ff-e0d8-4495-b447-ccdd7fb3935c\" (UID: \"e90e48ff-e0d8-4495-b447-ccdd7fb3935c\") " Oct 30 20:06:20.338233 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.338095 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-maas-api-openshift-ca\") pod \"e90e48ff-e0d8-4495-b447-ccdd7fb3935c\" (UID: \"e90e48ff-e0d8-4495-b447-ccdd7fb3935c\") " Oct 30 20:06:20.338810 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.338780 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-maas-api-openshift-ca" pod "e90e48ff-e0d8-4495-b447-ccdd7fb3935c" (UID: "e90e48ff-e0d8-4495-b447-ccdd7fb3935c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:20.339678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.339610 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-kube-api-access-kwv5v" pod "e90e48ff-e0d8-4495-b447-ccdd7fb3935c" (UID: "e90e48ff-e0d8-4495-b447-ccdd7fb3935c"). InnerVolumeSpecName "kube-api-access-kwv5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:20.340692 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.340669 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-maas-api-xks-ca" pod "e90e48ff-e0d8-4495-b447-ccdd7fb3935c" (UID: "e90e48ff-e0d8-4495-b447-ccdd7fb3935c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:20.440072 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.440000 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwv5v\" (UniqueName: \"kubernetes.io/projected/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-kube-api-access-kwv5v\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:20.440072 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.440030 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:20.440072 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.440049 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90e48ff-e0d8-4495-b447-ccdd7fb3935c-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:20.585661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.582443 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:20.696515 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.692588 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm"] Oct 30 20:06:20.895879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.895660 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" event={"ID":"e90e48ff-e0d8-4495-b447-ccdd7fb3935c","Type":"ContainerDied","Data":"3203feb834800aab2459957835a0ae2ab1ea01f60b724f66c102742c6e417001"} Oct 30 20:06:20.895879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.895719 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3203feb834800aab2459957835a0ae2ab1ea01f60b724f66c102742c6e417001" Oct 30 20:06:20.895879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.895787 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:20.895879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:20.895799 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f" Oct 30 20:06:21.055171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:21.045391 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:21.229200 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:21.228480 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:21.245182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:21.244567 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" start-of-body= Oct 30 20:06:21.245182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:21.244635 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.31:15021/healthz/ready\": dial tcp 10.132.1.31:15021: connect: connection refused" Oct 30 20:06:21.298112 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:21.298075 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 20:06:21.299732 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:21.299704 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 20:06:21.553660 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:21.553625 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:22.109817 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:22.106837 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:22.249651 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:22.249623 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:06:22.250129 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:22.250105 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:06:22.250972 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:22.250949 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:06:22.364242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:22.363921 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:23.034195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:23.034161 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:24.004196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:24.004141 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:24.155349 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:24.155315 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4"] Oct 30 20:06:24.212417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:24.212382 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:24.388425 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:24.388331 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:24.825018 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:24.824985 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:25.227050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:25.227015 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:25.693679 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:25.693630 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" podUID="95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc" containerName="payload-processing" containerID="cri-o://3260cbee6ac85fdec3db92654ebe711e720f8f47fb5bed6f28f5500de9576f45" gracePeriod=25 Oct 30 20:06:25.824382 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:25.824350 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:25.944562 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:25.944471 5091 generic.go:356] "Generic (PLEG): container finished" podID="95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc" containerID="3260cbee6ac85fdec3db92654ebe711e720f8f47fb5bed6f28f5500de9576f45" exitCode=0 Oct 30 20:06:25.944562 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:25.944546 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" event={"ID":"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc","Type":"ContainerDied","Data":"3260cbee6ac85fdec3db92654ebe711e720f8f47fb5bed6f28f5500de9576f45"} Oct 30 20:06:25.988878 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:25.988834 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:26.008985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.008962 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:06:26.050932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.050902 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-provider-credentials\") pod \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " Oct 30 20:06:26.051271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.051241 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-routing-overlay-volume\") pod \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " Oct 30 20:06:26.051629 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.051606 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-routing-overlay-volume" pod "95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc" (UID: "95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:26.051734 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.051685 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-plugins-config-volume\") pod \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " Oct 30 20:06:26.051824 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.051765 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-kube-api-access-ch8kb\" (UniqueName: \"kubernetes.io/projected/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-kube-api-access-ch8kb\") pod \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\" (UID: \"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc\") " Oct 30 20:06:26.052113 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.052086 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-plugins-config-volume" pod "95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc" (UID: "95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:26.054016 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.053988 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:26.054186 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.054021 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:26.056056 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.056029 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-provider-credentials" pod "95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc" (UID: "95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:26.056914 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.056890 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv"] Oct 30 20:06:26.060670 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.060625 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-kube-api-access-ch8kb" pod "95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc" (UID: "95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc"). InnerVolumeSpecName "kube-api-access-ch8kb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:26.071194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.070295 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05290867-64ttv"] Oct 30 20:06:26.155729 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.155687 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:26.155729 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.155726 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ch8kb\" (UniqueName: \"kubernetes.io/projected/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc-kube-api-access-ch8kb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:26.308172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.306855 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:26.446374 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.446335 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef30b5ec-0384-47c7-9251-6763ed94fe90" path="/var/lib/kubelet/pods/ef30b5ec-0384-47c7-9251-6763ed94fe90/volumes" Oct 30 20:06:26.956525 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.956497 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" Oct 30 20:06:26.956739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.956535 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm" event={"ID":"95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc","Type":"ContainerDied","Data":"a5e9ee124da30b03a84c2df8ed44972049b8b1203369cf3b9502de5f768caa54"} Oct 30 20:06:26.956739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.956608 5091 scope.go:122] "RemoveContainer" containerID="3260cbee6ac85fdec3db92654ebe711e720f8f47fb5bed6f28f5500de9576f45" Oct 30 20:06:26.991680 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:26.991641 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm"] Oct 30 20:06:27.014193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.014161 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-6c68kkkpm"] Oct 30 20:06:27.412541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.412503 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:27.609165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.604140 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:27.620172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.615193 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6"] Oct 30 20:06:27.620172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.616426 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e90e48ff-e0d8-4495-b447-ccdd7fb3935c" containerName="revoke-keys" Oct 30 20:06:27.620172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.616448 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90e48ff-e0d8-4495-b447-ccdd7fb3935c" containerName="revoke-keys" Oct 30 20:06:27.620172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.616507 5091 cpu_manager.go:414] "containerMap: removing container" podUID="95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc" containerName="payload-processing" Oct 30 20:06:27.620172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.616516 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc" containerName="payload-processing" Oct 30 20:06:27.620172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.616530 5091 cpu_manager.go:414] "containerMap: removing container" podUID="477ec777-3941-4e8b-a088-0877d7e1f5f3" containerName="payload-processing" Oct 30 20:06:27.620172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.616538 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="477ec777-3941-4e8b-a088-0877d7e1f5f3" containerName="payload-processing" Oct 30 20:06:27.620172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.616583 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c0c7bc95-72c8-4b53-9a9e-c0f069783697" containerName="revoke-keys" Oct 30 20:06:27.620172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.616590 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0c7bc95-72c8-4b53-9a9e-c0f069783697" containerName="revoke-keys" Oct 30 20:06:27.620172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.616785 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0c7bc95-72c8-4b53-9a9e-c0f069783697" containerName="revoke-keys" Oct 30 20:06:27.620172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.616804 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90e48ff-e0d8-4495-b447-ccdd7fb3935c" containerName="revoke-keys" Oct 30 20:06:27.620172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.616822 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc" containerName="payload-processing" Oct 30 20:06:27.620172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.616831 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="477ec777-3941-4e8b-a088-0877d7e1f5f3" containerName="payload-processing" Oct 30 20:06:27.637173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.634908 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:27.704958 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.704920 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6"] Oct 30 20:06:28.039432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.976422 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9599d312-d8a2-42bd-81df-9d34ae7d03d0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:28.039432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.976504 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9599d312-d8a2-42bd-81df-9d34ae7d03d0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:28.039432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.976591 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9599d312-d8a2-42bd-81df-9d34ae7d03d0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:28.039432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:27.976958 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzgjv\" (UniqueName: \"kubernetes.io/projected/9599d312-d8a2-42bd-81df-9d34ae7d03d0-kube-api-access-rzgjv\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:28.078451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.078419 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5"] Oct 30 20:06:28.082492 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.082454 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" Oct 30 20:06:28.085185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.085158 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5"] Oct 30 20:06:28.099463 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.099420 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:06:28.099814 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.099793 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:06:28.100376 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.100137 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9599d312-d8a2-42bd-81df-9d34ae7d03d0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:28.100480 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.100416 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9599d312-d8a2-42bd-81df-9d34ae7d03d0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:28.100480 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.100469 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9599d312-d8a2-42bd-81df-9d34ae7d03d0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:28.100683 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.100667 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzgjv\" (UniqueName: \"kubernetes.io/projected/9599d312-d8a2-42bd-81df-9d34ae7d03d0-kube-api-access-rzgjv\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:28.101873 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.101843 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9599d312-d8a2-42bd-81df-9d34ae7d03d0-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:28.102277 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.102249 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9599d312-d8a2-42bd-81df-9d34ae7d03d0-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:28.103773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.103749 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9599d312-d8a2-42bd-81df-9d34ae7d03d0-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:28.136657 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.136631 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzgjv\" (UniqueName: \"kubernetes.io/projected/9599d312-d8a2-42bd-81df-9d34ae7d03d0-kube-api-access-rzgjv\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:28.202336 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.202304 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500fdafc-51b5-4c27-9615-14899c5cc65a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcd17e7c-9n6p5\" (UID: \"500fdafc-51b5-4c27-9615-14899c5cc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" Oct 30 20:06:28.202509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.202371 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500fdafc-51b5-4c27-9615-14899c5cc65a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcd17e7c-9n6p5\" (UID: \"500fdafc-51b5-4c27-9615-14899c5cc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" Oct 30 20:06:28.202583 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.202554 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49qn4\" (UniqueName: \"kubernetes.io/projected/500fdafc-51b5-4c27-9615-14899c5cc65a-kube-api-access-49qn4\") pod \"maas-api-revoke-sub-dcd17e7c-9n6p5\" (UID: \"500fdafc-51b5-4c27-9615-14899c5cc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" Oct 30 20:06:28.276533 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.276499 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:28.305162 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.304679 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49qn4\" (UniqueName: \"kubernetes.io/projected/500fdafc-51b5-4c27-9615-14899c5cc65a-kube-api-access-49qn4\") pod \"maas-api-revoke-sub-dcd17e7c-9n6p5\" (UID: \"500fdafc-51b5-4c27-9615-14899c5cc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" Oct 30 20:06:28.305162 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.304980 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500fdafc-51b5-4c27-9615-14899c5cc65a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcd17e7c-9n6p5\" (UID: \"500fdafc-51b5-4c27-9615-14899c5cc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" Oct 30 20:06:28.305162 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.305056 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500fdafc-51b5-4c27-9615-14899c5cc65a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcd17e7c-9n6p5\" (UID: \"500fdafc-51b5-4c27-9615-14899c5cc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" Oct 30 20:06:28.305690 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.305658 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500fdafc-51b5-4c27-9615-14899c5cc65a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dcd17e7c-9n6p5\" (UID: \"500fdafc-51b5-4c27-9615-14899c5cc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" Oct 30 20:06:28.307958 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.307934 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500fdafc-51b5-4c27-9615-14899c5cc65a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dcd17e7c-9n6p5\" (UID: \"500fdafc-51b5-4c27-9615-14899c5cc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" Oct 30 20:06:28.323374 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.323337 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49qn4\" (UniqueName: \"kubernetes.io/projected/500fdafc-51b5-4c27-9615-14899c5cc65a-kube-api-access-49qn4\") pod \"maas-api-revoke-sub-dcd17e7c-9n6p5\" (UID: \"500fdafc-51b5-4c27-9615-14899c5cc65a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" Oct 30 20:06:28.399509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.399479 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" Oct 30 20:06:28.444311 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.444281 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc" path="/var/lib/kubelet/pods/95ab2e57-b1fb-4b5c-bbae-3f6a7c32e3dc/volumes" Oct 30 20:06:28.582296 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:06:28.582264 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9599d312_d8a2_42bd_81df_9d34ae7d03d0.slice/crio-5b4a8295699220bcd34e13fa137336bbd8540558e93c38312c1f63d94c6001cb WatchSource:0}: Error finding container 5b4a8295699220bcd34e13fa137336bbd8540558e93c38312c1f63d94c6001cb: Status 404 returned error can't find the container with id 5b4a8295699220bcd34e13fa137336bbd8540558e93c38312c1f63d94c6001cb Oct 30 20:06:28.583253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.583229 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6"] Oct 30 20:06:28.717015 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:28.716990 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5"] Oct 30 20:06:28.718099 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:06:28.718076 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500fdafc_51b5_4c27_9615_14899c5cc65a.slice/crio-544a0f81c661b08ade572b43f8e4fa47452cdc7d3970b3720695e80054df7bbb WatchSource:0}: Error finding container 544a0f81c661b08ade572b43f8e4fa47452cdc7d3970b3720695e80054df7bbb: Status 404 returned error can't find the container with id 544a0f81c661b08ade572b43f8e4fa47452cdc7d3970b3720695e80054df7bbb Oct 30 20:06:29.039926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.039887 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" event={"ID":"500fdafc-51b5-4c27-9615-14899c5cc65a","Type":"ContainerStarted","Data":"544a0f81c661b08ade572b43f8e4fa47452cdc7d3970b3720695e80054df7bbb"} Oct 30 20:06:29.041836 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.041805 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" event={"ID":"9599d312-d8a2-42bd-81df-9d34ae7d03d0","Type":"ContainerStarted","Data":"e4fd92620c121cdfb452ebb6acf7cde324d3a0483df07b39635c2aa25437311e"} Oct 30 20:06:29.041983 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.041856 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" event={"ID":"9599d312-d8a2-42bd-81df-9d34ae7d03d0","Type":"ContainerStarted","Data":"5b4a8295699220bcd34e13fa137336bbd8540558e93c38312c1f63d94c6001cb"} Oct 30 20:06:29.041983 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.041918 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:29.096786 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.095226 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:29.109887 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.109814 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" podStartSLOduration=2.1097904769999998 podStartE2EDuration="2.109790477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:29.075983076 +0000 UTC m=+2108.771191036" watchObservedRunningTime="2026-10-30 20:06:29.109790477 +0000 UTC m=+2108.804998436" Oct 30 20:06:29.156515 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.156480 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" podUID="63684302-4590-4e0b-b314-8b3a574ca02c" containerName="payload-processing" containerID="cri-o://78a1227891fffc67d4529c667bd554105a8be2fd0041001f55e46b2d7147e37c" gracePeriod=25 Oct 30 20:06:29.481591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.481559 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:06:29.626939 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.626861 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63684302-4590-4e0b-b314-8b3a574ca02c-kube-api-access-lcpbd\" (UniqueName: \"kubernetes.io/projected/63684302-4590-4e0b-b314-8b3a574ca02c-kube-api-access-lcpbd\") pod \"63684302-4590-4e0b-b314-8b3a574ca02c\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " Oct 30 20:06:29.627104 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.626956 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63684302-4590-4e0b-b314-8b3a574ca02c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63684302-4590-4e0b-b314-8b3a574ca02c-routing-overlay-volume\") pod \"63684302-4590-4e0b-b314-8b3a574ca02c\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " Oct 30 20:06:29.627104 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.627035 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63684302-4590-4e0b-b314-8b3a574ca02c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63684302-4590-4e0b-b314-8b3a574ca02c-plugins-config-volume\") pod \"63684302-4590-4e0b-b314-8b3a574ca02c\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " Oct 30 20:06:29.627311 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.627287 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63684302-4590-4e0b-b314-8b3a574ca02c-routing-overlay-volume" pod "63684302-4590-4e0b-b314-8b3a574ca02c" (UID: "63684302-4590-4e0b-b314-8b3a574ca02c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:29.627402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.627378 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63684302-4590-4e0b-b314-8b3a574ca02c-plugins-config-volume" pod "63684302-4590-4e0b-b314-8b3a574ca02c" (UID: "63684302-4590-4e0b-b314-8b3a574ca02c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:29.627462 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.627392 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63684302-4590-4e0b-b314-8b3a574ca02c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/63684302-4590-4e0b-b314-8b3a574ca02c-provider-credentials\") pod \"63684302-4590-4e0b-b314-8b3a574ca02c\" (UID: \"63684302-4590-4e0b-b314-8b3a574ca02c\") " Oct 30 20:06:29.628091 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.628068 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/63684302-4590-4e0b-b314-8b3a574ca02c-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:29.628091 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.628091 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/63684302-4590-4e0b-b314-8b3a574ca02c-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:29.629643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.629615 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63684302-4590-4e0b-b314-8b3a574ca02c-kube-api-access-lcpbd" pod "63684302-4590-4e0b-b314-8b3a574ca02c" (UID: "63684302-4590-4e0b-b314-8b3a574ca02c"). InnerVolumeSpecName "kube-api-access-lcpbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:29.629799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.629757 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63684302-4590-4e0b-b314-8b3a574ca02c-provider-credentials" pod "63684302-4590-4e0b-b314-8b3a574ca02c" (UID: "63684302-4590-4e0b-b314-8b3a574ca02c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:29.729823 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.729791 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/63684302-4590-4e0b-b314-8b3a574ca02c-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:29.729823 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.729819 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcpbd\" (UniqueName: \"kubernetes.io/projected/63684302-4590-4e0b-b314-8b3a574ca02c-kube-api-access-lcpbd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:29.886323 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:29.886235 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:30.047450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.047405 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:30.053649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.053613 5091 generic.go:356] "Generic (PLEG): container finished" podID="500fdafc-51b5-4c27-9615-14899c5cc65a" containerID="08ebc5f6f64877e818f1c034d72f4897838460f3b64725df95d96cbcea7853d3" exitCode=0 Oct 30 20:06:30.053818 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.053688 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" event={"ID":"500fdafc-51b5-4c27-9615-14899c5cc65a","Type":"ContainerDied","Data":"08ebc5f6f64877e818f1c034d72f4897838460f3b64725df95d96cbcea7853d3"} Oct 30 20:06:30.055551 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.055525 5091 generic.go:356] "Generic (PLEG): container finished" podID="63684302-4590-4e0b-b314-8b3a574ca02c" containerID="78a1227891fffc67d4529c667bd554105a8be2fd0041001f55e46b2d7147e37c" exitCode=0 Oct 30 20:06:30.055708 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.055580 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" Oct 30 20:06:30.055708 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.055607 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" event={"ID":"63684302-4590-4e0b-b314-8b3a574ca02c","Type":"ContainerDied","Data":"78a1227891fffc67d4529c667bd554105a8be2fd0041001f55e46b2d7147e37c"} Oct 30 20:06:30.055708 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.055661 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4" event={"ID":"63684302-4590-4e0b-b314-8b3a574ca02c","Type":"ContainerDied","Data":"049dfa419470cebf185afac97f112379901486d383aefef62eb1ff24487b8d73"} Oct 30 20:06:30.055708 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.055691 5091 scope.go:122] "RemoveContainer" containerID="78a1227891fffc67d4529c667bd554105a8be2fd0041001f55e46b2d7147e37c" Oct 30 20:06:30.070371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.070350 5091 scope.go:122] "RemoveContainer" containerID="78a1227891fffc67d4529c667bd554105a8be2fd0041001f55e46b2d7147e37c" Oct 30 20:06:30.070659 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:30.070640 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a1227891fffc67d4529c667bd554105a8be2fd0041001f55e46b2d7147e37c\": container with ID starting with 78a1227891fffc67d4529c667bd554105a8be2fd0041001f55e46b2d7147e37c not found: ID does not exist" containerID="78a1227891fffc67d4529c667bd554105a8be2fd0041001f55e46b2d7147e37c" Oct 30 20:06:30.070731 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.070664 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a1227891fffc67d4529c667bd554105a8be2fd0041001f55e46b2d7147e37c"} err="failed to get container status \"78a1227891fffc67d4529c667bd554105a8be2fd0041001f55e46b2d7147e37c\": rpc error: code = NotFound desc = could not find container \"78a1227891fffc67d4529c667bd554105a8be2fd0041001f55e46b2d7147e37c\": container with ID starting with 78a1227891fffc67d4529c667bd554105a8be2fd0041001f55e46b2d7147e37c not found: ID does not exist" Oct 30 20:06:30.103083 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.103056 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4"] Oct 30 20:06:30.110415 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.110387 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-2a2286/payload-processing-external-model-e2e-extmod-b-2a2286-6bfbz6gh4"] Oct 30 20:06:30.230218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.227855 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:30.463442 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.463409 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63684302-4590-4e0b-b314-8b3a574ca02c" path="/var/lib/kubelet/pods/63684302-4590-4e0b-b314-8b3a574ca02c/volumes" Oct 30 20:06:30.707155 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.707102 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:30.866769 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:30.866682 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:31.020388 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:31.020349 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:31.326138 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:31.323621 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:31.326507 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:31.326473 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" Oct 30 20:06:31.421409 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:31.421375 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500fdafc-51b5-4c27-9615-14899c5cc65a-kube-api-access-49qn4\" (UniqueName: \"kubernetes.io/projected/500fdafc-51b5-4c27-9615-14899c5cc65a-kube-api-access-49qn4\") pod \"500fdafc-51b5-4c27-9615-14899c5cc65a\" (UID: \"500fdafc-51b5-4c27-9615-14899c5cc65a\") " Oct 30 20:06:31.421609 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:31.421424 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/500fdafc-51b5-4c27-9615-14899c5cc65a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500fdafc-51b5-4c27-9615-14899c5cc65a-maas-api-xks-ca\") pod \"500fdafc-51b5-4c27-9615-14899c5cc65a\" (UID: \"500fdafc-51b5-4c27-9615-14899c5cc65a\") " Oct 30 20:06:31.421609 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:31.421557 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500fdafc-51b5-4c27-9615-14899c5cc65a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500fdafc-51b5-4c27-9615-14899c5cc65a-maas-api-openshift-ca\") pod \"500fdafc-51b5-4c27-9615-14899c5cc65a\" (UID: \"500fdafc-51b5-4c27-9615-14899c5cc65a\") " Oct 30 20:06:31.421911 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:31.421879 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500fdafc-51b5-4c27-9615-14899c5cc65a-maas-api-openshift-ca" pod "500fdafc-51b5-4c27-9615-14899c5cc65a" (UID: "500fdafc-51b5-4c27-9615-14899c5cc65a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:31.422240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:31.422210 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/500fdafc-51b5-4c27-9615-14899c5cc65a-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:31.423661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:31.423634 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/500fdafc-51b5-4c27-9615-14899c5cc65a-maas-api-xks-ca" pod "500fdafc-51b5-4c27-9615-14899c5cc65a" (UID: "500fdafc-51b5-4c27-9615-14899c5cc65a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:31.423743 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:31.423658 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500fdafc-51b5-4c27-9615-14899c5cc65a-kube-api-access-49qn4" pod "500fdafc-51b5-4c27-9615-14899c5cc65a" (UID: "500fdafc-51b5-4c27-9615-14899c5cc65a"). InnerVolumeSpecName "kube-api-access-49qn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:31.526085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:31.523988 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49qn4\" (UniqueName: \"kubernetes.io/projected/500fdafc-51b5-4c27-9615-14899c5cc65a-kube-api-access-49qn4\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:31.526085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:31.524025 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/500fdafc-51b5-4c27-9615-14899c5cc65a-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:31.562074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:31.562037 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:31.719550 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:31.719467 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:32.079470 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:32.079435 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" Oct 30 20:06:32.079653 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:32.079459 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5" event={"ID":"500fdafc-51b5-4c27-9615-14899c5cc65a","Type":"ContainerDied","Data":"544a0f81c661b08ade572b43f8e4fa47452cdc7d3970b3720695e80054df7bbb"} Oct 30 20:06:32.079653 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:32.079517 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="544a0f81c661b08ade572b43f8e4fa47452cdc7d3970b3720695e80054df7bbb" Oct 30 20:06:32.313369 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:32.313316 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:32.471744 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:32.471653 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:33.209380 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:33.205029 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:34.071194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:34.064010 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:35.156193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:35.151658 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:36.046046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:36.046007 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t"] Oct 30 20:06:36.051187 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:36.051122 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-02cca21c-v9z8t"] Oct 30 20:06:36.415644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:36.415547 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:36.451499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:36.451461 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be352a3e-b001-4260-b466-fe15f3d5037d" path="/var/lib/kubelet/pods/be352a3e-b001-4260-b466-fe15f3d5037d/volumes" Oct 30 20:06:37.208895 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.208852 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:37.360234 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.359971 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf"] Oct 30 20:06:37.361857 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.361824 5091 cpu_manager.go:414] "containerMap: removing container" podUID="500fdafc-51b5-4c27-9615-14899c5cc65a" containerName="revoke-keys" Oct 30 20:06:37.362022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.362006 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500fdafc-51b5-4c27-9615-14899c5cc65a" containerName="revoke-keys" Oct 30 20:06:37.362263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.362244 5091 cpu_manager.go:414] "containerMap: removing container" podUID="63684302-4590-4e0b-b314-8b3a574ca02c" containerName="payload-processing" Oct 30 20:06:37.362395 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.362381 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63684302-4590-4e0b-b314-8b3a574ca02c" containerName="payload-processing" Oct 30 20:06:37.362731 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.362709 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="63684302-4590-4e0b-b314-8b3a574ca02c" containerName="payload-processing" Oct 30 20:06:37.373380 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.373348 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="500fdafc-51b5-4c27-9615-14899c5cc65a" containerName="revoke-keys" Oct 30 20:06:37.389004 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.388972 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:37.409897 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.409842 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf"] Oct 30 20:06:37.429022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.428983 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:37.507588 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.502602 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22238b46-4abd-49a7-9299-736e7efab2f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:37.507588 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.503174 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22238b46-4abd-49a7-9299-736e7efab2f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:37.507588 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.503224 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgcdm\" (UniqueName: \"kubernetes.io/projected/22238b46-4abd-49a7-9299-736e7efab2f5-kube-api-access-mgcdm\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:37.507588 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.503417 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22238b46-4abd-49a7-9299-736e7efab2f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:37.611179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.610854 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22238b46-4abd-49a7-9299-736e7efab2f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:37.611179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.610945 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgcdm\" (UniqueName: \"kubernetes.io/projected/22238b46-4abd-49a7-9299-736e7efab2f5-kube-api-access-mgcdm\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:37.611179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.611106 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22238b46-4abd-49a7-9299-736e7efab2f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:37.611530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.611429 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22238b46-4abd-49a7-9299-736e7efab2f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:37.619178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.613114 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22238b46-4abd-49a7-9299-736e7efab2f5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:37.619178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.616829 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22238b46-4abd-49a7-9299-736e7efab2f5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:37.664526 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.657362 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22238b46-4abd-49a7-9299-736e7efab2f5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:37.691900 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.691830 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgcdm\" (UniqueName: \"kubernetes.io/projected/22238b46-4abd-49a7-9299-736e7efab2f5-kube-api-access-mgcdm\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:37.782646 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:37.782536 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:38.090199 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:06:38.090165 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22238b46_4abd_49a7_9299_736e7efab2f5.slice/crio-be0bc946f614ecfa4370ebfb046332b4982bb19639394110447cdb889c75aca3 WatchSource:0}: Error finding container be0bc946f614ecfa4370ebfb046332b4982bb19639394110447cdb889c75aca3: Status 404 returned error can't find the container with id be0bc946f614ecfa4370ebfb046332b4982bb19639394110447cdb889c75aca3 Oct 30 20:06:38.092371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:38.092340 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf"] Oct 30 20:06:38.188233 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:38.188197 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" event={"ID":"22238b46-4abd-49a7-9299-736e7efab2f5","Type":"ContainerStarted","Data":"c1c9b98ac55328719271c2e4c5de22bcfb42daba8de5ddb8574ff33cd0e687ab"} Oct 30 20:06:38.188400 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:38.188264 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" event={"ID":"22238b46-4abd-49a7-9299-736e7efab2f5","Type":"ContainerStarted","Data":"be0bc946f614ecfa4370ebfb046332b4982bb19639394110447cdb889c75aca3"} Oct 30 20:06:38.188400 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:38.188293 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:38.227604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:38.227553 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" podStartSLOduration=1.227539828 podStartE2EDuration="1.227539828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:38.225815566 +0000 UTC m=+2117.921023521" watchObservedRunningTime="2026-10-30 20:06:38.227539828 +0000 UTC m=+2117.922747786" Oct 30 20:06:38.563859 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:38.563816 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:38.781421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:38.781361 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:38.967937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:38.967838 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:39.241420 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:39.239285 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:39.439691 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:39.439644 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:39.580058 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:39.579861 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6"] Oct 30 20:06:39.586769 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:39.584720 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7"] Oct 30 20:06:39.598875 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:39.598051 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:39.686258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:39.686213 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:39.895520 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:39.895383 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:40.253304 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:40.253271 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:40.563550 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:40.563460 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:40.795290 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:40.794758 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:40.876293 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:40.876205 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt"] Oct 30 20:06:40.881371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:40.881350 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:40.892344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:40.892318 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-517027\"/\"kube-root-ca.crt\"" Oct 30 20:06:40.892992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:40.892973 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-517027\"/\"openshift-service-ca.crt\"" Oct 30 20:06:40.893082 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:40.893069 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-517027\"/\"routing-overlay\"" Oct 30 20:06:40.893205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:40.893189 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-517027\"/\"payload-processing-external-model-e2e--c9e3857f-dockercfg-dkbs4\"" Oct 30 20:06:40.893268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:40.893256 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-517027\"/\"agc-cred-7cc1575beefa\"" Oct 30 20:06:40.893350 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:40.893338 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-517027\"/\"payload-processing-external-model-plugins-e2e-extmod-a-517027\"" Oct 30 20:06:40.896126 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:40.896093 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt"] Oct 30 20:06:41.034178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.026221 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmhjc\" (UniqueName: \"kubernetes.io/projected/58ae2933-d944-463e-b43f-c406dcea7748-kube-api-access-hmhjc\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:41.034178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.026293 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58ae2933-d944-463e-b43f-c406dcea7748-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:41.034178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.026429 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ae2933-d944-463e-b43f-c406dcea7748-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:41.034178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.026598 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58ae2933-d944-463e-b43f-c406dcea7748-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:41.392959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.358765 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58ae2933-d944-463e-b43f-c406dcea7748-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:41.392959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.359128 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmhjc\" (UniqueName: \"kubernetes.io/projected/58ae2933-d944-463e-b43f-c406dcea7748-kube-api-access-hmhjc\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:41.392959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.360287 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58ae2933-d944-463e-b43f-c406dcea7748-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:41.392959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.360469 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58ae2933-d944-463e-b43f-c406dcea7748-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:41.392959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.361542 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ae2933-d944-463e-b43f-c406dcea7748-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:41.392959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.361603 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ae2933-d944-463e-b43f-c406dcea7748-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:41.392959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.379909 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58ae2933-d944-463e-b43f-c406dcea7748-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:41.392959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.384054 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmhjc\" (UniqueName: \"kubernetes.io/projected/58ae2933-d944-463e-b43f-c406dcea7748-kube-api-access-hmhjc\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:41.392959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.385940 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:41.517908 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.516497 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:41.777078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.776690 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:41.791176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.790984 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6"] Oct 30 20:06:41.799557 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.799505 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:41.803105 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.803071 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6"] Oct 30 20:06:41.812635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.812597 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-517027\"/\"agc-cred-9302415c93df\"" Oct 30 20:06:41.939937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.939834 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f77f10e-35b0-4413-88a9-0252cc4cdd86-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:41.940120 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.940010 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f77f10e-35b0-4413-88a9-0252cc4cdd86-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:41.940120 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.940042 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f77f10e-35b0-4413-88a9-0252cc4cdd86-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:41.940317 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:41.940119 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlrbc\" (UniqueName: \"kubernetes.io/projected/7f77f10e-35b0-4413-88a9-0252cc4cdd86-kube-api-access-nlrbc\") pod \"payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:42.043447 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.043395 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f77f10e-35b0-4413-88a9-0252cc4cdd86-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:42.043705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.043574 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f77f10e-35b0-4413-88a9-0252cc4cdd86-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:42.043705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.043606 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f77f10e-35b0-4413-88a9-0252cc4cdd86-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:42.043705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.043686 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlrbc\" (UniqueName: \"kubernetes.io/projected/7f77f10e-35b0-4413-88a9-0252cc4cdd86-kube-api-access-nlrbc\") pod \"payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:42.044592 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.044554 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f77f10e-35b0-4413-88a9-0252cc4cdd86-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:42.044843 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.044730 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f77f10e-35b0-4413-88a9-0252cc4cdd86-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:42.047481 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.047447 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f77f10e-35b0-4413-88a9-0252cc4cdd86-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:42.064705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.064677 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlrbc\" (UniqueName: \"kubernetes.io/projected/7f77f10e-35b0-4413-88a9-0252cc4cdd86-kube-api-access-nlrbc\") pod \"payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:42.099721 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:06:42.099685 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ae2933_d944_463e_b43f_c406dcea7748.slice/crio-8b1009f9a8ab88836f418e91b0b05736c834ffe218b7692ff3fe90d0f016c34c WatchSource:0}: Error finding container 8b1009f9a8ab88836f418e91b0b05736c834ffe218b7692ff3fe90d0f016c34c: Status 404 returned error can't find the container with id 8b1009f9a8ab88836f418e91b0b05736c834ffe218b7692ff3fe90d0f016c34c Oct 30 20:06:42.099812 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.099743 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt"] Oct 30 20:06:42.155333 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.155282 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:42.431498 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.431397 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:42.445625 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.445587 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" event={"ID":"58ae2933-d944-463e-b43f-c406dcea7748","Type":"ContainerStarted","Data":"4413da1fb9e9d023f2778ac77c6eec3942af7e05e5158f538c6f28ec384964a4"} Oct 30 20:06:42.445796 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.445656 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:42.445796 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.445680 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" event={"ID":"58ae2933-d944-463e-b43f-c406dcea7748","Type":"ContainerStarted","Data":"8b1009f9a8ab88836f418e91b0b05736c834ffe218b7692ff3fe90d0f016c34c"} Oct 30 20:06:42.469273 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.468285 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6"] Oct 30 20:06:42.470053 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:06:42.470019 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f77f10e_35b0_4413_88a9_0252cc4cdd86.slice/crio-caace1ba8ddf29b3bac34feb6305b1626ba20789ced436542e6164a1afa26a76 WatchSource:0}: Error finding container caace1ba8ddf29b3bac34feb6305b1626ba20789ced436542e6164a1afa26a76: Status 404 returned error can't find the container with id caace1ba8ddf29b3bac34feb6305b1626ba20789ced436542e6164a1afa26a76 Oct 30 20:06:42.486474 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.486418 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" podStartSLOduration=2.486401449 podStartE2EDuration="2.486401449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:42.485701926 +0000 UTC m=+2122.180909885" watchObservedRunningTime="2026-10-30 20:06:42.486401449 +0000 UTC m=+2122.181609410" Oct 30 20:06:42.691728 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.691683 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:42.899407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:42.899363 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:43.101735 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:43.101697 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:43.329897 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:43.329864 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:43.454179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:43.454119 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" event={"ID":"7f77f10e-35b0-4413-88a9-0252cc4cdd86","Type":"ContainerStarted","Data":"0f6295ef9ce886ff962f49786730cc80200a607571c6540cca304c4ef0d8426e"} Oct 30 20:06:43.454416 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:43.454218 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" event={"ID":"7f77f10e-35b0-4413-88a9-0252cc4cdd86","Type":"ContainerStarted","Data":"caace1ba8ddf29b3bac34feb6305b1626ba20789ced436542e6164a1afa26a76"} Oct 30 20:06:43.454819 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:43.454771 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:43.496252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:43.496205 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" podStartSLOduration=2.496192 podStartE2EDuration="2.496192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:43.494491566 +0000 UTC m=+2123.189699523" watchObservedRunningTime="2026-10-30 20:06:43.496192 +0000 UTC m=+2123.191399958" Oct 30 20:06:43.561881 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:43.561841 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:44.032693 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:44.032651 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:44.271245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:44.271209 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:44.583178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:44.582879 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" podUID="9599d312-d8a2-42bd-81df-9d34ae7d03d0" containerName="payload-processing" containerID="cri-o://e4fd92620c121cdfb452ebb6acf7cde324d3a0483df07b39635c2aa25437311e" gracePeriod=25 Oct 30 20:06:44.591168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:44.590268 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" podUID="f0803362-6d1e-4965-b31d-799329a92c99" containerName="payload-processing" containerID="cri-o://0eb062d04406bcdf766b270894d3e3a52a3cf8051deab04564f2c01773d1bdcd" gracePeriod=25 Oct 30 20:06:44.955776 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:44.947533 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:44.962592 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:44.962559 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:45.004245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.004216 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:06:45.032101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.032061 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f"] Oct 30 20:06:45.033302 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.033279 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9599d312-d8a2-42bd-81df-9d34ae7d03d0" containerName="payload-processing" Oct 30 20:06:45.033302 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.033301 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9599d312-d8a2-42bd-81df-9d34ae7d03d0" containerName="payload-processing" Oct 30 20:06:45.033784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.033319 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f0803362-6d1e-4965-b31d-799329a92c99" containerName="payload-processing" Oct 30 20:06:45.033784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.033326 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0803362-6d1e-4965-b31d-799329a92c99" containerName="payload-processing" Oct 30 20:06:45.033784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.033460 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9599d312-d8a2-42bd-81df-9d34ae7d03d0" containerName="payload-processing" Oct 30 20:06:45.033784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.033471 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0803362-6d1e-4965-b31d-799329a92c99" containerName="payload-processing" Oct 30 20:06:45.038842 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.038815 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:45.051067 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.051040 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"payload-processing-external-model-e2e--babb2686-dockercfg-tc6p4\"" Oct 30 20:06:45.051282 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.051259 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"routing-overlay\"" Oct 30 20:06:45.051370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.051168 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"kube-root-ca.crt\"" Oct 30 20:06:45.051370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.051090 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-68685b\"" Oct 30 20:06:45.051370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.051295 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9599d312-d8a2-42bd-81df-9d34ae7d03d0-kube-api-access-rzgjv\" (UniqueName: \"kubernetes.io/projected/9599d312-d8a2-42bd-81df-9d34ae7d03d0-kube-api-access-rzgjv\") pod \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " Oct 30 20:06:45.051529 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.051389 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9599d312-d8a2-42bd-81df-9d34ae7d03d0-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9599d312-d8a2-42bd-81df-9d34ae7d03d0-provider-credentials\") pod \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " Oct 30 20:06:45.051588 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.051559 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9599d312-d8a2-42bd-81df-9d34ae7d03d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9599d312-d8a2-42bd-81df-9d34ae7d03d0-plugins-config-volume\") pod \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " Oct 30 20:06:45.051721 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.051704 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9599d312-d8a2-42bd-81df-9d34ae7d03d0-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9599d312-d8a2-42bd-81df-9d34ae7d03d0-routing-overlay-volume\") pod \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\" (UID: \"9599d312-d8a2-42bd-81df-9d34ae7d03d0\") " Oct 30 20:06:45.052801 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.052777 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9599d312-d8a2-42bd-81df-9d34ae7d03d0-routing-overlay-volume" pod "9599d312-d8a2-42bd-81df-9d34ae7d03d0" (UID: "9599d312-d8a2-42bd-81df-9d34ae7d03d0"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:45.053858 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.053260 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9599d312-d8a2-42bd-81df-9d34ae7d03d0-plugins-config-volume" pod "9599d312-d8a2-42bd-81df-9d34ae7d03d0" (UID: "9599d312-d8a2-42bd-81df-9d34ae7d03d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:45.053858 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.053831 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"openshift-service-ca.crt\"" Oct 30 20:06:45.055015 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.054991 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9599d312-d8a2-42bd-81df-9d34ae7d03d0-kube-api-access-rzgjv" pod "9599d312-d8a2-42bd-81df-9d34ae7d03d0" (UID: "9599d312-d8a2-42bd-81df-9d34ae7d03d0"). InnerVolumeSpecName "kube-api-access-rzgjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:45.064413 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.064385 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f"] Oct 30 20:06:45.064546 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.064518 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9599d312-d8a2-42bd-81df-9d34ae7d03d0-provider-credentials" pod "9599d312-d8a2-42bd-81df-9d34ae7d03d0" (UID: "9599d312-d8a2-42bd-81df-9d34ae7d03d0"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:45.171222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.165845 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0803362-6d1e-4965-b31d-799329a92c99-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0803362-6d1e-4965-b31d-799329a92c99-routing-overlay-volume\") pod \"f0803362-6d1e-4965-b31d-799329a92c99\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " Oct 30 20:06:45.171222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.165919 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0803362-6d1e-4965-b31d-799329a92c99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0803362-6d1e-4965-b31d-799329a92c99-plugins-config-volume\") pod \"f0803362-6d1e-4965-b31d-799329a92c99\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " Oct 30 20:06:45.171222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.166089 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0803362-6d1e-4965-b31d-799329a92c99-kube-api-access-fbsv7\" (UniqueName: \"kubernetes.io/projected/f0803362-6d1e-4965-b31d-799329a92c99-kube-api-access-fbsv7\") pod \"f0803362-6d1e-4965-b31d-799329a92c99\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " Oct 30 20:06:45.171222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.166242 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0803362-6d1e-4965-b31d-799329a92c99-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0803362-6d1e-4965-b31d-799329a92c99-provider-credentials\") pod \"f0803362-6d1e-4965-b31d-799329a92c99\" (UID: \"f0803362-6d1e-4965-b31d-799329a92c99\") " Oct 30 20:06:45.171222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.166808 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:45.171222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.167009 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6edd28d3-81f6-4be5-807c-2616e250eebd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:45.171222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.167377 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6edd28d3-81f6-4be5-807c-2616e250eebd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:45.171222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.167435 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr67p\" (UniqueName: \"kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-kube-api-access-jr67p\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:45.171222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.167530 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9599d312-d8a2-42bd-81df-9d34ae7d03d0-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:45.171222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.167550 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9599d312-d8a2-42bd-81df-9d34ae7d03d0-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:45.171222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.167566 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzgjv\" (UniqueName: \"kubernetes.io/projected/9599d312-d8a2-42bd-81df-9d34ae7d03d0-kube-api-access-rzgjv\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:45.171222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.167581 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9599d312-d8a2-42bd-81df-9d34ae7d03d0-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:45.171222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.168984 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0803362-6d1e-4965-b31d-799329a92c99-plugins-config-volume" pod "f0803362-6d1e-4965-b31d-799329a92c99" (UID: "f0803362-6d1e-4965-b31d-799329a92c99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:45.195169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.177810 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0803362-6d1e-4965-b31d-799329a92c99-kube-api-access-fbsv7" pod "f0803362-6d1e-4965-b31d-799329a92c99" (UID: "f0803362-6d1e-4965-b31d-799329a92c99"). InnerVolumeSpecName "kube-api-access-fbsv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:45.214268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.202414 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0803362-6d1e-4965-b31d-799329a92c99-routing-overlay-volume" pod "f0803362-6d1e-4965-b31d-799329a92c99" (UID: "f0803362-6d1e-4965-b31d-799329a92c99"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:45.214268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.213729 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"agc-cred-8f19d92ff4bd\"" Oct 30 20:06:45.225500 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.225394 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0803362-6d1e-4965-b31d-799329a92c99-provider-credentials" pod "f0803362-6d1e-4965-b31d-799329a92c99" (UID: "f0803362-6d1e-4965-b31d-799329a92c99"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:45.298085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.298019 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6edd28d3-81f6-4be5-807c-2616e250eebd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:45.298292 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.297261 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6edd28d3-81f6-4be5-807c-2616e250eebd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:45.298486 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.298468 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr67p\" (UniqueName: \"kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-kube-api-access-jr67p\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:45.298628 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.298588 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:45.298879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.298866 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6edd28d3-81f6-4be5-807c-2616e250eebd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:45.307248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.307216 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbsv7\" (UniqueName: \"kubernetes.io/projected/f0803362-6d1e-4965-b31d-799329a92c99-kube-api-access-fbsv7\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:45.312169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.307464 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f0803362-6d1e-4965-b31d-799329a92c99-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:45.312361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.312340 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f0803362-6d1e-4965-b31d-799329a92c99-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:45.312474 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.312462 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0803362-6d1e-4965-b31d-799329a92c99-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:45.312554 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.311266 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6edd28d3-81f6-4be5-807c-2616e250eebd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:45.327229 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.327160 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:45.366177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.362979 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr67p\" (UniqueName: \"kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-kube-api-access-jr67p\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:45.407018 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.383258 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:45.461501 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.444903 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:45.544718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.544677 5091 generic.go:356] "Generic (PLEG): container finished" podID="f0803362-6d1e-4965-b31d-799329a92c99" containerID="0eb062d04406bcdf766b270894d3e3a52a3cf8051deab04564f2c01773d1bdcd" exitCode=0 Oct 30 20:06:45.545253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.545193 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" event={"ID":"f0803362-6d1e-4965-b31d-799329a92c99","Type":"ContainerDied","Data":"0eb062d04406bcdf766b270894d3e3a52a3cf8051deab04564f2c01773d1bdcd"} Oct 30 20:06:45.545587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.545560 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" event={"ID":"f0803362-6d1e-4965-b31d-799329a92c99","Type":"ContainerDied","Data":"a253313376b4c5ad8db8267cb72e726d505469c9e004ba2e190e7172a3a30e90"} Oct 30 20:06:45.545746 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.545731 5091 scope.go:122] "RemoveContainer" containerID="0eb062d04406bcdf766b270894d3e3a52a3cf8051deab04564f2c01773d1bdcd" Oct 30 20:06:45.545958 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.545497 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7" Oct 30 20:06:45.551589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.551371 5091 generic.go:356] "Generic (PLEG): container finished" podID="9599d312-d8a2-42bd-81df-9d34ae7d03d0" containerID="e4fd92620c121cdfb452ebb6acf7cde324d3a0483df07b39635c2aa25437311e" exitCode=0 Oct 30 20:06:45.551589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.551419 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" Oct 30 20:06:45.551589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.551526 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" event={"ID":"9599d312-d8a2-42bd-81df-9d34ae7d03d0","Type":"ContainerDied","Data":"e4fd92620c121cdfb452ebb6acf7cde324d3a0483df07b39635c2aa25437311e"} Oct 30 20:06:45.551589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.551560 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6" event={"ID":"9599d312-d8a2-42bd-81df-9d34ae7d03d0","Type":"ContainerDied","Data":"5b4a8295699220bcd34e13fa137336bbd8540558e93c38312c1f63d94c6001cb"} Oct 30 20:06:45.569093 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.569034 5091 scope.go:122] "RemoveContainer" containerID="0eb062d04406bcdf766b270894d3e3a52a3cf8051deab04564f2c01773d1bdcd" Oct 30 20:06:45.569652 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:45.569448 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb062d04406bcdf766b270894d3e3a52a3cf8051deab04564f2c01773d1bdcd\": container with ID starting with 0eb062d04406bcdf766b270894d3e3a52a3cf8051deab04564f2c01773d1bdcd not found: ID does not exist" containerID="0eb062d04406bcdf766b270894d3e3a52a3cf8051deab04564f2c01773d1bdcd" Oct 30 20:06:45.569652 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.569485 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb062d04406bcdf766b270894d3e3a52a3cf8051deab04564f2c01773d1bdcd"} err="failed to get container status \"0eb062d04406bcdf766b270894d3e3a52a3cf8051deab04564f2c01773d1bdcd\": rpc error: code = NotFound desc = could not find container \"0eb062d04406bcdf766b270894d3e3a52a3cf8051deab04564f2c01773d1bdcd\": container with ID starting with 0eb062d04406bcdf766b270894d3e3a52a3cf8051deab04564f2c01773d1bdcd not found: ID does not exist" Oct 30 20:06:45.569652 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.569509 5091 scope.go:122] "RemoveContainer" containerID="e4fd92620c121cdfb452ebb6acf7cde324d3a0483df07b39635c2aa25437311e" Oct 30 20:06:45.586845 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.583663 5091 scope.go:122] "RemoveContainer" containerID="e4fd92620c121cdfb452ebb6acf7cde324d3a0483df07b39635c2aa25437311e" Oct 30 20:06:45.586845 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:45.583969 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4fd92620c121cdfb452ebb6acf7cde324d3a0483df07b39635c2aa25437311e\": container with ID starting with e4fd92620c121cdfb452ebb6acf7cde324d3a0483df07b39635c2aa25437311e not found: ID does not exist" containerID="e4fd92620c121cdfb452ebb6acf7cde324d3a0483df07b39635c2aa25437311e" Oct 30 20:06:45.586845 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.584004 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4fd92620c121cdfb452ebb6acf7cde324d3a0483df07b39635c2aa25437311e"} err="failed to get container status \"e4fd92620c121cdfb452ebb6acf7cde324d3a0483df07b39635c2aa25437311e\": rpc error: code = NotFound desc = could not find container \"e4fd92620c121cdfb452ebb6acf7cde324d3a0483df07b39635c2aa25437311e\": container with ID starting with e4fd92620c121cdfb452ebb6acf7cde324d3a0483df07b39635c2aa25437311e not found: ID does not exist" Oct 30 20:06:45.599912 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.599883 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7"] Oct 30 20:06:45.608765 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.607342 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-67f6m88g7"] Oct 30 20:06:45.628995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.628968 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6"] Oct 30 20:06:45.636671 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.636366 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-66dclnlh6"] Oct 30 20:06:45.850244 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:06:45.850216 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6edd28d3_81f6_4be5_807c_2616e250eebd.slice/crio-f1c113c4d22426df5df5938ad8ae763d58e78498b9ae14db67d904c9e5ba8568 WatchSource:0}: Error finding container f1c113c4d22426df5df5938ad8ae763d58e78498b9ae14db67d904c9e5ba8568: Status 404 returned error can't find the container with id f1c113c4d22426df5df5938ad8ae763d58e78498b9ae14db67d904c9e5ba8568 Oct 30 20:06:45.852785 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:45.852759 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f"] Oct 30 20:06:45.977293 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:45.977258 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:06:46.444486 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:46.444456 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9599d312-d8a2-42bd-81df-9d34ae7d03d0" path="/var/lib/kubelet/pods/9599d312-d8a2-42bd-81df-9d34ae7d03d0/volumes" Oct 30 20:06:46.444960 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:46.444944 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0803362-6d1e-4965-b31d-799329a92c99" path="/var/lib/kubelet/pods/f0803362-6d1e-4965-b31d-799329a92c99/volumes" Oct 30 20:06:46.496182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:46.495121 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:46.564826 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:46.564790 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" event={"ID":"6edd28d3-81f6-4be5-807c-2616e250eebd","Type":"ContainerStarted","Data":"06e32f8f523e5719f75738b78c7316db15d64d88a4e524430e67f375a21b6dde"} Oct 30 20:06:46.565026 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:46.564842 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" event={"ID":"6edd28d3-81f6-4be5-807c-2616e250eebd","Type":"ContainerStarted","Data":"f1c113c4d22426df5df5938ad8ae763d58e78498b9ae14db67d904c9e5ba8568"} Oct 30 20:06:46.565026 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:46.565004 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:46.598157 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:46.598091 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" podStartSLOduration=2.598074084 podStartE2EDuration="2.598074084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:46.591009957 +0000 UTC m=+2126.286217914" watchObservedRunningTime="2026-10-30 20:06:46.598074084 +0000 UTC m=+2126.293282045" Oct 30 20:06:46.758456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:46.758420 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:47.214758 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.214669 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:47.345185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.344202 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj"] Oct 30 20:06:47.376355 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.376319 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:47.394166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.389712 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj"] Oct 30 20:06:47.516863 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.510802 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"e2e-cfl-19316d-provider-cred\"" Oct 30 20:06:47.555291 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.555243 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a331856-8296-4c36-8263-e88f28287497-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:47.555453 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.555372 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a331856-8296-4c36-8263-e88f28287497-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:47.555524 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.555455 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a331856-8296-4c36-8263-e88f28287497-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:47.555727 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.555584 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnnqs\" (UniqueName: \"kubernetes.io/projected/7a331856-8296-4c36-8263-e88f28287497-kube-api-access-jnnqs\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:47.556451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.555587 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:47.657301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.657255 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a331856-8296-4c36-8263-e88f28287497-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:47.657510 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.657377 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a331856-8296-4c36-8263-e88f28287497-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:47.657583 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.657562 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnnqs\" (UniqueName: \"kubernetes.io/projected/7a331856-8296-4c36-8263-e88f28287497-kube-api-access-jnnqs\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:47.657755 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.657733 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a331856-8296-4c36-8263-e88f28287497-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:47.657958 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:47.657936 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-68685b/agc-cred-8f19d92ff4bd: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:06:47.658659 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:47.658044 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:06:47.658792 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.658253 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a331856-8296-4c36-8263-e88f28287497-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:47.658855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.658670 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a331856-8296-4c36-8263-e88f28287497-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:47.658855 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:47.658781 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials podName:6edd28d3-81f6-4be5-807c-2616e250eebd nodeName:}" failed. No retries permitted until 2026-10-30 20:06:48.15875776 +0000 UTC m=+2127.853965718 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" (UID: "6edd28d3-81f6-4be5-807c-2616e250eebd") : secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:06:47.660209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.660182 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a331856-8296-4c36-8263-e88f28287497-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:47.677504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.677472 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnnqs\" (UniqueName: \"kubernetes.io/projected/7a331856-8296-4c36-8263-e88f28287497-kube-api-access-jnnqs\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:47.855406 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:47.855327 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:48.170347 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:48.169476 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-68685b/agc-cred-8f19d92ff4bd: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:06:48.170347 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:48.169631 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:06:48.170347 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:48.169737 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials podName:6edd28d3-81f6-4be5-807c-2616e250eebd nodeName:}" failed. No retries permitted until 2026-10-30 20:06:49.169713186 +0000 UTC m=+2128.864921150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" (UID: "6edd28d3-81f6-4be5-807c-2616e250eebd") : secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:06:48.186613 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:48.174759 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj"] Oct 30 20:06:48.189005 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:48.188971 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:48.587910 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:48.587870 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" event={"ID":"7a331856-8296-4c36-8263-e88f28287497","Type":"ContainerStarted","Data":"eba62ff5d1926089857435cb8f2bf6de957902622a627595d7457a4e7e685921"} Oct 30 20:06:48.588476 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:48.587938 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:48.588476 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:48.587959 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" event={"ID":"7a331856-8296-4c36-8263-e88f28287497","Type":"ContainerStarted","Data":"df7adcef97bbedc6e69d387072306799df5db3952e60c6e8ad4fcd38443b9da4"} Oct 30 20:06:48.616934 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:48.616873 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" podStartSLOduration=1.616858327 podStartE2EDuration="1.616858327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:48.614705066 +0000 UTC m=+2128.309913024" watchObservedRunningTime="2026-10-30 20:06:48.616858327 +0000 UTC m=+2128.312066285" Oct 30 20:06:49.187995 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:49.187953 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-68685b/agc-cred-8f19d92ff4bd: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:06:49.187995 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:49.187984 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:06:49.188284 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:49.188048 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials podName:6edd28d3-81f6-4be5-807c-2616e250eebd nodeName:}" failed. No retries permitted until 2026-10-30 20:06:51.188033157 +0000 UTC m=+2130.883241092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" (UID: "6edd28d3-81f6-4be5-807c-2616e250eebd") : secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:06:49.238169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:49.228526 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:06:49.238169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:49.231261 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:49.337220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:49.335509 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89"] Oct 30 20:06:49.632678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:49.632635 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:49.797834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:49.797793 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:49.988112 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:49.988033 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5"] Oct 30 20:06:49.992889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:49.992869 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" Oct 30 20:06:50.025052 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.001203 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:06:50.025052 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.002634 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:06:50.025052 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.015354 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5"] Oct 30 20:06:50.062593 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.062555 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:50.117063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.117032 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pth5h\" (UniqueName: \"kubernetes.io/projected/9de2d614-d949-43a8-805d-4c828da19cc6-kube-api-access-pth5h\") pod \"maas-api-revoke-sub-bbcf75d5-tc7v5\" (UID: \"9de2d614-d949-43a8-805d-4c828da19cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" Oct 30 20:06:50.117262 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.117101 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de2d614-d949-43a8-805d-4c828da19cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbcf75d5-tc7v5\" (UID: \"9de2d614-d949-43a8-805d-4c828da19cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" Oct 30 20:06:50.117340 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.117320 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de2d614-d949-43a8-805d-4c828da19cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbcf75d5-tc7v5\" (UID: \"9de2d614-d949-43a8-805d-4c828da19cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" Oct 30 20:06:50.218538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.218498 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de2d614-d949-43a8-805d-4c828da19cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbcf75d5-tc7v5\" (UID: \"9de2d614-d949-43a8-805d-4c828da19cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" Oct 30 20:06:50.218742 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.218714 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pth5h\" (UniqueName: \"kubernetes.io/projected/9de2d614-d949-43a8-805d-4c828da19cc6-kube-api-access-pth5h\") pod \"maas-api-revoke-sub-bbcf75d5-tc7v5\" (UID: \"9de2d614-d949-43a8-805d-4c828da19cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" Oct 30 20:06:50.218970 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.218936 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de2d614-d949-43a8-805d-4c828da19cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbcf75d5-tc7v5\" (UID: \"9de2d614-d949-43a8-805d-4c828da19cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" Oct 30 20:06:50.219739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.219711 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de2d614-d949-43a8-805d-4c828da19cc6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbcf75d5-tc7v5\" (UID: \"9de2d614-d949-43a8-805d-4c828da19cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" Oct 30 20:06:50.221325 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.221301 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de2d614-d949-43a8-805d-4c828da19cc6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbcf75d5-tc7v5\" (UID: \"9de2d614-d949-43a8-805d-4c828da19cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" Oct 30 20:06:50.251175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.251096 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pth5h\" (UniqueName: \"kubernetes.io/projected/9de2d614-d949-43a8-805d-4c828da19cc6-kube-api-access-pth5h\") pod \"maas-api-revoke-sub-bbcf75d5-tc7v5\" (UID: \"9de2d614-d949-43a8-805d-4c828da19cc6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" Oct 30 20:06:50.311615 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.311578 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:50.317566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.317543 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" Oct 30 20:06:50.566275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.564866 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:50.818067 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:06:50.817991 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9de2d614_d949_43a8_805d_4c828da19cc6.slice/crio-c399ad9a271900915a120f7b9570663bb0c76ae4cbd2d96f515dcb1b8483e256 WatchSource:0}: Error finding container c399ad9a271900915a120f7b9570663bb0c76ae4cbd2d96f515dcb1b8483e256: Status 404 returned error can't find the container with id c399ad9a271900915a120f7b9570663bb0c76ae4cbd2d96f515dcb1b8483e256 Oct 30 20:06:50.819121 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:50.819088 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5"] Oct 30 20:06:51.251693 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:51.251664 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-68685b/agc-cred-8f19d92ff4bd: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:06:51.251693 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:51.251691 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:06:51.251891 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:51.251741 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials podName:6edd28d3-81f6-4be5-807c-2616e250eebd nodeName:}" failed. No retries permitted until 2026-10-30 20:06:55.251725456 +0000 UTC m=+2134.946933404 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" (UID: "6edd28d3-81f6-4be5-807c-2616e250eebd") : secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:06:51.665232 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:51.664418 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:51.744056 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:51.744019 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" event={"ID":"9de2d614-d949-43a8-805d-4c828da19cc6","Type":"ContainerStarted","Data":"4bcaacd129c95f8d0807d8760238ff7f12f8dda32595631b3662b94e7354160c"} Oct 30 20:06:51.744230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:51.744083 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" event={"ID":"9de2d614-d949-43a8-805d-4c828da19cc6","Type":"ContainerStarted","Data":"c399ad9a271900915a120f7b9570663bb0c76ae4cbd2d96f515dcb1b8483e256"} Oct 30 20:06:51.769823 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:51.769771 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" podStartSLOduration=2.769756166 podStartE2EDuration="2.769756166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:06:51.767952479 +0000 UTC m=+2131.463160436" watchObservedRunningTime="2026-10-30 20:06:51.769756166 +0000 UTC m=+2131.464964123" Oct 30 20:06:52.192491 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:52.192453 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:52.756793 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:52.756756 5091 generic.go:356] "Generic (PLEG): container finished" podID="9de2d614-d949-43a8-805d-4c828da19cc6" containerID="4bcaacd129c95f8d0807d8760238ff7f12f8dda32595631b3662b94e7354160c" exitCode=0 Oct 30 20:06:52.756976 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:52.756832 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" event={"ID":"9de2d614-d949-43a8-805d-4c828da19cc6","Type":"ContainerDied","Data":"4bcaacd129c95f8d0807d8760238ff7f12f8dda32595631b3662b94e7354160c"} Oct 30 20:06:52.871827 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:52.871788 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:53.457726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:53.457696 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:53.936167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:53.935481 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:54.025839 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.025778 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" Oct 30 20:06:54.113817 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.113787 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9de2d614-d949-43a8-805d-4c828da19cc6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de2d614-d949-43a8-805d-4c828da19cc6-maas-api-xks-ca\") pod \"9de2d614-d949-43a8-805d-4c828da19cc6\" (UID: \"9de2d614-d949-43a8-805d-4c828da19cc6\") " Oct 30 20:06:54.113978 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.113923 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9de2d614-d949-43a8-805d-4c828da19cc6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de2d614-d949-43a8-805d-4c828da19cc6-maas-api-openshift-ca\") pod \"9de2d614-d949-43a8-805d-4c828da19cc6\" (UID: \"9de2d614-d949-43a8-805d-4c828da19cc6\") " Oct 30 20:06:54.114044 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.114013 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9de2d614-d949-43a8-805d-4c828da19cc6-kube-api-access-pth5h\" (UniqueName: \"kubernetes.io/projected/9de2d614-d949-43a8-805d-4c828da19cc6-kube-api-access-pth5h\") pod \"9de2d614-d949-43a8-805d-4c828da19cc6\" (UID: \"9de2d614-d949-43a8-805d-4c828da19cc6\") " Oct 30 20:06:54.114429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.114397 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9de2d614-d949-43a8-805d-4c828da19cc6-maas-api-openshift-ca" pod "9de2d614-d949-43a8-805d-4c828da19cc6" (UID: "9de2d614-d949-43a8-805d-4c828da19cc6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:54.114997 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.114965 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9de2d614-d949-43a8-805d-4c828da19cc6-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:54.116090 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.116066 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9de2d614-d949-43a8-805d-4c828da19cc6-maas-api-xks-ca" pod "9de2d614-d949-43a8-805d-4c828da19cc6" (UID: "9de2d614-d949-43a8-805d-4c828da19cc6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:06:54.116269 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.116247 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9de2d614-d949-43a8-805d-4c828da19cc6-kube-api-access-pth5h" pod "9de2d614-d949-43a8-805d-4c828da19cc6" (UID: "9de2d614-d949-43a8-805d-4c828da19cc6"). InnerVolumeSpecName "kube-api-access-pth5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:54.216360 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.216328 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9de2d614-d949-43a8-805d-4c828da19cc6-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:54.216360 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.216356 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pth5h\" (UniqueName: \"kubernetes.io/projected/9de2d614-d949-43a8-805d-4c828da19cc6-kube-api-access-pth5h\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:54.336348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.336257 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" podUID="b9294290-0590-4829-b4d5-9def2109de6b" containerName="payload-processing" containerID="cri-o://e7c363af74ec467bb4620cb4290cc050eab5b870e4f43ce57abb8d6cada02a59" gracePeriod=25 Oct 30 20:06:54.515624 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.515585 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:06:54.577944 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.577907 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt"] Oct 30 20:06:54.625627 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.625598 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:54.724402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.724369 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9294290-0590-4829-b4d5-9def2109de6b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9294290-0590-4829-b4d5-9def2109de6b-provider-credentials\") pod \"b9294290-0590-4829-b4d5-9def2109de6b\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " Oct 30 20:06:54.724612 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.724421 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9294290-0590-4829-b4d5-9def2109de6b-kube-api-access-dkmks\" (UniqueName: \"kubernetes.io/projected/b9294290-0590-4829-b4d5-9def2109de6b-kube-api-access-dkmks\") pod \"b9294290-0590-4829-b4d5-9def2109de6b\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " Oct 30 20:06:54.724612 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.724555 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9294290-0590-4829-b4d5-9def2109de6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9294290-0590-4829-b4d5-9def2109de6b-plugins-config-volume\") pod \"b9294290-0590-4829-b4d5-9def2109de6b\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " Oct 30 20:06:54.724756 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.724692 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9294290-0590-4829-b4d5-9def2109de6b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9294290-0590-4829-b4d5-9def2109de6b-routing-overlay-volume\") pod \"b9294290-0590-4829-b4d5-9def2109de6b\" (UID: \"b9294290-0590-4829-b4d5-9def2109de6b\") " Oct 30 20:06:54.725033 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.725004 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9294290-0590-4829-b4d5-9def2109de6b-plugins-config-volume" pod "b9294290-0590-4829-b4d5-9def2109de6b" (UID: "b9294290-0590-4829-b4d5-9def2109de6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:54.725167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.725006 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9294290-0590-4829-b4d5-9def2109de6b-routing-overlay-volume" pod "b9294290-0590-4829-b4d5-9def2109de6b" (UID: "b9294290-0590-4829-b4d5-9def2109de6b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:54.725478 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.725457 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9294290-0590-4829-b4d5-9def2109de6b-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:54.725567 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.725482 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9294290-0590-4829-b4d5-9def2109de6b-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:54.726692 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.726661 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9294290-0590-4829-b4d5-9def2109de6b-provider-credentials" pod "b9294290-0590-4829-b4d5-9def2109de6b" (UID: "b9294290-0590-4829-b4d5-9def2109de6b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:54.727213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.727189 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9294290-0590-4829-b4d5-9def2109de6b-kube-api-access-dkmks" pod "b9294290-0590-4829-b4d5-9def2109de6b" (UID: "b9294290-0590-4829-b4d5-9def2109de6b"). InnerVolumeSpecName "kube-api-access-dkmks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:54.778465 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.778438 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" Oct 30 20:06:54.778626 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.778464 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5" event={"ID":"9de2d614-d949-43a8-805d-4c828da19cc6","Type":"ContainerDied","Data":"c399ad9a271900915a120f7b9570663bb0c76ae4cbd2d96f515dcb1b8483e256"} Oct 30 20:06:54.778626 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.778525 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c399ad9a271900915a120f7b9570663bb0c76ae4cbd2d96f515dcb1b8483e256" Oct 30 20:06:54.780329 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.780299 5091 generic.go:356] "Generic (PLEG): container finished" podID="b9294290-0590-4829-b4d5-9def2109de6b" containerID="e7c363af74ec467bb4620cb4290cc050eab5b870e4f43ce57abb8d6cada02a59" exitCode=0 Oct 30 20:06:54.780519 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.780498 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" event={"ID":"b9294290-0590-4829-b4d5-9def2109de6b","Type":"ContainerDied","Data":"e7c363af74ec467bb4620cb4290cc050eab5b870e4f43ce57abb8d6cada02a59"} Oct 30 20:06:54.780648 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.780543 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" event={"ID":"b9294290-0590-4829-b4d5-9def2109de6b","Type":"ContainerDied","Data":"00820084343ea4fc3179639898cd0c291a293e76737eb38c42f1d314231ef818"} Oct 30 20:06:54.780648 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.780571 5091 scope.go:122] "RemoveContainer" containerID="e7c363af74ec467bb4620cb4290cc050eab5b870e4f43ce57abb8d6cada02a59" Oct 30 20:06:54.781017 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.780755 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89" Oct 30 20:06:54.857625 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.857303 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9294290-0590-4829-b4d5-9def2109de6b-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:54.857625 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.857344 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkmks\" (UniqueName: \"kubernetes.io/projected/b9294290-0590-4829-b4d5-9def2109de6b-kube-api-access-dkmks\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:54.861296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.861172 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:54.869546 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.865848 5091 scope.go:122] "RemoveContainer" containerID="e7c363af74ec467bb4620cb4290cc050eab5b870e4f43ce57abb8d6cada02a59" Oct 30 20:06:54.869546 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:54.867987 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c363af74ec467bb4620cb4290cc050eab5b870e4f43ce57abb8d6cada02a59\": container with ID starting with e7c363af74ec467bb4620cb4290cc050eab5b870e4f43ce57abb8d6cada02a59 not found: ID does not exist" containerID="e7c363af74ec467bb4620cb4290cc050eab5b870e4f43ce57abb8d6cada02a59" Oct 30 20:06:54.869546 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.868022 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c363af74ec467bb4620cb4290cc050eab5b870e4f43ce57abb8d6cada02a59"} err="failed to get container status \"e7c363af74ec467bb4620cb4290cc050eab5b870e4f43ce57abb8d6cada02a59\": rpc error: code = NotFound desc = could not find container \"e7c363af74ec467bb4620cb4290cc050eab5b870e4f43ce57abb8d6cada02a59\": container with ID starting with e7c363af74ec467bb4620cb4290cc050eab5b870e4f43ce57abb8d6cada02a59 not found: ID does not exist" Oct 30 20:06:54.895256 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.895230 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89"] Oct 30 20:06:54.899850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:54.899824 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-cc4dzfg89"] Oct 30 20:06:55.036213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:55.036177 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj"] Oct 30 20:06:55.044314 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:55.044287 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f428cc16-zf7zj"] Oct 30 20:06:55.264584 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:55.264552 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-68685b/agc-cred-8f19d92ff4bd: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:06:55.264584 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:55.264580 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:06:55.264816 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:06:55.264650 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials podName:6edd28d3-81f6-4be5-807c-2616e250eebd nodeName:}" failed. No retries permitted until 2026-10-30 20:07:03.264635907 +0000 UTC m=+2142.959843856 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" (UID: "6edd28d3-81f6-4be5-807c-2616e250eebd") : secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:06:55.722225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:55.721476 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:56.443716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:56.443673 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a386a76c-8c8c-4de2-a5f4-86560053751f" path="/var/lib/kubelet/pods/a386a76c-8c8c-4de2-a5f4-86560053751f/volumes" Oct 30 20:06:56.444302 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:56.444260 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9294290-0590-4829-b4d5-9def2109de6b" path="/var/lib/kubelet/pods/b9294290-0590-4829-b4d5-9def2109de6b/volumes" Oct 30 20:06:56.703225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:56.701935 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:57.509043 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:57.508813 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:57.599254 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:57.597684 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:06:57.684612 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:57.684551 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:58.106168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:58.100498 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:58.402232 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:58.392915 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:58.650424 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:58.650378 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:59.149941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.149904 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm"] Oct 30 20:06:59.151536 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.151508 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9de2d614-d949-43a8-805d-4c828da19cc6" containerName="revoke-keys" Oct 30 20:06:59.151728 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.151705 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9de2d614-d949-43a8-805d-4c828da19cc6" containerName="revoke-keys" Oct 30 20:06:59.151869 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.151855 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b9294290-0590-4829-b4d5-9def2109de6b" containerName="payload-processing" Oct 30 20:06:59.152003 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.151871 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9294290-0590-4829-b4d5-9def2109de6b" containerName="payload-processing" Oct 30 20:06:59.152333 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.152303 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9294290-0590-4829-b4d5-9def2109de6b" containerName="payload-processing" Oct 30 20:06:59.152449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.152358 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9de2d614-d949-43a8-805d-4c828da19cc6" containerName="revoke-keys" Oct 30 20:06:59.164260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.164219 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm"] Oct 30 20:06:59.165089 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.165067 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:06:59.171609 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.171588 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"openshift-service-ca.crt\"" Oct 30 20:06:59.171716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.171656 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"payload-processing-external-model-plugins-e2e-extmod-a-222480\"" Oct 30 20:06:59.171716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.171708 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"routing-overlay\"" Oct 30 20:06:59.174399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.174380 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"kube-root-ca.crt\"" Oct 30 20:06:59.174482 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.174443 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"e2e-cmp-legacy-595f66-provider-cred\"" Oct 30 20:06:59.174556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.174541 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"payload-processing-external-model-e2e--8e4491f3-dockercfg-fjf64\"" Oct 30 20:06:59.336169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.334867 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c69eee1-3140-4420-877f-274bd8d6eca5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:06:59.336169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.334932 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c69eee1-3140-4420-877f-274bd8d6eca5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:06:59.336169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.334984 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db2rw\" (UniqueName: \"kubernetes.io/projected/2c69eee1-3140-4420-877f-274bd8d6eca5-kube-api-access-db2rw\") pod \"payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:06:59.336169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.335119 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c69eee1-3140-4420-877f-274bd8d6eca5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:06:59.346029 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.345989 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:06:59.436764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.436675 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c69eee1-3140-4420-877f-274bd8d6eca5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:06:59.436764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.436720 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c69eee1-3140-4420-877f-274bd8d6eca5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:06:59.436764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.436755 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db2rw\" (UniqueName: \"kubernetes.io/projected/2c69eee1-3140-4420-877f-274bd8d6eca5-kube-api-access-db2rw\") pod \"payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:06:59.437047 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.436836 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c69eee1-3140-4420-877f-274bd8d6eca5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:06:59.437601 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.437567 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c69eee1-3140-4420-877f-274bd8d6eca5-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:06:59.437709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.437670 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c69eee1-3140-4420-877f-274bd8d6eca5-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:06:59.439543 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.439522 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c69eee1-3140-4420-877f-274bd8d6eca5-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:06:59.452679 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.452653 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db2rw\" (UniqueName: \"kubernetes.io/projected/2c69eee1-3140-4420-877f-274bd8d6eca5-kube-api-access-db2rw\") pod \"payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:06:59.480682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.480651 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:06:59.579274 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.579058 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" podUID="58ae2933-d944-463e-b43f-c406dcea7748" containerName="payload-processing" containerID="cri-o://4413da1fb9e9d023f2778ac77c6eec3942af7e05e5158f538c6f28ec384964a4" gracePeriod=25 Oct 30 20:06:59.613250 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.613223 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:06:59.682335 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.682287 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f"] Oct 30 20:06:59.779862 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:06:59.779825 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c69eee1_3140_4420_877f_274bd8d6eca5.slice/crio-c44336861a89baa7b1ec2ff01d500d526bdde57b0ab9a4a2c5f6000b80ccdfa3 WatchSource:0}: Error finding container c44336861a89baa7b1ec2ff01d500d526bdde57b0ab9a4a2c5f6000b80ccdfa3: Status 404 returned error can't find the container with id c44336861a89baa7b1ec2ff01d500d526bdde57b0ab9a4a2c5f6000b80ccdfa3 Oct 30 20:06:59.784133 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.783997 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm"] Oct 30 20:06:59.882054 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.882027 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:06:59.946200 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.946171 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ae2933-d944-463e-b43f-c406dcea7748-kube-api-access-hmhjc\" (UniqueName: \"kubernetes.io/projected/58ae2933-d944-463e-b43f-c406dcea7748-kube-api-access-hmhjc\") pod \"58ae2933-d944-463e-b43f-c406dcea7748\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " Oct 30 20:06:59.946417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.946242 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ae2933-d944-463e-b43f-c406dcea7748-provider-credentials\" (UniqueName: \"kubernetes.io/projected/58ae2933-d944-463e-b43f-c406dcea7748-provider-credentials\") pod \"58ae2933-d944-463e-b43f-c406dcea7748\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " Oct 30 20:06:59.946518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.946480 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ae2933-d944-463e-b43f-c406dcea7748-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58ae2933-d944-463e-b43f-c406dcea7748-routing-overlay-volume\") pod \"58ae2933-d944-463e-b43f-c406dcea7748\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " Oct 30 20:06:59.946638 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.946617 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ae2933-d944-463e-b43f-c406dcea7748-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ae2933-d944-463e-b43f-c406dcea7748-plugins-config-volume\") pod \"58ae2933-d944-463e-b43f-c406dcea7748\" (UID: \"58ae2933-d944-463e-b43f-c406dcea7748\") " Oct 30 20:06:59.947015 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.946924 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ae2933-d944-463e-b43f-c406dcea7748-routing-overlay-volume" pod "58ae2933-d944-463e-b43f-c406dcea7748" (UID: "58ae2933-d944-463e-b43f-c406dcea7748"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:59.947386 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.947325 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ae2933-d944-463e-b43f-c406dcea7748-plugins-config-volume" pod "58ae2933-d944-463e-b43f-c406dcea7748" (UID: "58ae2933-d944-463e-b43f-c406dcea7748"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:06:59.947535 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.947442 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/58ae2933-d944-463e-b43f-c406dcea7748-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:06:59.948933 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.948911 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ae2933-d944-463e-b43f-c406dcea7748-provider-credentials" pod "58ae2933-d944-463e-b43f-c406dcea7748" (UID: "58ae2933-d944-463e-b43f-c406dcea7748"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:06:59.949333 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:06:59.949314 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ae2933-d944-463e-b43f-c406dcea7748-kube-api-access-hmhjc" pod "58ae2933-d944-463e-b43f-c406dcea7748" (UID: "58ae2933-d944-463e-b43f-c406dcea7748"). InnerVolumeSpecName "kube-api-access-hmhjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:00.050344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.050292 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58ae2933-d944-463e-b43f-c406dcea7748-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:00.050585 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.050565 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmhjc\" (UniqueName: \"kubernetes.io/projected/58ae2933-d944-463e-b43f-c406dcea7748-kube-api-access-hmhjc\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:00.050723 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.050709 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/58ae2933-d944-463e-b43f-c406dcea7748-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:00.178899 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.178866 5091 generic.go:356] "Generic (PLEG): container finished" podID="58ae2933-d944-463e-b43f-c406dcea7748" containerID="4413da1fb9e9d023f2778ac77c6eec3942af7e05e5158f538c6f28ec384964a4" exitCode=0 Oct 30 20:07:00.179077 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.178939 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" Oct 30 20:07:00.179077 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.178953 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" event={"ID":"58ae2933-d944-463e-b43f-c406dcea7748","Type":"ContainerDied","Data":"4413da1fb9e9d023f2778ac77c6eec3942af7e05e5158f538c6f28ec384964a4"} Oct 30 20:07:00.179077 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.179027 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt" event={"ID":"58ae2933-d944-463e-b43f-c406dcea7748","Type":"ContainerDied","Data":"8b1009f9a8ab88836f418e91b0b05736c834ffe218b7692ff3fe90d0f016c34c"} Oct 30 20:07:00.179077 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.179059 5091 scope.go:122] "RemoveContainer" containerID="4413da1fb9e9d023f2778ac77c6eec3942af7e05e5158f538c6f28ec384964a4" Oct 30 20:07:00.181218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.181192 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" event={"ID":"2c69eee1-3140-4420-877f-274bd8d6eca5","Type":"ContainerStarted","Data":"157e05032d897cb4fd6da4c0fbdfc31a7bff839abcd4cbf0b68237879ff0b518"} Oct 30 20:07:00.181402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.181376 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:07:00.181477 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.181407 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" event={"ID":"2c69eee1-3140-4420-877f-274bd8d6eca5","Type":"ContainerStarted","Data":"c44336861a89baa7b1ec2ff01d500d526bdde57b0ab9a4a2c5f6000b80ccdfa3"} Oct 30 20:07:00.193695 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.193671 5091 scope.go:122] "RemoveContainer" containerID="4413da1fb9e9d023f2778ac77c6eec3942af7e05e5158f538c6f28ec384964a4" Oct 30 20:07:00.193968 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:07:00.193947 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4413da1fb9e9d023f2778ac77c6eec3942af7e05e5158f538c6f28ec384964a4\": container with ID starting with 4413da1fb9e9d023f2778ac77c6eec3942af7e05e5158f538c6f28ec384964a4 not found: ID does not exist" containerID="4413da1fb9e9d023f2778ac77c6eec3942af7e05e5158f538c6f28ec384964a4" Oct 30 20:07:00.194051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.193979 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4413da1fb9e9d023f2778ac77c6eec3942af7e05e5158f538c6f28ec384964a4"} err="failed to get container status \"4413da1fb9e9d023f2778ac77c6eec3942af7e05e5158f538c6f28ec384964a4\": rpc error: code = NotFound desc = could not find container \"4413da1fb9e9d023f2778ac77c6eec3942af7e05e5158f538c6f28ec384964a4\": container with ID starting with 4413da1fb9e9d023f2778ac77c6eec3942af7e05e5158f538c6f28ec384964a4 not found: ID does not exist" Oct 30 20:07:00.236138 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.236077 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt"] Oct 30 20:07:00.242203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.242175 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58mgmwt"] Oct 30 20:07:00.284821 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.284768 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" podStartSLOduration=2.284752815 podStartE2EDuration="2.284752815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:06:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:00.282530022 +0000 UTC m=+2139.977737983" watchObservedRunningTime="2026-10-30 20:07:00.284752815 +0000 UTC m=+2139.979960773" Oct 30 20:07:00.374662 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.374623 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:00.472133 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.471965 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ae2933-d944-463e-b43f-c406dcea7748" path="/var/lib/kubelet/pods/58ae2933-d944-463e-b43f-c406dcea7748/volumes" Oct 30 20:07:00.621311 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.620866 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:00.681060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.681019 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb"] Oct 30 20:07:00.681907 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.681886 5091 cpu_manager.go:414] "containerMap: removing container" podUID="58ae2933-d944-463e-b43f-c406dcea7748" containerName="payload-processing" Oct 30 20:07:00.681907 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.681906 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ae2933-d944-463e-b43f-c406dcea7748" containerName="payload-processing" Oct 30 20:07:00.682054 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.682044 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ae2933-d944-463e-b43f-c406dcea7748" containerName="payload-processing" Oct 30 20:07:00.687344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.687324 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" Oct 30 20:07:00.696007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.695568 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:07:00.696007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.695614 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:07:00.712975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.712950 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb"] Oct 30 20:07:00.773947 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.773911 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwkwb\" (UniqueName: \"kubernetes.io/projected/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-kube-api-access-rwkwb\") pod \"maas-api-revoke-sub-00b66701-rnqdb\" (UID: \"f5a86cae-73d3-44a4-8a42-c7bcddaf557d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" Oct 30 20:07:00.774230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.774202 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00b66701-rnqdb\" (UID: \"f5a86cae-73d3-44a4-8a42-c7bcddaf557d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" Oct 30 20:07:00.774333 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.774251 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00b66701-rnqdb\" (UID: \"f5a86cae-73d3-44a4-8a42-c7bcddaf557d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" Oct 30 20:07:00.796778 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.796753 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:00.876638 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.876529 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00b66701-rnqdb\" (UID: \"f5a86cae-73d3-44a4-8a42-c7bcddaf557d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" Oct 30 20:07:00.876638 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.876600 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00b66701-rnqdb\" (UID: \"f5a86cae-73d3-44a4-8a42-c7bcddaf557d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" Oct 30 20:07:00.876940 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.876816 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwkwb\" (UniqueName: \"kubernetes.io/projected/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-kube-api-access-rwkwb\") pod \"maas-api-revoke-sub-00b66701-rnqdb\" (UID: \"f5a86cae-73d3-44a4-8a42-c7bcddaf557d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" Oct 30 20:07:00.877800 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.877768 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00b66701-rnqdb\" (UID: \"f5a86cae-73d3-44a4-8a42-c7bcddaf557d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" Oct 30 20:07:00.913094 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.913059 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00b66701-rnqdb\" (UID: \"f5a86cae-73d3-44a4-8a42-c7bcddaf557d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" Oct 30 20:07:00.945904 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:00.945862 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwkwb\" (UniqueName: \"kubernetes.io/projected/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-kube-api-access-rwkwb\") pod \"maas-api-revoke-sub-00b66701-rnqdb\" (UID: \"f5a86cae-73d3-44a4-8a42-c7bcddaf557d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" Oct 30 20:07:01.006175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.005304 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" Oct 30 20:07:01.125164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.121790 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt"] Oct 30 20:07:01.185219 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.180896 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" Oct 30 20:07:01.221429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.221392 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcxfp\" (UniqueName: \"kubernetes.io/projected/f10798fc-4b1c-464d-88a0-0849293cf3ec-kube-api-access-fcxfp\") pod \"maas-api-revoke-sub-081b6330-h66dt\" (UID: \"f10798fc-4b1c-464d-88a0-0849293cf3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" Oct 30 20:07:01.221865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.221844 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10798fc-4b1c-464d-88a0-0849293cf3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-081b6330-h66dt\" (UID: \"f10798fc-4b1c-464d-88a0-0849293cf3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" Oct 30 20:07:01.222042 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.222028 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10798fc-4b1c-464d-88a0-0849293cf3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-081b6330-h66dt\" (UID: \"f10798fc-4b1c-464d-88a0-0849293cf3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" Oct 30 20:07:01.281412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.281380 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt"] Oct 30 20:07:01.391172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.333537 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10798fc-4b1c-464d-88a0-0849293cf3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-081b6330-h66dt\" (UID: \"f10798fc-4b1c-464d-88a0-0849293cf3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" Oct 30 20:07:01.391172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.333670 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10798fc-4b1c-464d-88a0-0849293cf3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-081b6330-h66dt\" (UID: \"f10798fc-4b1c-464d-88a0-0849293cf3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" Oct 30 20:07:01.391172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.334031 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcxfp\" (UniqueName: \"kubernetes.io/projected/f10798fc-4b1c-464d-88a0-0849293cf3ec-kube-api-access-fcxfp\") pod \"maas-api-revoke-sub-081b6330-h66dt\" (UID: \"f10798fc-4b1c-464d-88a0-0849293cf3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" Oct 30 20:07:01.391172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.343211 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10798fc-4b1c-464d-88a0-0849293cf3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-081b6330-h66dt\" (UID: \"f10798fc-4b1c-464d-88a0-0849293cf3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" Oct 30 20:07:01.391172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.343569 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10798fc-4b1c-464d-88a0-0849293cf3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-081b6330-h66dt\" (UID: \"f10798fc-4b1c-464d-88a0-0849293cf3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" Oct 30 20:07:01.391172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.365432 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd"] Oct 30 20:07:01.454308 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.454273 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-22a98f80-brltd"] Oct 30 20:07:01.458613 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.458578 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcxfp\" (UniqueName: \"kubernetes.io/projected/f10798fc-4b1c-464d-88a0-0849293cf3ec-kube-api-access-fcxfp\") pod \"maas-api-revoke-sub-081b6330-h66dt\" (UID: \"f10798fc-4b1c-464d-88a0-0849293cf3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" Oct 30 20:07:01.461517 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.461493 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" Oct 30 20:07:01.590403 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.590368 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:01.706759 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:07:01.706723 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5a86cae_73d3_44a4_8a42_c7bcddaf557d.slice/crio-198c7712698a31df07c824b4a06f7af1773625c280c2c583814dc41ef93ec94a WatchSource:0}: Error finding container 198c7712698a31df07c824b4a06f7af1773625c280c2c583814dc41ef93ec94a: Status 404 returned error can't find the container with id 198c7712698a31df07c824b4a06f7af1773625c280c2c583814dc41ef93ec94a Oct 30 20:07:01.710194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.710166 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb"] Oct 30 20:07:01.873674 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:07:01.873640 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf10798fc_4b1c_464d_88a0_0849293cf3ec.slice/crio-5b758d69ff3cb381164f502dc0b943681f2265dfeff229abb6aa1d57a678e3a5 WatchSource:0}: Error finding container 5b758d69ff3cb381164f502dc0b943681f2265dfeff229abb6aa1d57a678e3a5: Status 404 returned error can't find the container with id 5b758d69ff3cb381164f502dc0b943681f2265dfeff229abb6aa1d57a678e3a5 Oct 30 20:07:01.874872 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:01.874844 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt"] Oct 30 20:07:02.235714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:02.235637 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" event={"ID":"f10798fc-4b1c-464d-88a0-0849293cf3ec","Type":"ContainerStarted","Data":"5b758d69ff3cb381164f502dc0b943681f2265dfeff229abb6aa1d57a678e3a5"} Oct 30 20:07:02.237445 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:02.237410 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" event={"ID":"f5a86cae-73d3-44a4-8a42-c7bcddaf557d","Type":"ContainerStarted","Data":"6e63b4367765a78312b10a1593e9e9b85154afbbcf75160a5fde7406a8302002"} Oct 30 20:07:02.237571 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:02.237461 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" event={"ID":"f5a86cae-73d3-44a4-8a42-c7bcddaf557d","Type":"ContainerStarted","Data":"198c7712698a31df07c824b4a06f7af1773625c280c2c583814dc41ef93ec94a"} Oct 30 20:07:02.262331 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:02.262282 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" podStartSLOduration=2.262268456 podStartE2EDuration="2.262268456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:02.258855728 +0000 UTC m=+2141.954063689" watchObservedRunningTime="2026-10-30 20:07:02.262268456 +0000 UTC m=+2141.957476414" Oct 30 20:07:02.444699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:02.444667 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb262571-85ce-4f55-a2db-86068388b03f" path="/var/lib/kubelet/pods/eb262571-85ce-4f55-a2db-86068388b03f/volumes" Oct 30 20:07:02.585183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:02.580064 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:03.235124 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:03.235026 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs"] Oct 30 20:07:03.252225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:03.252185 5091 generic.go:356] "Generic (PLEG): container finished" podID="f10798fc-4b1c-464d-88a0-0849293cf3ec" containerID="f068d12cc015ffc759062dc68e539004f98e04cabd12fd143aeb880ff7045ee4" exitCode=0 Oct 30 20:07:03.252394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:03.252287 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" event={"ID":"f10798fc-4b1c-464d-88a0-0849293cf3ec","Type":"ContainerDied","Data":"f068d12cc015ffc759062dc68e539004f98e04cabd12fd143aeb880ff7045ee4"} Oct 30 20:07:03.254308 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:03.254274 5091 generic.go:356] "Generic (PLEG): container finished" podID="f5a86cae-73d3-44a4-8a42-c7bcddaf557d" containerID="6e63b4367765a78312b10a1593e9e9b85154afbbcf75160a5fde7406a8302002" exitCode=0 Oct 30 20:07:03.254428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:03.254369 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" event={"ID":"f5a86cae-73d3-44a4-8a42-c7bcddaf557d","Type":"ContainerDied","Data":"6e63b4367765a78312b10a1593e9e9b85154afbbcf75160a5fde7406a8302002"} Oct 30 20:07:03.305318 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:07:03.305066 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-68685b/agc-cred-8f19d92ff4bd: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:07:03.305318 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:07:03.305098 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:07:03.305318 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:07:03.305171 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials podName:6edd28d3-81f6-4be5-807c-2616e250eebd nodeName:}" failed. No retries permitted until 2026-10-30 20:07:19.305138922 +0000 UTC m=+2159.000346865 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" (UID: "6edd28d3-81f6-4be5-807c-2616e250eebd") : secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:07:03.575501 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:03.575458 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:04.498824 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.498795 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" Oct 30 20:07:04.502571 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.502547 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" Oct 30 20:07:04.606388 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.604887 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:04.627428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.623748 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-maas-api-openshift-ca\") pod \"f5a86cae-73d3-44a4-8a42-c7bcddaf557d\" (UID: \"f5a86cae-73d3-44a4-8a42-c7bcddaf557d\") " Oct 30 20:07:04.627428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.623867 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f10798fc-4b1c-464d-88a0-0849293cf3ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10798fc-4b1c-464d-88a0-0849293cf3ec-maas-api-openshift-ca\") pod \"f10798fc-4b1c-464d-88a0-0849293cf3ec\" (UID: \"f10798fc-4b1c-464d-88a0-0849293cf3ec\") " Oct 30 20:07:04.627428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.624028 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f10798fc-4b1c-464d-88a0-0849293cf3ec-kube-api-access-fcxfp\" (UniqueName: \"kubernetes.io/projected/f10798fc-4b1c-464d-88a0-0849293cf3ec-kube-api-access-fcxfp\") pod \"f10798fc-4b1c-464d-88a0-0849293cf3ec\" (UID: \"f10798fc-4b1c-464d-88a0-0849293cf3ec\") " Oct 30 20:07:04.627428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.624106 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-maas-api-xks-ca\") pod \"f5a86cae-73d3-44a4-8a42-c7bcddaf557d\" (UID: \"f5a86cae-73d3-44a4-8a42-c7bcddaf557d\") " Oct 30 20:07:04.627428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.624236 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-kube-api-access-rwkwb\" (UniqueName: \"kubernetes.io/projected/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-kube-api-access-rwkwb\") pod \"f5a86cae-73d3-44a4-8a42-c7bcddaf557d\" (UID: \"f5a86cae-73d3-44a4-8a42-c7bcddaf557d\") " Oct 30 20:07:04.627428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.624325 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f10798fc-4b1c-464d-88a0-0849293cf3ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10798fc-4b1c-464d-88a0-0849293cf3ec-maas-api-xks-ca\") pod \"f10798fc-4b1c-464d-88a0-0849293cf3ec\" (UID: \"f10798fc-4b1c-464d-88a0-0849293cf3ec\") " Oct 30 20:07:04.627428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.625596 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f10798fc-4b1c-464d-88a0-0849293cf3ec-maas-api-openshift-ca" pod "f10798fc-4b1c-464d-88a0-0849293cf3ec" (UID: "f10798fc-4b1c-464d-88a0-0849293cf3ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:04.627428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.625776 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-maas-api-openshift-ca" pod "f5a86cae-73d3-44a4-8a42-c7bcddaf557d" (UID: "f5a86cae-73d3-44a4-8a42-c7bcddaf557d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:04.627428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.626872 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:04.627428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.626895 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f10798fc-4b1c-464d-88a0-0849293cf3ec-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:04.630484 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.630454 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-kube-api-access-rwkwb" pod "f5a86cae-73d3-44a4-8a42-c7bcddaf557d" (UID: "f5a86cae-73d3-44a4-8a42-c7bcddaf557d"). InnerVolumeSpecName "kube-api-access-rwkwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:04.631958 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.631934 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f10798fc-4b1c-464d-88a0-0849293cf3ec-kube-api-access-fcxfp" pod "f10798fc-4b1c-464d-88a0-0849293cf3ec" (UID: "f10798fc-4b1c-464d-88a0-0849293cf3ec"). InnerVolumeSpecName "kube-api-access-fcxfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:04.632766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.632742 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-maas-api-xks-ca" pod "f5a86cae-73d3-44a4-8a42-c7bcddaf557d" (UID: "f5a86cae-73d3-44a4-8a42-c7bcddaf557d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:04.632860 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.632824 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f10798fc-4b1c-464d-88a0-0849293cf3ec-maas-api-xks-ca" pod "f10798fc-4b1c-464d-88a0-0849293cf3ec" (UID: "f10798fc-4b1c-464d-88a0-0849293cf3ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:04.683575 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.683540 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" podUID="6edd28d3-81f6-4be5-807c-2616e250eebd" containerName="payload-processing" containerID="cri-o://06e32f8f523e5719f75738b78c7316db15d64d88a4e524430e67f375a21b6dde" gracePeriod=25 Oct 30 20:07:04.728569 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.728536 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcxfp\" (UniqueName: \"kubernetes.io/projected/f10798fc-4b1c-464d-88a0-0849293cf3ec-kube-api-access-fcxfp\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:04.728569 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.728568 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:04.728741 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.728579 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwkwb\" (UniqueName: \"kubernetes.io/projected/f5a86cae-73d3-44a4-8a42-c7bcddaf557d-kube-api-access-rwkwb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:04.728741 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.728588 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f10798fc-4b1c-464d-88a0-0849293cf3ec-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:04.962063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:04.962033 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:07:05.032716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.032676 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6edd28d3-81f6-4be5-807c-2616e250eebd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6edd28d3-81f6-4be5-807c-2616e250eebd-routing-overlay-volume\") pod \"6edd28d3-81f6-4be5-807c-2616e250eebd\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " Oct 30 20:07:05.032932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.032817 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials\") pod \"6edd28d3-81f6-4be5-807c-2616e250eebd\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " Oct 30 20:07:05.032932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.032884 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-kube-api-access-jr67p\" (UniqueName: \"kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-kube-api-access-jr67p\") pod \"6edd28d3-81f6-4be5-807c-2616e250eebd\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " Oct 30 20:07:05.033134 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.033109 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6edd28d3-81f6-4be5-807c-2616e250eebd-routing-overlay-volume" pod "6edd28d3-81f6-4be5-807c-2616e250eebd" (UID: "6edd28d3-81f6-4be5-807c-2616e250eebd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:05.033134 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.033126 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6edd28d3-81f6-4be5-807c-2616e250eebd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6edd28d3-81f6-4be5-807c-2616e250eebd-plugins-config-volume\") pod \"6edd28d3-81f6-4be5-807c-2616e250eebd\" (UID: \"6edd28d3-81f6-4be5-807c-2616e250eebd\") " Oct 30 20:07:05.033691 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.033658 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6edd28d3-81f6-4be5-807c-2616e250eebd-plugins-config-volume" pod "6edd28d3-81f6-4be5-807c-2616e250eebd" (UID: "6edd28d3-81f6-4be5-807c-2616e250eebd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:05.033886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.033862 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6edd28d3-81f6-4be5-807c-2616e250eebd-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:05.033976 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.033891 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6edd28d3-81f6-4be5-807c-2616e250eebd-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:05.035568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.035538 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-kube-api-access-jr67p" pod "6edd28d3-81f6-4be5-807c-2616e250eebd" (UID: "6edd28d3-81f6-4be5-807c-2616e250eebd"). InnerVolumeSpecName "kube-api-access-jr67p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:05.036181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.036128 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials" pod "6edd28d3-81f6-4be5-807c-2616e250eebd" (UID: "6edd28d3-81f6-4be5-807c-2616e250eebd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:05.135538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.135445 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:05.135538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.135479 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr67p\" (UniqueName: \"kubernetes.io/projected/6edd28d3-81f6-4be5-807c-2616e250eebd-kube-api-access-jr67p\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:05.275794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.275759 5091 generic.go:356] "Generic (PLEG): container finished" podID="6edd28d3-81f6-4be5-807c-2616e250eebd" containerID="06e32f8f523e5719f75738b78c7316db15d64d88a4e524430e67f375a21b6dde" exitCode=0 Oct 30 20:07:05.275991 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.275810 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" Oct 30 20:07:05.275991 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.275844 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" event={"ID":"6edd28d3-81f6-4be5-807c-2616e250eebd","Type":"ContainerDied","Data":"06e32f8f523e5719f75738b78c7316db15d64d88a4e524430e67f375a21b6dde"} Oct 30 20:07:05.275991 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.275898 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f" event={"ID":"6edd28d3-81f6-4be5-807c-2616e250eebd","Type":"ContainerDied","Data":"f1c113c4d22426df5df5938ad8ae763d58e78498b9ae14db67d904c9e5ba8568"} Oct 30 20:07:05.275991 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.275928 5091 scope.go:122] "RemoveContainer" containerID="06e32f8f523e5719f75738b78c7316db15d64d88a4e524430e67f375a21b6dde" Oct 30 20:07:05.277825 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.277803 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" Oct 30 20:07:05.277934 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.277804 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt" event={"ID":"f10798fc-4b1c-464d-88a0-0849293cf3ec","Type":"ContainerDied","Data":"5b758d69ff3cb381164f502dc0b943681f2265dfeff229abb6aa1d57a678e3a5"} Oct 30 20:07:05.278003 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.277944 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b758d69ff3cb381164f502dc0b943681f2265dfeff229abb6aa1d57a678e3a5" Oct 30 20:07:05.279554 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.279533 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" Oct 30 20:07:05.279659 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.279564 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb" event={"ID":"f5a86cae-73d3-44a4-8a42-c7bcddaf557d","Type":"ContainerDied","Data":"198c7712698a31df07c824b4a06f7af1773625c280c2c583814dc41ef93ec94a"} Oct 30 20:07:05.279659 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.279608 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="198c7712698a31df07c824b4a06f7af1773625c280c2c583814dc41ef93ec94a" Oct 30 20:07:05.291748 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.291729 5091 scope.go:122] "RemoveContainer" containerID="06e32f8f523e5719f75738b78c7316db15d64d88a4e524430e67f375a21b6dde" Oct 30 20:07:05.292040 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:07:05.292018 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e32f8f523e5719f75738b78c7316db15d64d88a4e524430e67f375a21b6dde\": container with ID starting with 06e32f8f523e5719f75738b78c7316db15d64d88a4e524430e67f375a21b6dde not found: ID does not exist" containerID="06e32f8f523e5719f75738b78c7316db15d64d88a4e524430e67f375a21b6dde" Oct 30 20:07:05.292121 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.292048 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e32f8f523e5719f75738b78c7316db15d64d88a4e524430e67f375a21b6dde"} err="failed to get container status \"06e32f8f523e5719f75738b78c7316db15d64d88a4e524430e67f375a21b6dde\": rpc error: code = NotFound desc = could not find container \"06e32f8f523e5719f75738b78c7316db15d64d88a4e524430e67f375a21b6dde\": container with ID starting with 06e32f8f523e5719f75738b78c7316db15d64d88a4e524430e67f375a21b6dde not found: ID does not exist" Oct 30 20:07:05.325600 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.325568 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f"] Oct 30 20:07:05.330190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.330167 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4v8m5f"] Oct 30 20:07:05.596814 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:05.596776 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:06.032991 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:06.032957 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz"] Oct 30 20:07:06.040051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:06.040023 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5444ffaa-wffcz"] Oct 30 20:07:06.447273 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:06.447194 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6edd28d3-81f6-4be5-807c-2616e250eebd" path="/var/lib/kubelet/pods/6edd28d3-81f6-4be5-807c-2616e250eebd/volumes" Oct 30 20:07:06.447907 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:06.447887 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c9ff71-ad8e-4eab-8950-8f1034481710" path="/var/lib/kubelet/pods/c2c9ff71-ad8e-4eab-8950-8f1034481710/volumes" Oct 30 20:07:06.522185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:06.519715 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:07.374199 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:07.365824 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:08.181894 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:08.181857 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:08.236890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:08.236843 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" podUID="e7491db9-6f6b-4f3b-bddf-397c96e2ac87" containerName="payload-processing" containerID="cri-o://b03d119c8711a9d300a36fe7082edb461633459226ef36276c45e65a4f07e606" gracePeriod=25 Oct 30 20:07:08.535225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:08.535198 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:07:08.637528 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:08.637483 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-routing-overlay-volume\") pod \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " Oct 30 20:07:08.637713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:08.637622 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-kube-api-access-6qkj7\" (UniqueName: \"kubernetes.io/projected/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-kube-api-access-6qkj7\") pod \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " Oct 30 20:07:08.637787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:08.637721 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-plugins-config-volume\") pod \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " Oct 30 20:07:08.637883 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:08.637863 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-provider-credentials\") pod \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\" (UID: \"e7491db9-6f6b-4f3b-bddf-397c96e2ac87\") " Oct 30 20:07:08.637944 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:08.637916 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-routing-overlay-volume" pod "e7491db9-6f6b-4f3b-bddf-397c96e2ac87" (UID: "e7491db9-6f6b-4f3b-bddf-397c96e2ac87"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:08.638528 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:08.638493 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-plugins-config-volume" pod "e7491db9-6f6b-4f3b-bddf-397c96e2ac87" (UID: "e7491db9-6f6b-4f3b-bddf-397c96e2ac87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:08.638887 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:08.638865 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:08.639048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:08.638892 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:08.640494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:08.640461 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-kube-api-access-6qkj7" pod "e7491db9-6f6b-4f3b-bddf-397c96e2ac87" (UID: "e7491db9-6f6b-4f3b-bddf-397c96e2ac87"). InnerVolumeSpecName "kube-api-access-6qkj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:08.640704 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:08.640684 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-provider-credentials" pod "e7491db9-6f6b-4f3b-bddf-397c96e2ac87" (UID: "e7491db9-6f6b-4f3b-bddf-397c96e2ac87"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:08.741031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:08.740996 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qkj7\" (UniqueName: \"kubernetes.io/projected/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-kube-api-access-6qkj7\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:08.741031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:08.741027 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/e7491db9-6f6b-4f3b-bddf-397c96e2ac87-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:09.009287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:09.009248 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:09.332402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:09.332308 5091 generic.go:356] "Generic (PLEG): container finished" podID="e7491db9-6f6b-4f3b-bddf-397c96e2ac87" containerID="b03d119c8711a9d300a36fe7082edb461633459226ef36276c45e65a4f07e606" exitCode=0 Oct 30 20:07:09.332402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:09.332366 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" Oct 30 20:07:09.332644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:09.332395 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" event={"ID":"e7491db9-6f6b-4f3b-bddf-397c96e2ac87","Type":"ContainerDied","Data":"b03d119c8711a9d300a36fe7082edb461633459226ef36276c45e65a4f07e606"} Oct 30 20:07:09.332644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:09.332453 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs" event={"ID":"e7491db9-6f6b-4f3b-bddf-397c96e2ac87","Type":"ContainerDied","Data":"3022ae9d1e4fa3ebcf71bc28ee1d0e09db0375e86119210676474b2e878b7cfe"} Oct 30 20:07:09.332644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:09.332477 5091 scope.go:122] "RemoveContainer" containerID="b03d119c8711a9d300a36fe7082edb461633459226ef36276c45e65a4f07e606" Oct 30 20:07:09.345456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:09.345438 5091 scope.go:122] "RemoveContainer" containerID="b03d119c8711a9d300a36fe7082edb461633459226ef36276c45e65a4f07e606" Oct 30 20:07:09.345754 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:07:09.345726 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03d119c8711a9d300a36fe7082edb461633459226ef36276c45e65a4f07e606\": container with ID starting with b03d119c8711a9d300a36fe7082edb461633459226ef36276c45e65a4f07e606 not found: ID does not exist" containerID="b03d119c8711a9d300a36fe7082edb461633459226ef36276c45e65a4f07e606" Oct 30 20:07:09.345807 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:09.345765 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03d119c8711a9d300a36fe7082edb461633459226ef36276c45e65a4f07e606"} err="failed to get container status \"b03d119c8711a9d300a36fe7082edb461633459226ef36276c45e65a4f07e606\": rpc error: code = NotFound desc = could not find container \"b03d119c8711a9d300a36fe7082edb461633459226ef36276c45e65a4f07e606\": container with ID starting with b03d119c8711a9d300a36fe7082edb461633459226ef36276c45e65a4f07e606 not found: ID does not exist" Oct 30 20:07:09.393795 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:09.393767 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs"] Oct 30 20:07:09.398049 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:09.398025 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-969878/payload-processing-external-model-e2e-extmod-b-969878-6888z87hs"] Oct 30 20:07:09.980174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:09.978223 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:10.443300 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:10.443269 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7491db9-6f6b-4f3b-bddf-397c96e2ac87" path="/var/lib/kubelet/pods/e7491db9-6f6b-4f3b-bddf-397c96e2ac87/volumes" Oct 30 20:07:10.996219 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:10.995666 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:11.048378 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:11.048341 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx"] Oct 30 20:07:11.053803 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:11.053768 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-784a0698-tdjjx"] Oct 30 20:07:11.229861 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:11.229825 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:07:11.938848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:11.930298 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:12.117737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.117705 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r"] Oct 30 20:07:12.118724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.118700 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f5a86cae-73d3-44a4-8a42-c7bcddaf557d" containerName="revoke-keys" Oct 30 20:07:12.118724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.118725 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a86cae-73d3-44a4-8a42-c7bcddaf557d" containerName="revoke-keys" Oct 30 20:07:12.118926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.118781 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e7491db9-6f6b-4f3b-bddf-397c96e2ac87" containerName="payload-processing" Oct 30 20:07:12.118926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.118790 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7491db9-6f6b-4f3b-bddf-397c96e2ac87" containerName="payload-processing" Oct 30 20:07:12.118926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.118818 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f10798fc-4b1c-464d-88a0-0849293cf3ec" containerName="revoke-keys" Oct 30 20:07:12.118926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.118826 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f10798fc-4b1c-464d-88a0-0849293cf3ec" containerName="revoke-keys" Oct 30 20:07:12.118926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.118839 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6edd28d3-81f6-4be5-807c-2616e250eebd" containerName="payload-processing" Oct 30 20:07:12.118926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.118847 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6edd28d3-81f6-4be5-807c-2616e250eebd" containerName="payload-processing" Oct 30 20:07:12.119312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.119032 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a86cae-73d3-44a4-8a42-c7bcddaf557d" containerName="revoke-keys" Oct 30 20:07:12.119312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.119057 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7491db9-6f6b-4f3b-bddf-397c96e2ac87" containerName="payload-processing" Oct 30 20:07:12.119312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.119074 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f10798fc-4b1c-464d-88a0-0849293cf3ec" containerName="revoke-keys" Oct 30 20:07:12.119312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.119085 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6edd28d3-81f6-4be5-807c-2616e250eebd" containerName="payload-processing" Oct 30 20:07:12.125718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.125584 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" Oct 30 20:07:12.134573 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.134546 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:07:12.134699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.134676 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:07:12.141579 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.141550 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r"] Oct 30 20:07:12.215415 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.215333 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq9rz\" (UniqueName: \"kubernetes.io/projected/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-kube-api-access-sq9rz\") pod \"maas-api-revoke-sub-059fd74b-hsq5r\" (UID: \"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" Oct 30 20:07:12.215563 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.215534 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-059fd74b-hsq5r\" (UID: \"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" Oct 30 20:07:12.215682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.215668 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-059fd74b-hsq5r\" (UID: \"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" Oct 30 20:07:12.317135 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.317099 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq9rz\" (UniqueName: \"kubernetes.io/projected/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-kube-api-access-sq9rz\") pod \"maas-api-revoke-sub-059fd74b-hsq5r\" (UID: \"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" Oct 30 20:07:12.317346 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.317322 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-059fd74b-hsq5r\" (UID: \"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" Oct 30 20:07:12.317447 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.317434 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-059fd74b-hsq5r\" (UID: \"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" Oct 30 20:07:12.318258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.318231 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-059fd74b-hsq5r\" (UID: \"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" Oct 30 20:07:12.320007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.319976 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-059fd74b-hsq5r\" (UID: \"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" Oct 30 20:07:12.332529 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.332502 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq9rz\" (UniqueName: \"kubernetes.io/projected/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-kube-api-access-sq9rz\") pod \"maas-api-revoke-sub-059fd74b-hsq5r\" (UID: \"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" Oct 30 20:07:12.443207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.443177 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="348fb41b-1181-40c8-9193-9bbcfa3ea9a1" path="/var/lib/kubelet/pods/348fb41b-1181-40c8-9193-9bbcfa3ea9a1/volumes" Oct 30 20:07:12.453559 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.453531 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" Oct 30 20:07:12.756637 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:12.756595 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r"] Oct 30 20:07:12.757723 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:07:12.757691 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf848a7a0_fafe_4b2b_b8f7_e97ecc993e37.slice/crio-cc66b61a78e6fc27a7aaca8bcc79ba0e45635a5554f5b9a280816372671d110e WatchSource:0}: Error finding container cc66b61a78e6fc27a7aaca8bcc79ba0e45635a5554f5b9a280816372671d110e: Status 404 returned error can't find the container with id cc66b61a78e6fc27a7aaca8bcc79ba0e45635a5554f5b9a280816372671d110e Oct 30 20:07:13.004389 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.004300 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:13.187213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.187163 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj"] Oct 30 20:07:13.192068 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.192049 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" Oct 30 20:07:13.208280 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.208247 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj"] Oct 30 20:07:13.336432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.336401 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrscm\" (UniqueName: \"kubernetes.io/projected/997a1e67-2b6d-4322-b333-04eb9b498bd3-kube-api-access-lrscm\") pod \"maas-api-revoke-sub-13a45d69-4ljnj\" (UID: \"997a1e67-2b6d-4322-b333-04eb9b498bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" Oct 30 20:07:13.336589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.336446 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997a1e67-2b6d-4322-b333-04eb9b498bd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a45d69-4ljnj\" (UID: \"997a1e67-2b6d-4322-b333-04eb9b498bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" Oct 30 20:07:13.336589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.336554 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997a1e67-2b6d-4322-b333-04eb9b498bd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a45d69-4ljnj\" (UID: \"997a1e67-2b6d-4322-b333-04eb9b498bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" Oct 30 20:07:13.382774 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.382736 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" event={"ID":"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37","Type":"ContainerStarted","Data":"4194747c3e019e86a87163b95c772085ddcf60063ab4de329f6f3f775b1b7f96"} Oct 30 20:07:13.382933 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.382800 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" event={"ID":"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37","Type":"ContainerStarted","Data":"cc66b61a78e6fc27a7aaca8bcc79ba0e45635a5554f5b9a280816372671d110e"} Oct 30 20:07:13.405974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.405922 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" podStartSLOduration=1.4059091590000001 podStartE2EDuration="1.405909159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:13.404497591 +0000 UTC m=+2153.099705550" watchObservedRunningTime="2026-10-30 20:07:13.405909159 +0000 UTC m=+2153.101117117" Oct 30 20:07:13.437999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.437968 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrscm\" (UniqueName: \"kubernetes.io/projected/997a1e67-2b6d-4322-b333-04eb9b498bd3-kube-api-access-lrscm\") pod \"maas-api-revoke-sub-13a45d69-4ljnj\" (UID: \"997a1e67-2b6d-4322-b333-04eb9b498bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" Oct 30 20:07:13.438217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.438012 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997a1e67-2b6d-4322-b333-04eb9b498bd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a45d69-4ljnj\" (UID: \"997a1e67-2b6d-4322-b333-04eb9b498bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" Oct 30 20:07:13.438217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.438080 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997a1e67-2b6d-4322-b333-04eb9b498bd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a45d69-4ljnj\" (UID: \"997a1e67-2b6d-4322-b333-04eb9b498bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" Oct 30 20:07:13.438838 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.438813 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997a1e67-2b6d-4322-b333-04eb9b498bd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13a45d69-4ljnj\" (UID: \"997a1e67-2b6d-4322-b333-04eb9b498bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" Oct 30 20:07:13.440681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.440651 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997a1e67-2b6d-4322-b333-04eb9b498bd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13a45d69-4ljnj\" (UID: \"997a1e67-2b6d-4322-b333-04eb9b498bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" Oct 30 20:07:13.452908 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.452886 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrscm\" (UniqueName: \"kubernetes.io/projected/997a1e67-2b6d-4322-b333-04eb9b498bd3-kube-api-access-lrscm\") pod \"maas-api-revoke-sub-13a45d69-4ljnj\" (UID: \"997a1e67-2b6d-4322-b333-04eb9b498bd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" Oct 30 20:07:13.510191 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.510132 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" Oct 30 20:07:13.808394 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:07:13.808357 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod997a1e67_2b6d_4322_b333_04eb9b498bd3.slice/crio-2a17a0deb93da2c1cba7fa32d4e4100cd17c2ebea94c140c7c832b5e78075d91 WatchSource:0}: Error finding container 2a17a0deb93da2c1cba7fa32d4e4100cd17c2ebea94c140c7c832b5e78075d91: Status 404 returned error can't find the container with id 2a17a0deb93da2c1cba7fa32d4e4100cd17c2ebea94c140c7c832b5e78075d91 Oct 30 20:07:13.809496 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:13.809470 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj"] Oct 30 20:07:14.019171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:14.019069 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:14.223364 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:14.223328 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:14.395220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:14.395183 5091 generic.go:356] "Generic (PLEG): container finished" podID="f848a7a0-fafe-4b2b-b8f7-e97ecc993e37" containerID="4194747c3e019e86a87163b95c772085ddcf60063ab4de329f6f3f775b1b7f96" exitCode=0 Oct 30 20:07:14.395395 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:14.395274 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" event={"ID":"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37","Type":"ContainerDied","Data":"4194747c3e019e86a87163b95c772085ddcf60063ab4de329f6f3f775b1b7f96"} Oct 30 20:07:14.397161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:14.397114 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" event={"ID":"997a1e67-2b6d-4322-b333-04eb9b498bd3","Type":"ContainerStarted","Data":"59011d31bdbeb2af7a8742305171428df448ece644feb3832c2b1b30b77302a8"} Oct 30 20:07:14.397288 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:14.397185 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" event={"ID":"997a1e67-2b6d-4322-b333-04eb9b498bd3","Type":"ContainerStarted","Data":"2a17a0deb93da2c1cba7fa32d4e4100cd17c2ebea94c140c7c832b5e78075d91"} Oct 30 20:07:14.453600 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:14.453503 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" podStartSLOduration=1.453491754 podStartE2EDuration="1.453491754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:14.446024411 +0000 UTC m=+2154.141232368" watchObservedRunningTime="2026-10-30 20:07:14.453491754 +0000 UTC m=+2154.148699711" Oct 30 20:07:14.950201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:14.948349 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:15.407051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:15.407016 5091 generic.go:356] "Generic (PLEG): container finished" podID="997a1e67-2b6d-4322-b333-04eb9b498bd3" containerID="59011d31bdbeb2af7a8742305171428df448ece644feb3832c2b1b30b77302a8" exitCode=0 Oct 30 20:07:15.407526 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:15.407100 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" event={"ID":"997a1e67-2b6d-4322-b333-04eb9b498bd3","Type":"ContainerDied","Data":"59011d31bdbeb2af7a8742305171428df448ece644feb3832c2b1b30b77302a8"} Oct 30 20:07:15.609867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:15.609840 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" Oct 30 20:07:15.673977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:15.673903 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-kube-api-access-sq9rz\" (UniqueName: \"kubernetes.io/projected/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-kube-api-access-sq9rz\") pod \"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37\" (UID: \"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37\") " Oct 30 20:07:15.674134 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:15.674091 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-maas-api-xks-ca\") pod \"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37\" (UID: \"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37\") " Oct 30 20:07:15.674134 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:15.674129 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-maas-api-openshift-ca\") pod \"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37\" (UID: \"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37\") " Oct 30 20:07:15.674900 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:15.674715 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-maas-api-openshift-ca" pod "f848a7a0-fafe-4b2b-b8f7-e97ecc993e37" (UID: "f848a7a0-fafe-4b2b-b8f7-e97ecc993e37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:15.674900 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:15.674869 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:15.678288 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:15.678259 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-kube-api-access-sq9rz" pod "f848a7a0-fafe-4b2b-b8f7-e97ecc993e37" (UID: "f848a7a0-fafe-4b2b-b8f7-e97ecc993e37"). InnerVolumeSpecName "kube-api-access-sq9rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:15.678528 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:15.678510 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-maas-api-xks-ca" pod "f848a7a0-fafe-4b2b-b8f7-e97ecc993e37" (UID: "f848a7a0-fafe-4b2b-b8f7-e97ecc993e37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:15.776358 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:15.776322 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq9rz\" (UniqueName: \"kubernetes.io/projected/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-kube-api-access-sq9rz\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:15.776358 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:15.776355 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:15.970986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:15.970901 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:16.044594 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:07:16.044557 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:07:16.418215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:16.418181 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" Oct 30 20:07:16.418645 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:16.418181 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r" event={"ID":"f848a7a0-fafe-4b2b-b8f7-e97ecc993e37","Type":"ContainerDied","Data":"cc66b61a78e6fc27a7aaca8bcc79ba0e45635a5554f5b9a280816372671d110e"} Oct 30 20:07:16.418645 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:16.418308 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc66b61a78e6fc27a7aaca8bcc79ba0e45635a5554f5b9a280816372671d110e" Oct 30 20:07:16.620416 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:16.620391 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" Oct 30 20:07:16.704253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:16.704117 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/997a1e67-2b6d-4322-b333-04eb9b498bd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997a1e67-2b6d-4322-b333-04eb9b498bd3-maas-api-xks-ca\") pod \"997a1e67-2b6d-4322-b333-04eb9b498bd3\" (UID: \"997a1e67-2b6d-4322-b333-04eb9b498bd3\") " Oct 30 20:07:16.704442 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:16.704270 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/997a1e67-2b6d-4322-b333-04eb9b498bd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997a1e67-2b6d-4322-b333-04eb9b498bd3-maas-api-openshift-ca\") pod \"997a1e67-2b6d-4322-b333-04eb9b498bd3\" (UID: \"997a1e67-2b6d-4322-b333-04eb9b498bd3\") " Oct 30 20:07:16.704442 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:16.704310 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997a1e67-2b6d-4322-b333-04eb9b498bd3-kube-api-access-lrscm\" (UniqueName: \"kubernetes.io/projected/997a1e67-2b6d-4322-b333-04eb9b498bd3-kube-api-access-lrscm\") pod \"997a1e67-2b6d-4322-b333-04eb9b498bd3\" (UID: \"997a1e67-2b6d-4322-b333-04eb9b498bd3\") " Oct 30 20:07:16.704718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:16.704687 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/997a1e67-2b6d-4322-b333-04eb9b498bd3-maas-api-openshift-ca" pod "997a1e67-2b6d-4322-b333-04eb9b498bd3" (UID: "997a1e67-2b6d-4322-b333-04eb9b498bd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:16.705412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:16.705375 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/997a1e67-2b6d-4322-b333-04eb9b498bd3-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:16.706821 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:16.706796 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997a1e67-2b6d-4322-b333-04eb9b498bd3-maas-api-xks-ca" pod "997a1e67-2b6d-4322-b333-04eb9b498bd3" (UID: "997a1e67-2b6d-4322-b333-04eb9b498bd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:07:16.706921 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:16.706868 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997a1e67-2b6d-4322-b333-04eb9b498bd3-kube-api-access-lrscm" pod "997a1e67-2b6d-4322-b333-04eb9b498bd3" (UID: "997a1e67-2b6d-4322-b333-04eb9b498bd3"). InnerVolumeSpecName "kube-api-access-lrscm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:16.806503 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:16.806470 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/997a1e67-2b6d-4322-b333-04eb9b498bd3-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:16.806503 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:16.806499 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrscm\" (UniqueName: \"kubernetes.io/projected/997a1e67-2b6d-4322-b333-04eb9b498bd3-kube-api-access-lrscm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:16.900971 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:16.900797 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:17.428021 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:17.427992 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" Oct 30 20:07:17.428441 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:17.428031 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj" event={"ID":"997a1e67-2b6d-4322-b333-04eb9b498bd3","Type":"ContainerDied","Data":"2a17a0deb93da2c1cba7fa32d4e4100cd17c2ebea94c140c7c832b5e78075d91"} Oct 30 20:07:17.428441 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:17.428078 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a17a0deb93da2c1cba7fa32d4e4100cd17c2ebea94c140c7c832b5e78075d91" Oct 30 20:07:17.825198 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:17.825158 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:18.782668 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:18.782633 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:19.669194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:19.667815 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:20.843508 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:20.843475 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:21.676062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:21.676023 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:22.713729 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:22.713682 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:23.231384 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:23.231345 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:23.437173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:23.435709 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:23.907434 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:23.907402 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:24.102510 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:24.102469 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:24.718301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:24.718268 5091 scope.go:122] "RemoveContainer" containerID="48220fe0de7b3e7984998a5e12b99b704bd0fff4625db00aa18bc67e3f2d85a3" Oct 30 20:07:24.730874 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:24.730849 5091 scope.go:122] "RemoveContainer" containerID="3f2600e43b0c20a30f644256b056c491efbe5998fb796435e2941663b185abe3" Oct 30 20:07:24.742181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:24.742141 5091 scope.go:122] "RemoveContainer" containerID="3c0a1f8067e47010be8b331ee71554d59dc15ef14aadf46d4d785efb3c3e05f1" Oct 30 20:07:24.856267 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:24.856214 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:25.275171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:25.269960 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:25.288168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:25.286303 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf"] Oct 30 20:07:25.565221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:25.565056 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:25.745610 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:25.727483 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:26.529937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:26.529902 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:27.490260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:27.490219 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:28.390850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:28.390811 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:29.425169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:29.424282 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:30.281052 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.281013 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:30.288188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.288124 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" podUID="22238b46-4abd-49a7-9299-736e7efab2f5" containerName="payload-processing" containerID="cri-o://c1c9b98ac55328719271c2e4c5de22bcfb42daba8de5ddb8574ff33cd0e687ab" gracePeriod=25 Oct 30 20:07:30.622282 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.622254 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:07:30.709643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.709604 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22238b46-4abd-49a7-9299-736e7efab2f5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22238b46-4abd-49a7-9299-736e7efab2f5-routing-overlay-volume\") pod \"22238b46-4abd-49a7-9299-736e7efab2f5\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " Oct 30 20:07:30.709849 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.709802 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22238b46-4abd-49a7-9299-736e7efab2f5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/22238b46-4abd-49a7-9299-736e7efab2f5-provider-credentials\") pod \"22238b46-4abd-49a7-9299-736e7efab2f5\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " Oct 30 20:07:30.709973 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.709958 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22238b46-4abd-49a7-9299-736e7efab2f5-kube-api-access-mgcdm\" (UniqueName: \"kubernetes.io/projected/22238b46-4abd-49a7-9299-736e7efab2f5-kube-api-access-mgcdm\") pod \"22238b46-4abd-49a7-9299-736e7efab2f5\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " Oct 30 20:07:30.710038 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.709978 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22238b46-4abd-49a7-9299-736e7efab2f5-routing-overlay-volume" pod "22238b46-4abd-49a7-9299-736e7efab2f5" (UID: "22238b46-4abd-49a7-9299-736e7efab2f5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:30.710098 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.710054 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22238b46-4abd-49a7-9299-736e7efab2f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22238b46-4abd-49a7-9299-736e7efab2f5-plugins-config-volume\") pod \"22238b46-4abd-49a7-9299-736e7efab2f5\" (UID: \"22238b46-4abd-49a7-9299-736e7efab2f5\") " Oct 30 20:07:30.710513 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.710489 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22238b46-4abd-49a7-9299-736e7efab2f5-plugins-config-volume" pod "22238b46-4abd-49a7-9299-736e7efab2f5" (UID: "22238b46-4abd-49a7-9299-736e7efab2f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:07:30.710829 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.710809 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22238b46-4abd-49a7-9299-736e7efab2f5-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:30.710925 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.710832 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/22238b46-4abd-49a7-9299-736e7efab2f5-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:30.712309 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.712279 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22238b46-4abd-49a7-9299-736e7efab2f5-provider-credentials" pod "22238b46-4abd-49a7-9299-736e7efab2f5" (UID: "22238b46-4abd-49a7-9299-736e7efab2f5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:30.712529 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.712508 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22238b46-4abd-49a7-9299-736e7efab2f5-kube-api-access-mgcdm" pod "22238b46-4abd-49a7-9299-736e7efab2f5" (UID: "22238b46-4abd-49a7-9299-736e7efab2f5"). InnerVolumeSpecName "kube-api-access-mgcdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:07:30.812809 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.812771 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgcdm\" (UniqueName: \"kubernetes.io/projected/22238b46-4abd-49a7-9299-736e7efab2f5-kube-api-access-mgcdm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:30.812809 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.812809 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/22238b46-4abd-49a7-9299-736e7efab2f5-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:07:30.839825 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.839788 5091 generic.go:356] "Generic (PLEG): container finished" podID="22238b46-4abd-49a7-9299-736e7efab2f5" containerID="c1c9b98ac55328719271c2e4c5de22bcfb42daba8de5ddb8574ff33cd0e687ab" exitCode=0 Oct 30 20:07:30.840032 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.839831 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" Oct 30 20:07:30.840032 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.839862 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" event={"ID":"22238b46-4abd-49a7-9299-736e7efab2f5","Type":"ContainerDied","Data":"c1c9b98ac55328719271c2e4c5de22bcfb42daba8de5ddb8574ff33cd0e687ab"} Oct 30 20:07:30.840032 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.839917 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf" event={"ID":"22238b46-4abd-49a7-9299-736e7efab2f5","Type":"ContainerDied","Data":"be0bc946f614ecfa4370ebfb046332b4982bb19639394110447cdb889c75aca3"} Oct 30 20:07:30.840032 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.839945 5091 scope.go:122] "RemoveContainer" containerID="c1c9b98ac55328719271c2e4c5de22bcfb42daba8de5ddb8574ff33cd0e687ab" Oct 30 20:07:30.855864 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.855838 5091 scope.go:122] "RemoveContainer" containerID="c1c9b98ac55328719271c2e4c5de22bcfb42daba8de5ddb8574ff33cd0e687ab" Oct 30 20:07:30.856313 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:07:30.856289 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c9b98ac55328719271c2e4c5de22bcfb42daba8de5ddb8574ff33cd0e687ab\": container with ID starting with c1c9b98ac55328719271c2e4c5de22bcfb42daba8de5ddb8574ff33cd0e687ab not found: ID does not exist" containerID="c1c9b98ac55328719271c2e4c5de22bcfb42daba8de5ddb8574ff33cd0e687ab" Oct 30 20:07:30.856424 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.856318 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c9b98ac55328719271c2e4c5de22bcfb42daba8de5ddb8574ff33cd0e687ab"} err="failed to get container status \"c1c9b98ac55328719271c2e4c5de22bcfb42daba8de5ddb8574ff33cd0e687ab\": rpc error: code = NotFound desc = could not find container \"c1c9b98ac55328719271c2e4c5de22bcfb42daba8de5ddb8574ff33cd0e687ab\": container with ID starting with c1c9b98ac55328719271c2e4c5de22bcfb42daba8de5ddb8574ff33cd0e687ab not found: ID does not exist" Oct 30 20:07:30.872713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.872686 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf"] Oct 30 20:07:30.914170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.904973 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-58569r7bf"] Oct 30 20:07:30.937216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:30.933086 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:31.085932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:31.085884 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:31.245076 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:31.244974 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:32.153166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:32.152698 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:32.442930 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:32.442852 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22238b46-4abd-49a7-9299-736e7efab2f5" path="/var/lib/kubelet/pods/22238b46-4abd-49a7-9299-736e7efab2f5/volumes" Oct 30 20:07:32.920826 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:32.920788 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:33.076090 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:33.076048 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:33.216905 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:33.216821 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:34.002170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:34.002035 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:34.944173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:34.938295 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:36.083799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:36.079119 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:37.250411 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:37.250371 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:38.083593 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:38.083559 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:39.000229 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:39.000189 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:39.482256 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:39.482141 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:39.673756 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:39.673720 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:39.889021 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:39.888981 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:40.095579 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.095541 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:40.328948 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.328914 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:40.645225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.625304 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:40.728206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.720399 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c"] Oct 30 20:07:40.728206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.721711 5091 cpu_manager.go:414] "containerMap: removing container" podUID="997a1e67-2b6d-4322-b333-04eb9b498bd3" containerName="revoke-keys" Oct 30 20:07:40.728206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.721739 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997a1e67-2b6d-4322-b333-04eb9b498bd3" containerName="revoke-keys" Oct 30 20:07:40.728206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.721762 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f848a7a0-fafe-4b2b-b8f7-e97ecc993e37" containerName="revoke-keys" Oct 30 20:07:40.728206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.721772 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f848a7a0-fafe-4b2b-b8f7-e97ecc993e37" containerName="revoke-keys" Oct 30 20:07:40.728206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.721789 5091 cpu_manager.go:414] "containerMap: removing container" podUID="22238b46-4abd-49a7-9299-736e7efab2f5" containerName="payload-processing" Oct 30 20:07:40.728206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.721796 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22238b46-4abd-49a7-9299-736e7efab2f5" containerName="payload-processing" Oct 30 20:07:40.728206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.721982 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f848a7a0-fafe-4b2b-b8f7-e97ecc993e37" containerName="revoke-keys" Oct 30 20:07:40.728206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.721996 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="22238b46-4abd-49a7-9299-736e7efab2f5" containerName="payload-processing" Oct 30 20:07:40.728206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.722025 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="997a1e67-2b6d-4322-b333-04eb9b498bd3" containerName="revoke-keys" Oct 30 20:07:40.776181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.773879 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:40.790168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.782190 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c"] Oct 30 20:07:40.813167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.797331 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8c8241\"/\"routing-overlay\"" Oct 30 20:07:40.813167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.798375 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8c8241\"/\"payload-processing-external-model-plugins-e2e-extmod-a-8c8241\"" Oct 30 20:07:40.813167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.798430 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8c8241\"/\"kube-root-ca.crt\"" Oct 30 20:07:40.855177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.849070 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8c8241\"/\"e2e-cmp-legacy-8a1b3e-provider-cred\"" Oct 30 20:07:40.855177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.849220 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-8c8241\"/\"payload-processing-external-model-e2e--6c3d2551-dockercfg-qbqjd\"" Oct 30 20:07:40.910648 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:40.899882 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-8c8241\"/\"openshift-service-ca.crt\"" Oct 30 20:07:41.057169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.054928 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81323fbd-decc-4170-bcfc-87351d7e8cca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:41.057169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.055003 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb2rx\" (UniqueName: \"kubernetes.io/projected/81323fbd-decc-4170-bcfc-87351d7e8cca-kube-api-access-cb2rx\") pod \"payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:41.057169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.055051 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81323fbd-decc-4170-bcfc-87351d7e8cca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:41.057169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.055115 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81323fbd-decc-4170-bcfc-87351d7e8cca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:41.157047 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.157008 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81323fbd-decc-4170-bcfc-87351d7e8cca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:41.157613 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.157053 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2rx\" (UniqueName: \"kubernetes.io/projected/81323fbd-decc-4170-bcfc-87351d7e8cca-kube-api-access-cb2rx\") pod \"payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:41.157613 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.157079 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81323fbd-decc-4170-bcfc-87351d7e8cca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:41.157613 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.157120 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81323fbd-decc-4170-bcfc-87351d7e8cca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:41.157931 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.157905 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81323fbd-decc-4170-bcfc-87351d7e8cca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:41.158027 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.158002 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81323fbd-decc-4170-bcfc-87351d7e8cca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:41.160370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.160340 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81323fbd-decc-4170-bcfc-87351d7e8cca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:41.178783 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.178717 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb2rx\" (UniqueName: \"kubernetes.io/projected/81323fbd-decc-4170-bcfc-87351d7e8cca-kube-api-access-cb2rx\") pod \"payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:41.328207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.327400 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:41.376653 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.376601 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:41.674191 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:07:41.674116 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81323fbd_decc_4170_bcfc_87351d7e8cca.slice/crio-4a0ff4c33d1feec3799c9b415b12bb26d13624701224dcd82128c7e04021f986 WatchSource:0}: Error finding container 4a0ff4c33d1feec3799c9b415b12bb26d13624701224dcd82128c7e04021f986: Status 404 returned error can't find the container with id 4a0ff4c33d1feec3799c9b415b12bb26d13624701224dcd82128c7e04021f986 Oct 30 20:07:41.675814 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.675784 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c"] Oct 30 20:07:41.703699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.703669 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:41.963616 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:41.963529 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:42.149131 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:42.149093 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" event={"ID":"81323fbd-decc-4170-bcfc-87351d7e8cca","Type":"ContainerStarted","Data":"fce07879cec2c6c434be0b62c74b6b82364d2e35c9111899e8b9bbf7637b875d"} Oct 30 20:07:42.149353 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:42.149191 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" event={"ID":"81323fbd-decc-4170-bcfc-87351d7e8cca","Type":"ContainerStarted","Data":"4a0ff4c33d1feec3799c9b415b12bb26d13624701224dcd82128c7e04021f986"} Oct 30 20:07:42.149353 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:42.149231 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:42.167751 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:42.167672 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:42.188869 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:42.188801 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" podStartSLOduration=2.188782007 podStartE2EDuration="2.188782007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:42.184712329 +0000 UTC m=+2181.879920288" watchObservedRunningTime="2026-10-30 20:07:42.188782007 +0000 UTC m=+2181.883989966" Oct 30 20:07:42.447959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:42.447866 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:42.854164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:42.853255 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:43.123426 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:43.123331 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:43.381220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:43.381103 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:43.896421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:43.896376 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:43.943366 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:43.943336 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk"] Oct 30 20:07:43.948368 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:43.948347 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:43.961649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:43.961359 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"agc-cred-34e5760b7a4e\"" Oct 30 20:07:43.961649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:43.961415 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"openshift-service-ca.crt\"" Oct 30 20:07:43.961649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:43.961436 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"payload-processing-external-model-e2e--d955aedb-dockercfg-xhz2l\"" Oct 30 20:07:43.961649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:43.961499 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"kube-root-ca.crt\"" Oct 30 20:07:43.961649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:43.961516 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"routing-overlay\"" Oct 30 20:07:43.961649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:43.961537 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c9bce8\"" Oct 30 20:07:43.964542 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:43.964513 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk"] Oct 30 20:07:44.037117 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.037082 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:44.037305 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.037263 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:44.037371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.037354 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcntm\" (UniqueName: \"kubernetes.io/projected/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-kube-api-access-rcntm\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:44.037423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.037382 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:44.139608 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.139564 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:44.139833 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.139758 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:44.139926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.139891 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcntm\" (UniqueName: \"kubernetes.io/projected/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-kube-api-access-rcntm\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:44.140002 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.139940 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:44.140751 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.140724 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:44.140884 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.140764 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:44.142433 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.142413 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:44.154724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.154671 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcntm\" (UniqueName: \"kubernetes.io/projected/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-kube-api-access-rcntm\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:44.260852 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.260811 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:44.275808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.275764 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:44.423577 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.423120 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:44.794039 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:07:44.794004 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fe22a50_9a48_41bb_b837_7b3eb3b2b6a4.slice/crio-a2f287c9f7950ab4756e3872868b469b75f7baef8072ec4853a0f40bc12147a9 WatchSource:0}: Error finding container a2f287c9f7950ab4756e3872868b469b75f7baef8072ec4853a0f40bc12147a9: Status 404 returned error can't find the container with id a2f287c9f7950ab4756e3872868b469b75f7baef8072ec4853a0f40bc12147a9 Oct 30 20:07:44.795621 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:44.795591 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk"] Oct 30 20:07:45.221220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:45.214804 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:45.271590 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:45.271540 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" event={"ID":"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4","Type":"ContainerStarted","Data":"d17a0d3c6e7399b78f4f001c3465334b2c8cd6b5535385ae6c0edd6c8c882f75"} Oct 30 20:07:45.271770 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:45.271608 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" event={"ID":"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4","Type":"ContainerStarted","Data":"a2f287c9f7950ab4756e3872868b469b75f7baef8072ec4853a0f40bc12147a9"} Oct 30 20:07:45.271770 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:45.271637 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:45.299019 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:45.298957 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" podStartSLOduration=2.298940053 podStartE2EDuration="2.298940053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:07:45.296814804 +0000 UTC m=+2184.992022763" watchObservedRunningTime="2026-10-30 20:07:45.298940053 +0000 UTC m=+2184.994148012" Oct 30 20:07:45.986647 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:07:45.986604 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:07:46.582218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:46.582179 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:46.739173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:46.734961 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:47.020076 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:47.020027 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:48.235699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:48.235651 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:49.055677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:49.055641 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:49.924882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:49.924774 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:51.016707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:51.016663 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:52.006705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:52.006670 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:53.002166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:52.998824 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:53.161560 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:53.161529 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:07:53.918294 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:53.918255 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:54.844530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:54.844488 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:55.716848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:55.716809 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:56.284434 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:56.284402 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:07:56.840500 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:56.840465 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:57.926734 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:57.926694 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:58.797875 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:58.788883 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:07:59.500384 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:59.500350 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj"] Oct 30 20:07:59.783975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:07:59.783871 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:00.844834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:00.843966 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:01.796171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:01.789727 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:02.676271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:02.658695 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:03.475062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:03.475019 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:04.370836 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:04.370796 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:04.501512 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:04.501467 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" podUID="7a331856-8296-4c36-8263-e88f28287497" containerName="payload-processing" containerID="cri-o://eba62ff5d1926089857435cb8f2bf6de957902622a627595d7457a4e7e685921" gracePeriod=25 Oct 30 20:08:04.795754 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:04.795726 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:08:04.920759 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:04.920659 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a331856-8296-4c36-8263-e88f28287497-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a331856-8296-4c36-8263-e88f28287497-routing-overlay-volume\") pod \"7a331856-8296-4c36-8263-e88f28287497\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " Oct 30 20:08:04.920759 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:04.920755 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a331856-8296-4c36-8263-e88f28287497-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a331856-8296-4c36-8263-e88f28287497-plugins-config-volume\") pod \"7a331856-8296-4c36-8263-e88f28287497\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " Oct 30 20:08:04.921016 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:04.920875 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a331856-8296-4c36-8263-e88f28287497-kube-api-access-jnnqs\" (UniqueName: \"kubernetes.io/projected/7a331856-8296-4c36-8263-e88f28287497-kube-api-access-jnnqs\") pod \"7a331856-8296-4c36-8263-e88f28287497\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " Oct 30 20:08:04.921084 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:04.921037 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a331856-8296-4c36-8263-e88f28287497-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a331856-8296-4c36-8263-e88f28287497-provider-credentials\") pod \"7a331856-8296-4c36-8263-e88f28287497\" (UID: \"7a331856-8296-4c36-8263-e88f28287497\") " Oct 30 20:08:04.921137 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:04.921092 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a331856-8296-4c36-8263-e88f28287497-routing-overlay-volume" pod "7a331856-8296-4c36-8263-e88f28287497" (UID: "7a331856-8296-4c36-8263-e88f28287497"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:04.921215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:04.921167 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a331856-8296-4c36-8263-e88f28287497-plugins-config-volume" pod "7a331856-8296-4c36-8263-e88f28287497" (UID: "7a331856-8296-4c36-8263-e88f28287497"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:04.921733 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:04.921706 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7a331856-8296-4c36-8263-e88f28287497-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:04.921733 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:04.921733 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a331856-8296-4c36-8263-e88f28287497-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:04.923113 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:04.923083 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a331856-8296-4c36-8263-e88f28287497-provider-credentials" pod "7a331856-8296-4c36-8263-e88f28287497" (UID: "7a331856-8296-4c36-8263-e88f28287497"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:04.923323 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:04.923303 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a331856-8296-4c36-8263-e88f28287497-kube-api-access-jnnqs" pod "7a331856-8296-4c36-8263-e88f28287497" (UID: "7a331856-8296-4c36-8263-e88f28287497"). InnerVolumeSpecName "kube-api-access-jnnqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:05.023378 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:05.023344 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnnqs\" (UniqueName: \"kubernetes.io/projected/7a331856-8296-4c36-8263-e88f28287497-kube-api-access-jnnqs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:05.023378 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:05.023373 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7a331856-8296-4c36-8263-e88f28287497-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:05.292016 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:05.290345 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:05.462669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:05.462630 5091 generic.go:356] "Generic (PLEG): container finished" podID="7a331856-8296-4c36-8263-e88f28287497" containerID="eba62ff5d1926089857435cb8f2bf6de957902622a627595d7457a4e7e685921" exitCode=0 Oct 30 20:08:05.463235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:05.462678 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" Oct 30 20:08:05.463235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:05.462715 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" event={"ID":"7a331856-8296-4c36-8263-e88f28287497","Type":"ContainerDied","Data":"eba62ff5d1926089857435cb8f2bf6de957902622a627595d7457a4e7e685921"} Oct 30 20:08:05.463235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:05.462775 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj" event={"ID":"7a331856-8296-4c36-8263-e88f28287497","Type":"ContainerDied","Data":"df7adcef97bbedc6e69d387072306799df5db3952e60c6e8ad4fcd38443b9da4"} Oct 30 20:08:05.463235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:05.462804 5091 scope.go:122] "RemoveContainer" containerID="eba62ff5d1926089857435cb8f2bf6de957902622a627595d7457a4e7e685921" Oct 30 20:08:05.478106 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:05.478079 5091 scope.go:122] "RemoveContainer" containerID="eba62ff5d1926089857435cb8f2bf6de957902622a627595d7457a4e7e685921" Oct 30 20:08:05.478464 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:05.478423 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eba62ff5d1926089857435cb8f2bf6de957902622a627595d7457a4e7e685921\": container with ID starting with eba62ff5d1926089857435cb8f2bf6de957902622a627595d7457a4e7e685921 not found: ID does not exist" containerID="eba62ff5d1926089857435cb8f2bf6de957902622a627595d7457a4e7e685921" Oct 30 20:08:05.478541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:05.478478 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba62ff5d1926089857435cb8f2bf6de957902622a627595d7457a4e7e685921"} err="failed to get container status \"eba62ff5d1926089857435cb8f2bf6de957902622a627595d7457a4e7e685921\": rpc error: code = NotFound desc = could not find container \"eba62ff5d1926089857435cb8f2bf6de957902622a627595d7457a4e7e685921\": container with ID starting with eba62ff5d1926089857435cb8f2bf6de957902622a627595d7457a4e7e685921 not found: ID does not exist" Oct 30 20:08:05.495668 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:05.495636 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj"] Oct 30 20:08:05.500237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:05.500213 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-84846dvwj"] Oct 30 20:08:06.122171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:06.113249 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:06.444122 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:06.444030 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a331856-8296-4c36-8263-e88f28287497" path="/var/lib/kubelet/pods/7a331856-8296-4c36-8263-e88f28287497/volumes" Oct 30 20:08:07.039233 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:07.027649 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:07.941473 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:07.941440 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:08.187410 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:08.187368 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:08.795585 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:08.795552 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:09.728324 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:09.727388 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:10.583024 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:10.582986 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:11.462204 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:11.459945 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:12.673407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:12.673372 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:14.200879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:14.200839 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:14.386573 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:14.386540 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:15.187922 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:15.187876 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:15.979846 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:15.979799 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:08:16.045221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.045185 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2"] Oct 30 20:08:16.046082 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.046057 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7a331856-8296-4c36-8263-e88f28287497" containerName="payload-processing" Oct 30 20:08:16.046082 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.046081 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a331856-8296-4c36-8263-e88f28287497" containerName="payload-processing" Oct 30 20:08:16.046301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.046265 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a331856-8296-4c36-8263-e88f28287497" containerName="payload-processing" Oct 30 20:08:16.050241 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.050222 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" Oct 30 20:08:16.056856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.056831 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:08:16.058016 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.057992 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2"] Oct 30 20:08:16.059730 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.059251 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:08:16.114082 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.114046 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29893af6-a166-4973-b38d-55ab6177a749-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a1a1863-tzhc2\" (UID: \"29893af6-a166-4973-b38d-55ab6177a749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" Oct 30 20:08:16.114348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.114231 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29893af6-a166-4973-b38d-55ab6177a749-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a1a1863-tzhc2\" (UID: \"29893af6-a166-4973-b38d-55ab6177a749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" Oct 30 20:08:16.114478 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.114455 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdlxb\" (UniqueName: \"kubernetes.io/projected/29893af6-a166-4973-b38d-55ab6177a749-kube-api-access-pdlxb\") pod \"maas-api-revoke-sub-7a1a1863-tzhc2\" (UID: \"29893af6-a166-4973-b38d-55ab6177a749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" Oct 30 20:08:16.215855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.215816 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29893af6-a166-4973-b38d-55ab6177a749-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a1a1863-tzhc2\" (UID: \"29893af6-a166-4973-b38d-55ab6177a749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" Oct 30 20:08:16.216048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.215930 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29893af6-a166-4973-b38d-55ab6177a749-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a1a1863-tzhc2\" (UID: \"29893af6-a166-4973-b38d-55ab6177a749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" Oct 30 20:08:16.216183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.216156 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdlxb\" (UniqueName: \"kubernetes.io/projected/29893af6-a166-4973-b38d-55ab6177a749-kube-api-access-pdlxb\") pod \"maas-api-revoke-sub-7a1a1863-tzhc2\" (UID: \"29893af6-a166-4973-b38d-55ab6177a749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" Oct 30 20:08:16.216662 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.216634 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29893af6-a166-4973-b38d-55ab6177a749-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a1a1863-tzhc2\" (UID: \"29893af6-a166-4973-b38d-55ab6177a749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" Oct 30 20:08:16.218530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.218510 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29893af6-a166-4973-b38d-55ab6177a749-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a1a1863-tzhc2\" (UID: \"29893af6-a166-4973-b38d-55ab6177a749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" Oct 30 20:08:16.230626 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.230576 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdlxb\" (UniqueName: \"kubernetes.io/projected/29893af6-a166-4973-b38d-55ab6177a749-kube-api-access-pdlxb\") pod \"maas-api-revoke-sub-7a1a1863-tzhc2\" (UID: \"29893af6-a166-4973-b38d-55ab6177a749\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" Oct 30 20:08:16.370056 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.370025 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" Oct 30 20:08:16.597783 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.597353 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:16.668880 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:08:16.668851 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29893af6_a166_4973_b38d_55ab6177a749.slice/crio-41de0f40406d0f3f9cf22a823614e78216fb681d8f0a86513f617c90027f362e WatchSource:0}: Error finding container 41de0f40406d0f3f9cf22a823614e78216fb681d8f0a86513f617c90027f362e: Status 404 returned error can't find the container with id 41de0f40406d0f3f9cf22a823614e78216fb681d8f0a86513f617c90027f362e Oct 30 20:08:16.671518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:16.671491 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2"] Oct 30 20:08:17.577550 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:17.576381 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:17.674379 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:17.674353 5091 generic.go:356] "Generic (PLEG): container finished" podID="29893af6-a166-4973-b38d-55ab6177a749" containerID="8d97b3d0fb05faf7e28561e8575eaf86a26e70b2ce241078f3c079d3d05c0889" exitCode=0 Oct 30 20:08:17.674526 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:17.674441 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" event={"ID":"29893af6-a166-4973-b38d-55ab6177a749","Type":"ContainerDied","Data":"8d97b3d0fb05faf7e28561e8575eaf86a26e70b2ce241078f3c079d3d05c0889"} Oct 30 20:08:17.674526 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:17.674513 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" event={"ID":"29893af6-a166-4973-b38d-55ab6177a749","Type":"ContainerStarted","Data":"41de0f40406d0f3f9cf22a823614e78216fb681d8f0a86513f617c90027f362e"} Oct 30 20:08:18.499857 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:18.499816 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:18.874873 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:18.874843 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" Oct 30 20:08:19.008483 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:19.008442 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29893af6-a166-4973-b38d-55ab6177a749-kube-api-access-pdlxb\" (UniqueName: \"kubernetes.io/projected/29893af6-a166-4973-b38d-55ab6177a749-kube-api-access-pdlxb\") pod \"29893af6-a166-4973-b38d-55ab6177a749\" (UID: \"29893af6-a166-4973-b38d-55ab6177a749\") " Oct 30 20:08:19.008667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:19.008568 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29893af6-a166-4973-b38d-55ab6177a749-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29893af6-a166-4973-b38d-55ab6177a749-maas-api-xks-ca\") pod \"29893af6-a166-4973-b38d-55ab6177a749\" (UID: \"29893af6-a166-4973-b38d-55ab6177a749\") " Oct 30 20:08:19.008667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:19.008637 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29893af6-a166-4973-b38d-55ab6177a749-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29893af6-a166-4973-b38d-55ab6177a749-maas-api-openshift-ca\") pod \"29893af6-a166-4973-b38d-55ab6177a749\" (UID: \"29893af6-a166-4973-b38d-55ab6177a749\") " Oct 30 20:08:19.009045 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:19.009013 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29893af6-a166-4973-b38d-55ab6177a749-maas-api-openshift-ca" pod "29893af6-a166-4973-b38d-55ab6177a749" (UID: "29893af6-a166-4973-b38d-55ab6177a749"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:19.009303 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:19.009283 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29893af6-a166-4973-b38d-55ab6177a749-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:19.010707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:19.010679 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29893af6-a166-4973-b38d-55ab6177a749-maas-api-xks-ca" pod "29893af6-a166-4973-b38d-55ab6177a749" (UID: "29893af6-a166-4973-b38d-55ab6177a749"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:08:19.010831 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:19.010783 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29893af6-a166-4973-b38d-55ab6177a749-kube-api-access-pdlxb" pod "29893af6-a166-4973-b38d-55ab6177a749" (UID: "29893af6-a166-4973-b38d-55ab6177a749"). InnerVolumeSpecName "kube-api-access-pdlxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:19.110988 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:19.110898 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29893af6-a166-4973-b38d-55ab6177a749-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:19.110988 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:19.110932 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdlxb\" (UniqueName: \"kubernetes.io/projected/29893af6-a166-4973-b38d-55ab6177a749-kube-api-access-pdlxb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:19.383328 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:19.378993 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:19.695547 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:19.695457 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" event={"ID":"29893af6-a166-4973-b38d-55ab6177a749","Type":"ContainerDied","Data":"41de0f40406d0f3f9cf22a823614e78216fb681d8f0a86513f617c90027f362e"} Oct 30 20:08:19.695547 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:19.695513 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41de0f40406d0f3f9cf22a823614e78216fb681d8f0a86513f617c90027f362e" Oct 30 20:08:19.695547 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:19.695471 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2" Oct 30 20:08:19.749947 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:19.749915 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29893af6_a166_4973_b38d_55ab6177a749.slice\": RecentStats: unable to find data in memory cache]" Oct 30 20:08:20.425170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:20.420899 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:21.688090 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.688049 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:21.739640 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.739607 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5"] Oct 30 20:08:21.740404 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.740386 5091 cpu_manager.go:414] "containerMap: removing container" podUID="29893af6-a166-4973-b38d-55ab6177a749" containerName="revoke-keys" Oct 30 20:08:21.740404 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.740403 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29893af6-a166-4973-b38d-55ab6177a749" containerName="revoke-keys" Oct 30 20:08:21.740578 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.740568 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="29893af6-a166-4973-b38d-55ab6177a749" containerName="revoke-keys" Oct 30 20:08:21.744518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.744499 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" Oct 30 20:08:21.757107 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.757080 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:08:21.759634 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.759609 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:08:21.766358 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.766329 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5"] Oct 30 20:08:21.787568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.787530 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvnd\" (UniqueName: \"kubernetes.io/projected/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-kube-api-access-kkvnd\") pod \"maas-api-revoke-sub-f8abbcd9-zklq5\" (UID: \"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" Oct 30 20:08:21.787915 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.787891 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8abbcd9-zklq5\" (UID: \"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" Oct 30 20:08:21.788096 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.788077 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8abbcd9-zklq5\" (UID: \"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" Oct 30 20:08:21.890683 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.890640 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvnd\" (UniqueName: \"kubernetes.io/projected/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-kube-api-access-kkvnd\") pod \"maas-api-revoke-sub-f8abbcd9-zklq5\" (UID: \"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" Oct 30 20:08:21.890906 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.890743 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8abbcd9-zklq5\" (UID: \"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" Oct 30 20:08:21.890906 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.890785 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8abbcd9-zklq5\" (UID: \"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" Oct 30 20:08:21.891513 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.891491 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f8abbcd9-zklq5\" (UID: \"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" Oct 30 20:08:21.894058 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.894028 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f8abbcd9-zklq5\" (UID: \"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" Oct 30 20:08:21.906941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:21.906916 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvnd\" (UniqueName: \"kubernetes.io/projected/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-kube-api-access-kkvnd\") pod \"maas-api-revoke-sub-f8abbcd9-zklq5\" (UID: \"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" Oct 30 20:08:22.069694 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.069628 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" Oct 30 20:08:22.300305 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.300265 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr"] Oct 30 20:08:22.305223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.305199 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" Oct 30 20:08:22.314971 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.314939 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr"] Oct 30 20:08:22.397272 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.397242 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5"] Oct 30 20:08:22.399445 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.399332 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be037c9f-4faf-42a4-b160-1b92dc24dcd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-299e8d76-92nlr\" (UID: \"be037c9f-4faf-42a4-b160-1b92dc24dcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" Oct 30 20:08:22.400004 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.399951 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be037c9f-4faf-42a4-b160-1b92dc24dcd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-299e8d76-92nlr\" (UID: \"be037c9f-4faf-42a4-b160-1b92dc24dcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" Oct 30 20:08:22.400220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.400201 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttvs2\" (UniqueName: \"kubernetes.io/projected/be037c9f-4faf-42a4-b160-1b92dc24dcd0-kube-api-access-ttvs2\") pod \"maas-api-revoke-sub-299e8d76-92nlr\" (UID: \"be037c9f-4faf-42a4-b160-1b92dc24dcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" Oct 30 20:08:22.501476 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.501440 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttvs2\" (UniqueName: \"kubernetes.io/projected/be037c9f-4faf-42a4-b160-1b92dc24dcd0-kube-api-access-ttvs2\") pod \"maas-api-revoke-sub-299e8d76-92nlr\" (UID: \"be037c9f-4faf-42a4-b160-1b92dc24dcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" Oct 30 20:08:22.501721 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.501697 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be037c9f-4faf-42a4-b160-1b92dc24dcd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-299e8d76-92nlr\" (UID: \"be037c9f-4faf-42a4-b160-1b92dc24dcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" Oct 30 20:08:22.501879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.501856 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be037c9f-4faf-42a4-b160-1b92dc24dcd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-299e8d76-92nlr\" (UID: \"be037c9f-4faf-42a4-b160-1b92dc24dcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" Oct 30 20:08:22.502490 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.502463 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be037c9f-4faf-42a4-b160-1b92dc24dcd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-299e8d76-92nlr\" (UID: \"be037c9f-4faf-42a4-b160-1b92dc24dcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" Oct 30 20:08:22.547563 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.547538 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be037c9f-4faf-42a4-b160-1b92dc24dcd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-299e8d76-92nlr\" (UID: \"be037c9f-4faf-42a4-b160-1b92dc24dcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" Oct 30 20:08:22.547810 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.547775 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttvs2\" (UniqueName: \"kubernetes.io/projected/be037c9f-4faf-42a4-b160-1b92dc24dcd0-kube-api-access-ttvs2\") pod \"maas-api-revoke-sub-299e8d76-92nlr\" (UID: \"be037c9f-4faf-42a4-b160-1b92dc24dcd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" Oct 30 20:08:22.630259 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.630138 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" Oct 30 20:08:22.724350 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.724311 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" event={"ID":"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67","Type":"ContainerStarted","Data":"bdc220f79537c085a86c0e5b6498bc30bee90e3fc21b985027afcdcb47180651"} Oct 30 20:08:22.811603 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.811469 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:22.940952 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:08:22.940911 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe037c9f_4faf_42a4_b160_1b92dc24dcd0.slice/crio-fc6ff1218688379ab9d53a3f460e18630faa337cd8457dfb72470a5491c0e879 WatchSource:0}: Error finding container fc6ff1218688379ab9d53a3f460e18630faa337cd8457dfb72470a5491c0e879: Status 404 returned error can't find the container with id fc6ff1218688379ab9d53a3f460e18630faa337cd8457dfb72470a5491c0e879 Oct 30 20:08:22.942316 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:22.942287 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr"] Oct 30 20:08:23.737553 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:23.737458 5091 generic.go:356] "Generic (PLEG): container finished" podID="138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67" containerID="deef98030540979c2e1b8544522f1efd70e0aec8a8679d1e50259563e6aa8fc7" exitCode=0 Oct 30 20:08:23.738085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:23.737554 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" event={"ID":"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67","Type":"ContainerDied","Data":"deef98030540979c2e1b8544522f1efd70e0aec8a8679d1e50259563e6aa8fc7"} Oct 30 20:08:23.739441 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:23.739409 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" event={"ID":"be037c9f-4faf-42a4-b160-1b92dc24dcd0","Type":"ContainerStarted","Data":"1c66aceccd78751682872ab78d7801ed10ceabd0d9ae599b6566ecf33fbcb5d3"} Oct 30 20:08:23.739575 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:23.739474 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" event={"ID":"be037c9f-4faf-42a4-b160-1b92dc24dcd0","Type":"ContainerStarted","Data":"fc6ff1218688379ab9d53a3f460e18630faa337cd8457dfb72470a5491c0e879"} Oct 30 20:08:23.835190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:23.829938 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:23.850229 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:23.844497 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" podStartSLOduration=1.84447325 podStartE2EDuration="1.84447325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:23.795550436 +0000 UTC m=+2223.490758408" watchObservedRunningTime="2026-10-30 20:08:23.84447325 +0000 UTC m=+2223.539681208" Oct 30 20:08:24.749828 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:24.749794 5091 generic.go:356] "Generic (PLEG): container finished" podID="be037c9f-4faf-42a4-b160-1b92dc24dcd0" containerID="1c66aceccd78751682872ab78d7801ed10ceabd0d9ae599b6566ecf33fbcb5d3" exitCode=0 Oct 30 20:08:24.750382 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:24.749880 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" event={"ID":"be037c9f-4faf-42a4-b160-1b92dc24dcd0","Type":"ContainerDied","Data":"1c66aceccd78751682872ab78d7801ed10ceabd0d9ae599b6566ecf33fbcb5d3"} Oct 30 20:08:24.838112 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:24.838078 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:24.889860 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:24.889825 5091 scope.go:122] "RemoveContainer" containerID="516cc1f28b5d90936a314734534a3ee142e42cb68311d0b5f099736597e2ecd4" Oct 30 20:08:24.907189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:24.906223 5091 scope.go:122] "RemoveContainer" containerID="1c2fd532b63bde926ee92192e4db9ba4359264cb0336478f203fc1a31c53d7ff" Oct 30 20:08:24.917079 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:24.917058 5091 scope.go:122] "RemoveContainer" containerID="63946c076c0b6bd62dbf063d506906cd8595712156d1271ac21c9de2c5789f89" Oct 30 20:08:24.936494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:24.936405 5091 scope.go:122] "RemoveContainer" containerID="6ec1950c5c991b0ef0351fb75b865b991c871480c12332f071623d17a2fee018" Oct 30 20:08:24.949021 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:24.948738 5091 scope.go:122] "RemoveContainer" containerID="33291e8e5bcdcbdb80807bfa50f9f3f282d9dd31a646516f72f3237544deef95" Oct 30 20:08:25.062661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:25.062635 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" Oct 30 20:08:25.164323 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:25.164287 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-kube-api-access-kkvnd\" (UniqueName: \"kubernetes.io/projected/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-kube-api-access-kkvnd\") pod \"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67\" (UID: \"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67\") " Oct 30 20:08:25.164522 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:25.164365 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-maas-api-xks-ca\") pod \"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67\" (UID: \"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67\") " Oct 30 20:08:25.164522 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:25.164494 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-maas-api-openshift-ca\") pod \"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67\" (UID: \"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67\") " Oct 30 20:08:25.165008 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:25.164967 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-maas-api-openshift-ca" pod "138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67" (UID: "138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:25.165502 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:25.165474 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:25.166807 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:25.166776 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-maas-api-xks-ca" pod "138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67" (UID: "138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:08:25.166908 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:25.166874 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-kube-api-access-kkvnd" pod "138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67" (UID: "138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67"). InnerVolumeSpecName "kube-api-access-kkvnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:25.267104 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:25.267018 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkvnd\" (UniqueName: \"kubernetes.io/projected/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-kube-api-access-kkvnd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:25.267104 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:25.267050 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:25.762658 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:25.762630 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" Oct 30 20:08:25.763324 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:25.762667 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5" event={"ID":"138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67","Type":"ContainerDied","Data":"bdc220f79537c085a86c0e5b6498bc30bee90e3fc21b985027afcdcb47180651"} Oct 30 20:08:25.763324 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:25.762723 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bdc220f79537c085a86c0e5b6498bc30bee90e3fc21b985027afcdcb47180651" Oct 30 20:08:25.973435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:25.961786 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:26.014699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.014637 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" Oct 30 20:08:26.050677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.050639 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be037c9f-4faf-42a4-b160-1b92dc24dcd0-kube-api-access-ttvs2\" (UniqueName: \"kubernetes.io/projected/be037c9f-4faf-42a4-b160-1b92dc24dcd0-kube-api-access-ttvs2\") pod \"be037c9f-4faf-42a4-b160-1b92dc24dcd0\" (UID: \"be037c9f-4faf-42a4-b160-1b92dc24dcd0\") " Oct 30 20:08:26.050862 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.050841 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be037c9f-4faf-42a4-b160-1b92dc24dcd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be037c9f-4faf-42a4-b160-1b92dc24dcd0-maas-api-xks-ca\") pod \"be037c9f-4faf-42a4-b160-1b92dc24dcd0\" (UID: \"be037c9f-4faf-42a4-b160-1b92dc24dcd0\") " Oct 30 20:08:26.050985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.050966 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be037c9f-4faf-42a4-b160-1b92dc24dcd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be037c9f-4faf-42a4-b160-1b92dc24dcd0-maas-api-openshift-ca\") pod \"be037c9f-4faf-42a4-b160-1b92dc24dcd0\" (UID: \"be037c9f-4faf-42a4-b160-1b92dc24dcd0\") " Oct 30 20:08:26.052013 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.051985 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be037c9f-4faf-42a4-b160-1b92dc24dcd0-maas-api-openshift-ca" pod "be037c9f-4faf-42a4-b160-1b92dc24dcd0" (UID: "be037c9f-4faf-42a4-b160-1b92dc24dcd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:26.055199 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.055127 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be037c9f-4faf-42a4-b160-1b92dc24dcd0-maas-api-xks-ca" pod "be037c9f-4faf-42a4-b160-1b92dc24dcd0" (UID: "be037c9f-4faf-42a4-b160-1b92dc24dcd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:08:26.073717 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.073691 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be037c9f-4faf-42a4-b160-1b92dc24dcd0-kube-api-access-ttvs2" pod "be037c9f-4faf-42a4-b160-1b92dc24dcd0" (UID: "be037c9f-4faf-42a4-b160-1b92dc24dcd0"). InnerVolumeSpecName "kube-api-access-ttvs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:26.153512 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.153475 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttvs2\" (UniqueName: \"kubernetes.io/projected/be037c9f-4faf-42a4-b160-1b92dc24dcd0-kube-api-access-ttvs2\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:26.153512 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.153506 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be037c9f-4faf-42a4-b160-1b92dc24dcd0-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:26.153726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.153521 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be037c9f-4faf-42a4-b160-1b92dc24dcd0-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:26.725191 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.725099 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr"] Oct 30 20:08:26.725935 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.725917 5091 cpu_manager.go:414] "containerMap: removing container" podUID="138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67" containerName="revoke-keys" Oct 30 20:08:26.725979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.725937 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67" containerName="revoke-keys" Oct 30 20:08:26.725979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.725957 5091 cpu_manager.go:414] "containerMap: removing container" podUID="be037c9f-4faf-42a4-b160-1b92dc24dcd0" containerName="revoke-keys" Oct 30 20:08:26.725979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.725963 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be037c9f-4faf-42a4-b160-1b92dc24dcd0" containerName="revoke-keys" Oct 30 20:08:26.726131 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.726121 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67" containerName="revoke-keys" Oct 30 20:08:26.726190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.726133 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="be037c9f-4faf-42a4-b160-1b92dc24dcd0" containerName="revoke-keys" Oct 30 20:08:26.730125 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.730096 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" Oct 30 20:08:26.742613 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.741001 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr"] Oct 30 20:08:26.765889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.764953 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/659451eb-0ad3-439c-8a9d-02fa6a9431d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14a6ab4-vddjr\" (UID: \"659451eb-0ad3-439c-8a9d-02fa6a9431d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" Oct 30 20:08:26.765889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.765190 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/659451eb-0ad3-439c-8a9d-02fa6a9431d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14a6ab4-vddjr\" (UID: \"659451eb-0ad3-439c-8a9d-02fa6a9431d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" Oct 30 20:08:26.765889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.765355 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-756hw\" (UniqueName: \"kubernetes.io/projected/659451eb-0ad3-439c-8a9d-02fa6a9431d9-kube-api-access-756hw\") pod \"maas-api-revoke-sub-f14a6ab4-vddjr\" (UID: \"659451eb-0ad3-439c-8a9d-02fa6a9431d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" Oct 30 20:08:26.784194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.784045 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" event={"ID":"be037c9f-4faf-42a4-b160-1b92dc24dcd0","Type":"ContainerDied","Data":"fc6ff1218688379ab9d53a3f460e18630faa337cd8457dfb72470a5491c0e879"} Oct 30 20:08:26.784194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.784124 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc6ff1218688379ab9d53a3f460e18630faa337cd8457dfb72470a5491c0e879" Oct 30 20:08:26.784419 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.784256 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr" Oct 30 20:08:26.867008 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.866968 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/659451eb-0ad3-439c-8a9d-02fa6a9431d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14a6ab4-vddjr\" (UID: \"659451eb-0ad3-439c-8a9d-02fa6a9431d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" Oct 30 20:08:26.867235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.867082 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-756hw\" (UniqueName: \"kubernetes.io/projected/659451eb-0ad3-439c-8a9d-02fa6a9431d9-kube-api-access-756hw\") pod \"maas-api-revoke-sub-f14a6ab4-vddjr\" (UID: \"659451eb-0ad3-439c-8a9d-02fa6a9431d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" Oct 30 20:08:26.867388 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.867370 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/659451eb-0ad3-439c-8a9d-02fa6a9431d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14a6ab4-vddjr\" (UID: \"659451eb-0ad3-439c-8a9d-02fa6a9431d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" Oct 30 20:08:26.867630 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.867610 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/659451eb-0ad3-439c-8a9d-02fa6a9431d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14a6ab4-vddjr\" (UID: \"659451eb-0ad3-439c-8a9d-02fa6a9431d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" Oct 30 20:08:26.869938 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.869915 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/659451eb-0ad3-439c-8a9d-02fa6a9431d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14a6ab4-vddjr\" (UID: \"659451eb-0ad3-439c-8a9d-02fa6a9431d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" Oct 30 20:08:26.923926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.923840 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-756hw\" (UniqueName: \"kubernetes.io/projected/659451eb-0ad3-439c-8a9d-02fa6a9431d9-kube-api-access-756hw\") pod \"maas-api-revoke-sub-f14a6ab4-vddjr\" (UID: \"659451eb-0ad3-439c-8a9d-02fa6a9431d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" Oct 30 20:08:26.930176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:26.925909 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:27.052546 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:27.052507 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" Oct 30 20:08:27.350255 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:08:27.350220 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659451eb_0ad3_439c_8a9d_02fa6a9431d9.slice/crio-0bf063980cfceb3952f4af7ca77452c3bfe6a278256841048e3e19f58f4bcc19 WatchSource:0}: Error finding container 0bf063980cfceb3952f4af7ca77452c3bfe6a278256841048e3e19f58f4bcc19: Status 404 returned error can't find the container with id 0bf063980cfceb3952f4af7ca77452c3bfe6a278256841048e3e19f58f4bcc19 Oct 30 20:08:27.354781 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:27.354752 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr"] Oct 30 20:08:27.796835 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:27.796799 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" event={"ID":"659451eb-0ad3-439c-8a9d-02fa6a9431d9","Type":"ContainerStarted","Data":"da2c6817fcdb39609bd5391ce9f76c76fda163ae308f534945b36a3209ce3428"} Oct 30 20:08:27.797266 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:27.796859 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" event={"ID":"659451eb-0ad3-439c-8a9d-02fa6a9431d9","Type":"ContainerStarted","Data":"0bf063980cfceb3952f4af7ca77452c3bfe6a278256841048e3e19f58f4bcc19"} Oct 30 20:08:27.825709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:27.825662 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" podStartSLOduration=1.825647283 podStartE2EDuration="1.825647283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:27.821812254 +0000 UTC m=+2227.517020215" watchObservedRunningTime="2026-10-30 20:08:27.825647283 +0000 UTC m=+2227.520855241" Oct 30 20:08:27.920171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:27.916166 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:28.848182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:28.847306 5091 generic.go:356] "Generic (PLEG): container finished" podID="659451eb-0ad3-439c-8a9d-02fa6a9431d9" containerID="da2c6817fcdb39609bd5391ce9f76c76fda163ae308f534945b36a3209ce3428" exitCode=0 Oct 30 20:08:28.848182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:28.847533 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" event={"ID":"659451eb-0ad3-439c-8a9d-02fa6a9431d9","Type":"ContainerDied","Data":"da2c6817fcdb39609bd5391ce9f76c76fda163ae308f534945b36a3209ce3428"} Oct 30 20:08:28.868976 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:28.868937 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:29.806471 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:29.806430 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:30.117803 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:30.117773 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" Oct 30 20:08:30.280955 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:30.280919 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/659451eb-0ad3-439c-8a9d-02fa6a9431d9-kube-api-access-756hw\" (UniqueName: \"kubernetes.io/projected/659451eb-0ad3-439c-8a9d-02fa6a9431d9-kube-api-access-756hw\") pod \"659451eb-0ad3-439c-8a9d-02fa6a9431d9\" (UID: \"659451eb-0ad3-439c-8a9d-02fa6a9431d9\") " Oct 30 20:08:30.281183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:30.281001 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/659451eb-0ad3-439c-8a9d-02fa6a9431d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/659451eb-0ad3-439c-8a9d-02fa6a9431d9-maas-api-openshift-ca\") pod \"659451eb-0ad3-439c-8a9d-02fa6a9431d9\" (UID: \"659451eb-0ad3-439c-8a9d-02fa6a9431d9\") " Oct 30 20:08:30.281270 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:30.281214 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/659451eb-0ad3-439c-8a9d-02fa6a9431d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/659451eb-0ad3-439c-8a9d-02fa6a9431d9-maas-api-xks-ca\") pod \"659451eb-0ad3-439c-8a9d-02fa6a9431d9\" (UID: \"659451eb-0ad3-439c-8a9d-02fa6a9431d9\") " Oct 30 20:08:30.281490 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:30.281462 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/659451eb-0ad3-439c-8a9d-02fa6a9431d9-maas-api-openshift-ca" pod "659451eb-0ad3-439c-8a9d-02fa6a9431d9" (UID: "659451eb-0ad3-439c-8a9d-02fa6a9431d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:30.281981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:30.281958 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/659451eb-0ad3-439c-8a9d-02fa6a9431d9-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:30.283703 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:30.283664 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/659451eb-0ad3-439c-8a9d-02fa6a9431d9-kube-api-access-756hw" pod "659451eb-0ad3-439c-8a9d-02fa6a9431d9" (UID: "659451eb-0ad3-439c-8a9d-02fa6a9431d9"). InnerVolumeSpecName "kube-api-access-756hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:30.283990 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:30.283966 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/659451eb-0ad3-439c-8a9d-02fa6a9431d9-maas-api-xks-ca" pod "659451eb-0ad3-439c-8a9d-02fa6a9431d9" (UID: "659451eb-0ad3-439c-8a9d-02fa6a9431d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:08:30.383352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:30.383261 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/659451eb-0ad3-439c-8a9d-02fa6a9431d9-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:30.383352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:30.383298 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-756hw\" (UniqueName: \"kubernetes.io/projected/659451eb-0ad3-439c-8a9d-02fa6a9431d9-kube-api-access-756hw\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:30.667225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:30.656831 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:30.898609 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:30.898583 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" Oct 30 20:08:30.898778 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:30.898582 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr" event={"ID":"659451eb-0ad3-439c-8a9d-02fa6a9431d9","Type":"ContainerDied","Data":"0bf063980cfceb3952f4af7ca77452c3bfe6a278256841048e3e19f58f4bcc19"} Oct 30 20:08:30.898778 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:30.898713 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bf063980cfceb3952f4af7ca77452c3bfe6a278256841048e3e19f58f4bcc19" Oct 30 20:08:31.486168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:31.485835 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:32.375943 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:32.375849 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:32.894478 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:32.894443 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:33.256519 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.256484 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:33.365637 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.365605 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp"] Oct 30 20:08:33.366473 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.366457 5091 cpu_manager.go:414] "containerMap: removing container" podUID="659451eb-0ad3-439c-8a9d-02fa6a9431d9" containerName="revoke-keys" Oct 30 20:08:33.366534 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.366474 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="659451eb-0ad3-439c-8a9d-02fa6a9431d9" containerName="revoke-keys" Oct 30 20:08:33.366625 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.366616 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="659451eb-0ad3-439c-8a9d-02fa6a9431d9" containerName="revoke-keys" Oct 30 20:08:33.370606 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.370588 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" Oct 30 20:08:33.379675 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.379476 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:08:33.379675 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.379536 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:08:33.382593 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.382569 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp"] Oct 30 20:08:33.397117 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.397069 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53568d0f-819b-4557-905f-72bdcc424d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f983d766-dmvrp\" (UID: \"53568d0f-819b-4557-905f-72bdcc424d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" Oct 30 20:08:33.397355 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.397330 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkb6l\" (UniqueName: \"kubernetes.io/projected/53568d0f-819b-4557-905f-72bdcc424d97-kube-api-access-zkb6l\") pod \"maas-api-revoke-sub-f983d766-dmvrp\" (UID: \"53568d0f-819b-4557-905f-72bdcc424d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" Oct 30 20:08:33.397452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.397379 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53568d0f-819b-4557-905f-72bdcc424d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f983d766-dmvrp\" (UID: \"53568d0f-819b-4557-905f-72bdcc424d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" Oct 30 20:08:33.498701 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.498664 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkb6l\" (UniqueName: \"kubernetes.io/projected/53568d0f-819b-4557-905f-72bdcc424d97-kube-api-access-zkb6l\") pod \"maas-api-revoke-sub-f983d766-dmvrp\" (UID: \"53568d0f-819b-4557-905f-72bdcc424d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" Oct 30 20:08:33.498905 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.498711 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53568d0f-819b-4557-905f-72bdcc424d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f983d766-dmvrp\" (UID: \"53568d0f-819b-4557-905f-72bdcc424d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" Oct 30 20:08:33.499261 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.499223 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53568d0f-819b-4557-905f-72bdcc424d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f983d766-dmvrp\" (UID: \"53568d0f-819b-4557-905f-72bdcc424d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" Oct 30 20:08:33.499887 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.499860 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53568d0f-819b-4557-905f-72bdcc424d97-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f983d766-dmvrp\" (UID: \"53568d0f-819b-4557-905f-72bdcc424d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" Oct 30 20:08:33.501443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.501418 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53568d0f-819b-4557-905f-72bdcc424d97-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f983d766-dmvrp\" (UID: \"53568d0f-819b-4557-905f-72bdcc424d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" Oct 30 20:08:33.513591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.513526 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkb6l\" (UniqueName: \"kubernetes.io/projected/53568d0f-819b-4557-905f-72bdcc424d97-kube-api-access-zkb6l\") pod \"maas-api-revoke-sub-f983d766-dmvrp\" (UID: \"53568d0f-819b-4557-905f-72bdcc424d97\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" Oct 30 20:08:33.519843 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.519821 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:33.700170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.699851 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" Oct 30 20:08:33.732188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.723752 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm"] Oct 30 20:08:33.832378 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:33.832291 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:34.256585 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:34.256015 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:34.418979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:34.418872 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:34.583320 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:34.579120 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:34.587665 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:08:34.587633 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53568d0f_819b_4557_905f_72bdcc424d97.slice/crio-354a46022245e31d3c3761737d51b3fdecd6d0ec2cbf7c566d88b8eaaa08972b WatchSource:0}: Error finding container 354a46022245e31d3c3761737d51b3fdecd6d0ec2cbf7c566d88b8eaaa08972b: Status 404 returned error can't find the container with id 354a46022245e31d3c3761737d51b3fdecd6d0ec2cbf7c566d88b8eaaa08972b Oct 30 20:08:34.588577 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:34.588553 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp"] Oct 30 20:08:34.849934 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:34.849904 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:35.080136 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:35.080047 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:35.172831 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:35.172782 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" event={"ID":"53568d0f-819b-4557-905f-72bdcc424d97","Type":"ContainerStarted","Data":"7fd889561d439776c6efdcdf06b465b6daa0962f7aa075c5296b2cfc20899976"} Oct 30 20:08:35.173001 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:35.172858 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" event={"ID":"53568d0f-819b-4557-905f-72bdcc424d97","Type":"ContainerStarted","Data":"354a46022245e31d3c3761737d51b3fdecd6d0ec2cbf7c566d88b8eaaa08972b"} Oct 30 20:08:35.203968 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:35.203910 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" podStartSLOduration=2.203894278 podStartE2EDuration="2.203894278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:35.198896258 +0000 UTC m=+2234.894104217" watchObservedRunningTime="2026-10-30 20:08:35.203894278 +0000 UTC m=+2234.899102235" Oct 30 20:08:35.723541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:35.723494 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:35.940259 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:35.932770 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:36.199063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:36.198975 5091 generic.go:356] "Generic (PLEG): container finished" podID="53568d0f-819b-4557-905f-72bdcc424d97" containerID="7fd889561d439776c6efdcdf06b465b6daa0962f7aa075c5296b2cfc20899976" exitCode=0 Oct 30 20:08:36.199380 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:36.199109 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" event={"ID":"53568d0f-819b-4557-905f-72bdcc424d97","Type":"ContainerDied","Data":"7fd889561d439776c6efdcdf06b465b6daa0962f7aa075c5296b2cfc20899976"} Oct 30 20:08:36.571779 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:36.571738 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:36.888221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:36.871469 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:37.388779 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:37.388751 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" Oct 30 20:08:37.509198 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:37.509164 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53568d0f-819b-4557-905f-72bdcc424d97-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53568d0f-819b-4557-905f-72bdcc424d97-maas-api-xks-ca\") pod \"53568d0f-819b-4557-905f-72bdcc424d97\" (UID: \"53568d0f-819b-4557-905f-72bdcc424d97\") " Oct 30 20:08:37.509496 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:37.509473 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53568d0f-819b-4557-905f-72bdcc424d97-kube-api-access-zkb6l\" (UniqueName: \"kubernetes.io/projected/53568d0f-819b-4557-905f-72bdcc424d97-kube-api-access-zkb6l\") pod \"53568d0f-819b-4557-905f-72bdcc424d97\" (UID: \"53568d0f-819b-4557-905f-72bdcc424d97\") " Oct 30 20:08:37.509611 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:37.509597 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53568d0f-819b-4557-905f-72bdcc424d97-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53568d0f-819b-4557-905f-72bdcc424d97-maas-api-openshift-ca\") pod \"53568d0f-819b-4557-905f-72bdcc424d97\" (UID: \"53568d0f-819b-4557-905f-72bdcc424d97\") " Oct 30 20:08:37.509981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:37.509958 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53568d0f-819b-4557-905f-72bdcc424d97-maas-api-openshift-ca" pod "53568d0f-819b-4557-905f-72bdcc424d97" (UID: "53568d0f-819b-4557-905f-72bdcc424d97"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:37.510459 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:37.510368 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53568d0f-819b-4557-905f-72bdcc424d97-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:37.511554 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:37.511534 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53568d0f-819b-4557-905f-72bdcc424d97-maas-api-xks-ca" pod "53568d0f-819b-4557-905f-72bdcc424d97" (UID: "53568d0f-819b-4557-905f-72bdcc424d97"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:08:37.511688 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:37.511666 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53568d0f-819b-4557-905f-72bdcc424d97-kube-api-access-zkb6l" pod "53568d0f-819b-4557-905f-72bdcc424d97" (UID: "53568d0f-819b-4557-905f-72bdcc424d97"). InnerVolumeSpecName "kube-api-access-zkb6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:37.611343 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:37.611311 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkb6l\" (UniqueName: \"kubernetes.io/projected/53568d0f-819b-4557-905f-72bdcc424d97-kube-api-access-zkb6l\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:37.611343 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:37.611341 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53568d0f-819b-4557-905f-72bdcc424d97-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:37.866218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:37.865452 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:38.166367 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.166257 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:38.254056 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.254016 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" event={"ID":"53568d0f-819b-4557-905f-72bdcc424d97","Type":"ContainerDied","Data":"354a46022245e31d3c3761737d51b3fdecd6d0ec2cbf7c566d88b8eaaa08972b"} Oct 30 20:08:38.254315 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.254298 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="354a46022245e31d3c3761737d51b3fdecd6d0ec2cbf7c566d88b8eaaa08972b" Oct 30 20:08:38.254467 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.254455 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp" Oct 30 20:08:38.378731 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.360751 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:38.383806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.383773 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr"] Oct 30 20:08:38.387530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.385919 5091 cpu_manager.go:414] "containerMap: removing container" podUID="53568d0f-819b-4557-905f-72bdcc424d97" containerName="revoke-keys" Oct 30 20:08:38.390788 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.390763 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53568d0f-819b-4557-905f-72bdcc424d97" containerName="revoke-keys" Oct 30 20:08:38.402112 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.393735 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="53568d0f-819b-4557-905f-72bdcc424d97" containerName="revoke-keys" Oct 30 20:08:38.402112 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.401430 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:38.402112 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.401605 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr"] Oct 30 20:08:38.483989 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.483941 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:38.484306 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.484282 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:38.484464 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.484369 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:38.484608 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.484583 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnsr9\" (UniqueName: \"kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-kube-api-access-gnsr9\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:38.586021 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.585988 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:38.586216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.586084 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:38.586216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.586140 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:38.586375 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.586330 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnsr9\" (UniqueName: \"kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-kube-api-access-gnsr9\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:38.586703 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.586662 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:38.586934 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.586910 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:38.588897 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.588874 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:38.602474 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.602455 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnsr9\" (UniqueName: \"kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-kube-api-access-gnsr9\") pod \"payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:38.724694 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.724584 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" podUID="2c69eee1-3140-4420-877f-274bd8d6eca5" containerName="payload-processing" containerID="cri-o://157e05032d897cb4fd6da4c0fbdfc31a7bff839abcd4cbf0b68237879ff0b518" gracePeriod=25 Oct 30 20:08:38.733314 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:38.733286 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:39.061881 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.061844 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:39.098236 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.098204 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr"] Oct 30 20:08:39.101663 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:08:39.101636 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66210f4_c998_4be7_b46a_9b907fe77610.slice/crio-f113cb9846dbb66c4d55a6f3c2e3659ee032a58fa7a5411de54994bf9a09cad0 WatchSource:0}: Error finding container f113cb9846dbb66c4d55a6f3c2e3659ee032a58fa7a5411de54994bf9a09cad0: Status 404 returned error can't find the container with id f113cb9846dbb66c4d55a6f3c2e3659ee032a58fa7a5411de54994bf9a09cad0 Oct 30 20:08:39.107088 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.107071 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:08:39.211898 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.211869 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c69eee1-3140-4420-877f-274bd8d6eca5-kube-api-access-db2rw\" (UniqueName: \"kubernetes.io/projected/2c69eee1-3140-4420-877f-274bd8d6eca5-kube-api-access-db2rw\") pod \"2c69eee1-3140-4420-877f-274bd8d6eca5\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " Oct 30 20:08:39.212392 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.211972 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c69eee1-3140-4420-877f-274bd8d6eca5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c69eee1-3140-4420-877f-274bd8d6eca5-plugins-config-volume\") pod \"2c69eee1-3140-4420-877f-274bd8d6eca5\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " Oct 30 20:08:39.212392 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.212098 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c69eee1-3140-4420-877f-274bd8d6eca5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c69eee1-3140-4420-877f-274bd8d6eca5-routing-overlay-volume\") pod \"2c69eee1-3140-4420-877f-274bd8d6eca5\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " Oct 30 20:08:39.212509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.212414 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c69eee1-3140-4420-877f-274bd8d6eca5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c69eee1-3140-4420-877f-274bd8d6eca5-provider-credentials\") pod \"2c69eee1-3140-4420-877f-274bd8d6eca5\" (UID: \"2c69eee1-3140-4420-877f-274bd8d6eca5\") " Oct 30 20:08:39.212509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.212430 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c69eee1-3140-4420-877f-274bd8d6eca5-plugins-config-volume" pod "2c69eee1-3140-4420-877f-274bd8d6eca5" (UID: "2c69eee1-3140-4420-877f-274bd8d6eca5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:39.212509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.212455 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c69eee1-3140-4420-877f-274bd8d6eca5-routing-overlay-volume" pod "2c69eee1-3140-4420-877f-274bd8d6eca5" (UID: "2c69eee1-3140-4420-877f-274bd8d6eca5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:39.213091 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.213066 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2c69eee1-3140-4420-877f-274bd8d6eca5-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:39.213269 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.213097 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c69eee1-3140-4420-877f-274bd8d6eca5-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:39.214587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.214559 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c69eee1-3140-4420-877f-274bd8d6eca5-kube-api-access-db2rw" pod "2c69eee1-3140-4420-877f-274bd8d6eca5" (UID: "2c69eee1-3140-4420-877f-274bd8d6eca5"). InnerVolumeSpecName "kube-api-access-db2rw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:39.214682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.214612 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c69eee1-3140-4420-877f-274bd8d6eca5-provider-credentials" pod "2c69eee1-3140-4420-877f-274bd8d6eca5" (UID: "2c69eee1-3140-4420-877f-274bd8d6eca5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:39.265184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.265039 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" event={"ID":"b66210f4-c998-4be7-b46a-9b907fe77610","Type":"ContainerStarted","Data":"639dc31cdd193781d4ffabd7a30b259d83c9ad97aa34bee7f32dfd94ac8e7f21"} Oct 30 20:08:39.265184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.265106 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" event={"ID":"b66210f4-c998-4be7-b46a-9b907fe77610","Type":"ContainerStarted","Data":"f113cb9846dbb66c4d55a6f3c2e3659ee032a58fa7a5411de54994bf9a09cad0"} Oct 30 20:08:39.265184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.265177 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:39.267968 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.267934 5091 generic.go:356] "Generic (PLEG): container finished" podID="2c69eee1-3140-4420-877f-274bd8d6eca5" containerID="157e05032d897cb4fd6da4c0fbdfc31a7bff839abcd4cbf0b68237879ff0b518" exitCode=0 Oct 30 20:08:39.268107 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.268057 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" event={"ID":"2c69eee1-3140-4420-877f-274bd8d6eca5","Type":"ContainerDied","Data":"157e05032d897cb4fd6da4c0fbdfc31a7bff839abcd4cbf0b68237879ff0b518"} Oct 30 20:08:39.268107 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.268100 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" event={"ID":"2c69eee1-3140-4420-877f-274bd8d6eca5","Type":"ContainerDied","Data":"c44336861a89baa7b1ec2ff01d500d526bdde57b0ab9a4a2c5f6000b80ccdfa3"} Oct 30 20:08:39.268300 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.268124 5091 scope.go:122] "RemoveContainer" containerID="157e05032d897cb4fd6da4c0fbdfc31a7bff839abcd4cbf0b68237879ff0b518" Oct 30 20:08:39.268352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.268334 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm" Oct 30 20:08:39.313041 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.312939 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" podStartSLOduration=1.3129204780000001 podStartE2EDuration="1.312920478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:39.30667966 +0000 UTC m=+2239.001887629" watchObservedRunningTime="2026-10-30 20:08:39.312920478 +0000 UTC m=+2239.008128437" Oct 30 20:08:39.314958 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.314929 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-db2rw\" (UniqueName: \"kubernetes.io/projected/2c69eee1-3140-4420-877f-274bd8d6eca5-kube-api-access-db2rw\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:39.314958 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.314960 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2c69eee1-3140-4420-877f-274bd8d6eca5-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:39.315660 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:39.315637 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-517027/agc-cred-7cc1575beefa: secret "agc-cred-7cc1575beefa" not found Oct 30 20:08:39.315660 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:39.315659 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr: secret "agc-cred-7cc1575beefa" not found Oct 30 20:08:39.315843 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:39.315737 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-provider-credentials podName:b66210f4-c998-4be7-b46a-9b907fe77610 nodeName:}" failed. No retries permitted until 2026-10-30 20:08:39.815719069 +0000 UTC m=+2239.510927018 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" (UID: "b66210f4-c998-4be7-b46a-9b907fe77610") : secret "agc-cred-7cc1575beefa" not found Oct 30 20:08:39.321925 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.321902 5091 scope.go:122] "RemoveContainer" containerID="157e05032d897cb4fd6da4c0fbdfc31a7bff839abcd4cbf0b68237879ff0b518" Oct 30 20:08:39.322536 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:39.322504 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"157e05032d897cb4fd6da4c0fbdfc31a7bff839abcd4cbf0b68237879ff0b518\": container with ID starting with 157e05032d897cb4fd6da4c0fbdfc31a7bff839abcd4cbf0b68237879ff0b518 not found: ID does not exist" containerID="157e05032d897cb4fd6da4c0fbdfc31a7bff839abcd4cbf0b68237879ff0b518" Oct 30 20:08:39.322634 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.322547 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"157e05032d897cb4fd6da4c0fbdfc31a7bff839abcd4cbf0b68237879ff0b518"} err="failed to get container status \"157e05032d897cb4fd6da4c0fbdfc31a7bff839abcd4cbf0b68237879ff0b518\": rpc error: code = NotFound desc = could not find container \"157e05032d897cb4fd6da4c0fbdfc31a7bff839abcd4cbf0b68237879ff0b518\": container with ID starting with 157e05032d897cb4fd6da4c0fbdfc31a7bff839abcd4cbf0b68237879ff0b518 not found: ID does not exist" Oct 30 20:08:39.329436 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.329410 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6"] Oct 30 20:08:39.335238 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.335210 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr"] Oct 30 20:08:39.340435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.340407 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm"] Oct 30 20:08:39.345068 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.345043 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-67ff9xrqm"] Oct 30 20:08:39.394951 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.394919 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:39.596014 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:39.595930 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:39.894127 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:39.893593 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-517027/agc-cred-7cc1575beefa: secret "agc-cred-7cc1575beefa" not found Oct 30 20:08:39.894127 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:39.893625 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr: secret "agc-cred-7cc1575beefa" not found Oct 30 20:08:39.930165 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:39.910166 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-provider-credentials podName:b66210f4-c998-4be7-b46a-9b907fe77610 nodeName:}" failed. No retries permitted until 2026-10-30 20:08:40.91012077 +0000 UTC m=+2240.605328721 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" (UID: "b66210f4-c998-4be7-b46a-9b907fe77610") : secret "agc-cred-7cc1575beefa" not found Oct 30 20:08:40.035738 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:40.035705 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:40.340807 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:40.340763 5091 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" secret="" err="secret \"payload-processing-external-model-e2e--c9e3857f-dockercfg-dkbs4\" not found" Oct 30 20:08:40.440391 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:40.439719 5091 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-plugins-e2e-extmod-a-517027: configmap "payload-processing-external-model-plugins-e2e-extmod-a-517027" not found Oct 30 20:08:40.440391 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:40.439796 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-plugins-config-volume podName:b66210f4-c998-4be7-b46a-9b907fe77610 nodeName:}" failed. No retries permitted until 2026-10-30 20:08:40.939776559 +0000 UTC m=+2240.634984508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" (UID: "b66210f4-c998-4be7-b46a-9b907fe77610") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-517027" not found Oct 30 20:08:40.441764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:40.441734 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:40.445559 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:40.445538 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c69eee1-3140-4420-877f-274bd8d6eca5" path="/var/lib/kubelet/pods/2c69eee1-3140-4420-877f-274bd8d6eca5/volumes" Oct 30 20:08:40.945620 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:40.945588 5091 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-plugins-e2e-extmod-a-517027: configmap "payload-processing-external-model-plugins-e2e-extmod-a-517027" not found Oct 30 20:08:40.945802 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:40.945672 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-plugins-config-volume podName:b66210f4-c998-4be7-b46a-9b907fe77610 nodeName:}" failed. No retries permitted until 2026-10-30 20:08:41.945652509 +0000 UTC m=+2241.640860459 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" (UID: "b66210f4-c998-4be7-b46a-9b907fe77610") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-517027" not found Oct 30 20:08:40.945802 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:40.945678 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-517027/agc-cred-7cc1575beefa: secret "agc-cred-7cc1575beefa" not found Oct 30 20:08:40.945802 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:40.945700 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr: secret "agc-cred-7cc1575beefa" not found Oct 30 20:08:40.945802 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:40.945755 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-provider-credentials podName:b66210f4-c998-4be7-b46a-9b907fe77610 nodeName:}" failed. No retries permitted until 2026-10-30 20:08:42.945743401 +0000 UTC m=+2242.640951336 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" (UID: "b66210f4-c998-4be7-b46a-9b907fe77610") : secret "agc-cred-7cc1575beefa" not found Oct 30 20:08:41.311134 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:41.311095 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:41.960033 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:41.959997 5091 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-plugins-e2e-extmod-a-517027: configmap "payload-processing-external-model-plugins-e2e-extmod-a-517027" not found Oct 30 20:08:41.960551 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:41.960067 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-plugins-config-volume podName:b66210f4-c998-4be7-b46a-9b907fe77610 nodeName:}" failed. No retries permitted until 2026-10-30 20:08:43.960052931 +0000 UTC m=+2243.655260870 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" (UID: "b66210f4-c998-4be7-b46a-9b907fe77610") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-517027" not found Oct 30 20:08:42.361407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:42.361361 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:42.978303 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:42.978266 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-517027/agc-cred-7cc1575beefa: secret "agc-cred-7cc1575beefa" not found Oct 30 20:08:42.978303 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:42.978297 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr: secret "agc-cred-7cc1575beefa" not found Oct 30 20:08:42.979139 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:42.978361 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-provider-credentials podName:b66210f4-c998-4be7-b46a-9b907fe77610 nodeName:}" failed. No retries permitted until 2026-10-30 20:08:46.978343856 +0000 UTC m=+2246.673551800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" (UID: "b66210f4-c998-4be7-b46a-9b907fe77610") : secret "agc-cred-7cc1575beefa" not found Oct 30 20:08:43.209942 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:43.209902 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:44.039216 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:44.011913 5091 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-plugins-e2e-extmod-a-517027: configmap "payload-processing-external-model-plugins-e2e-extmod-a-517027" not found Oct 30 20:08:44.039216 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:44.012009 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-plugins-config-volume podName:b66210f4-c998-4be7-b46a-9b907fe77610 nodeName:}" failed. No retries permitted until 2026-10-30 20:08:48.011987027 +0000 UTC m=+2247.707194983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" (UID: "b66210f4-c998-4be7-b46a-9b907fe77610") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-517027" not found Oct 30 20:08:44.063171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.058738 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:44.330355 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.330267 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" podUID="7f77f10e-35b0-4413-88a9-0252cc4cdd86" containerName="payload-processing" containerID="cri-o://0f6295ef9ce886ff962f49786730cc80200a607571c6540cca304c4ef0d8426e" gracePeriod=25 Oct 30 20:08:44.505223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.504996 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:44.512414 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:08:44.512386 5091 logging.go:55] [core] [Channel #5955 SubChannel #5956]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.42:50052", ServerName: "10.132.1.42:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.42:50052: connect: connection refused" Oct 30 20:08:44.641011 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.640980 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:08:44.697595 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.697560 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:44.724343 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.724307 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f77f10e-35b0-4413-88a9-0252cc4cdd86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f77f10e-35b0-4413-88a9-0252cc4cdd86-plugins-config-volume\") pod \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " Oct 30 20:08:44.724499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.724374 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f77f10e-35b0-4413-88a9-0252cc4cdd86-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f77f10e-35b0-4413-88a9-0252cc4cdd86-routing-overlay-volume\") pod \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " Oct 30 20:08:44.724580 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.724518 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f77f10e-35b0-4413-88a9-0252cc4cdd86-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f77f10e-35b0-4413-88a9-0252cc4cdd86-provider-credentials\") pod \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " Oct 30 20:08:44.724654 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.724591 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f77f10e-35b0-4413-88a9-0252cc4cdd86-kube-api-access-nlrbc\" (UniqueName: \"kubernetes.io/projected/7f77f10e-35b0-4413-88a9-0252cc4cdd86-kube-api-access-nlrbc\") pod \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\" (UID: \"7f77f10e-35b0-4413-88a9-0252cc4cdd86\") " Oct 30 20:08:44.724821 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.724795 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f77f10e-35b0-4413-88a9-0252cc4cdd86-routing-overlay-volume" pod "7f77f10e-35b0-4413-88a9-0252cc4cdd86" (UID: "7f77f10e-35b0-4413-88a9-0252cc4cdd86"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:44.724910 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.724817 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f77f10e-35b0-4413-88a9-0252cc4cdd86-plugins-config-volume" pod "7f77f10e-35b0-4413-88a9-0252cc4cdd86" (UID: "7f77f10e-35b0-4413-88a9-0252cc4cdd86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:44.725690 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.725665 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f77f10e-35b0-4413-88a9-0252cc4cdd86-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:44.725799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.725695 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7f77f10e-35b0-4413-88a9-0252cc4cdd86-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:44.726938 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.726917 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f77f10e-35b0-4413-88a9-0252cc4cdd86-provider-credentials" pod "7f77f10e-35b0-4413-88a9-0252cc4cdd86" (UID: "7f77f10e-35b0-4413-88a9-0252cc4cdd86"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:44.727495 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.727475 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f77f10e-35b0-4413-88a9-0252cc4cdd86-kube-api-access-nlrbc" pod "7f77f10e-35b0-4413-88a9-0252cc4cdd86" (UID: "7f77f10e-35b0-4413-88a9-0252cc4cdd86"). InnerVolumeSpecName "kube-api-access-nlrbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:44.848169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.841207 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7f77f10e-35b0-4413-88a9-0252cc4cdd86-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:44.848169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.841255 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlrbc\" (UniqueName: \"kubernetes.io/projected/7f77f10e-35b0-4413-88a9-0252cc4cdd86-kube-api-access-nlrbc\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:44.893222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:44.892741 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:45.245227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.207104 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:45.440176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.433406 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h"] Oct 30 20:08:45.524910 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.522326 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" podUID="7f77f10e-35b0-4413-88a9-0252cc4cdd86" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.42:50052\" within 1s: context deadline exceeded" Oct 30 20:08:45.524910 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:08:45.522656 5091 logging.go:55] [core] [Channel #5955 SubChannel #5956]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.42:50052", ServerName: "10.132.1.42:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.42:50052: operation was canceled" Oct 30 20:08:45.525733 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.525701 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2c69eee1-3140-4420-877f-274bd8d6eca5" containerName="payload-processing" Oct 30 20:08:45.525900 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.525885 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c69eee1-3140-4420-877f-274bd8d6eca5" containerName="payload-processing" Oct 30 20:08:45.526042 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.526030 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7f77f10e-35b0-4413-88a9-0252cc4cdd86" containerName="payload-processing" Oct 30 20:08:45.526134 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.526123 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f77f10e-35b0-4413-88a9-0252cc4cdd86" containerName="payload-processing" Oct 30 20:08:45.532196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.526592 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c69eee1-3140-4420-877f-274bd8d6eca5" containerName="payload-processing" Oct 30 20:08:45.563245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.563211 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f77f10e-35b0-4413-88a9-0252cc4cdd86" containerName="payload-processing" Oct 30 20:08:45.703842 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.703682 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:45.748170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.744476 5091 generic.go:356] "Generic (PLEG): container finished" podID="7f77f10e-35b0-4413-88a9-0252cc4cdd86" containerID="0f6295ef9ce886ff962f49786730cc80200a607571c6540cca304c4ef0d8426e" exitCode=0 Oct 30 20:08:45.748170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.744847 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" Oct 30 20:08:45.826710 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.824309 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h"] Oct 30 20:08:45.895582 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.895545 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" event={"ID":"7f77f10e-35b0-4413-88a9-0252cc4cdd86","Type":"ContainerDied","Data":"0f6295ef9ce886ff962f49786730cc80200a607571c6540cca304c4ef0d8426e"} Oct 30 20:08:45.895824 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.895799 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6" event={"ID":"7f77f10e-35b0-4413-88a9-0252cc4cdd86","Type":"ContainerDied","Data":"caace1ba8ddf29b3bac34feb6305b1626ba20789ced436542e6164a1afa26a76"} Oct 30 20:08:45.896751 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.843407 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:45.897337 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.897320 5091 scope.go:122] "RemoveContainer" containerID="0f6295ef9ce886ff962f49786730cc80200a607571c6540cca304c4ef0d8426e" Oct 30 20:08:45.917718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.908086 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"agc-cred-8f19d92ff4bd\"" Oct 30 20:08:45.917718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.908801 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"payload-processing-external-model-e2e--babb2686-dockercfg-c22qm\"" Oct 30 20:08:45.917718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.914551 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"kube-root-ca.crt\"" Oct 30 20:08:45.917718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.914622 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"payload-processing-external-model-plugins-e2e-extmod-a-68685b\"" Oct 30 20:08:45.917718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.914679 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"openshift-service-ca.crt\"" Oct 30 20:08:45.917718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.914737 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-68685b\"/\"routing-overlay\"" Oct 30 20:08:45.925173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.922204 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:45.934812 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.934771 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h"] Oct 30 20:08:45.944030 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:45.943994 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-74mps plugins-config-volume provider-credentials routing-overlay-volume], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" podUID="b13acecb-716d-4f9f-b00e-ccd80316a7ad" Oct 30 20:08:45.953870 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.953849 5091 scope.go:122] "RemoveContainer" containerID="0f6295ef9ce886ff962f49786730cc80200a607571c6540cca304c4ef0d8426e" Oct 30 20:08:45.954275 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:45.954233 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f6295ef9ce886ff962f49786730cc80200a607571c6540cca304c4ef0d8426e\": container with ID starting with 0f6295ef9ce886ff962f49786730cc80200a607571c6540cca304c4ef0d8426e not found: ID does not exist" containerID="0f6295ef9ce886ff962f49786730cc80200a607571c6540cca304c4ef0d8426e" Oct 30 20:08:45.954370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.954277 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f6295ef9ce886ff962f49786730cc80200a607571c6540cca304c4ef0d8426e"} err="failed to get container status \"0f6295ef9ce886ff962f49786730cc80200a607571c6540cca304c4ef0d8426e\": rpc error: code = NotFound desc = could not find container \"0f6295ef9ce886ff962f49786730cc80200a607571c6540cca304c4ef0d8426e\": container with ID starting with 0f6295ef9ce886ff962f49786730cc80200a607571c6540cca304c4ef0d8426e not found: ID does not exist" Oct 30 20:08:45.974009 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.973977 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6"] Oct 30 20:08:45.994431 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:45.990494 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-6d48fd7x6"] Oct 30 20:08:46.049170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.048659 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74mps\" (UniqueName: \"kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-kube-api-access-74mps\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:46.049170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.049036 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:46.049454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.049296 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:46.049454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.049367 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:46.147247 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.126795 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:08:46.157169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.156778 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:46.157169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.156926 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:46.157169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.156978 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:46.157169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.157158 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74mps\" (UniqueName: \"kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-kube-api-access-74mps\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:46.167231 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.157770 5091 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-plugins-e2e-extmod-a-68685b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-68685b" not found Oct 30 20:08:46.167231 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.157845 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-plugins-config-volume podName:b13acecb-716d-4f9f-b00e-ccd80316a7ad nodeName:}" failed. No retries permitted until 2026-10-30 20:08:46.657824338 +0000 UTC m=+2246.353032296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" (UID: "b13acecb-716d-4f9f-b00e-ccd80316a7ad") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-68685b" not found Oct 30 20:08:46.167231 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.158843 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:46.189452 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.187299 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-68685b/agc-cred-8f19d92ff4bd: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:08:46.189452 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.187332 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:08:46.189452 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.187394 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-provider-credentials podName:b13acecb-716d-4f9f-b00e-ccd80316a7ad nodeName:}" failed. No retries permitted until 2026-10-30 20:08:46.687373873 +0000 UTC m=+2246.382581822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" (UID: "b13acecb-716d-4f9f-b00e-ccd80316a7ad") : secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:08:46.189452 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.189439 5091 projected.go:196] Error preparing data for projected volume kube-api-access-74mps for pod ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h: failed to fetch token: serviceaccounts "payload-processing-external-model-e2e-extmod-a-68685b" not found Oct 30 20:08:46.189834 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.189506 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-kube-api-access-74mps podName:b13acecb-716d-4f9f-b00e-ccd80316a7ad nodeName:}" failed. No retries permitted until 2026-10-30 20:08:46.689486143 +0000 UTC m=+2246.384694086 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-74mps" (UniqueName: "kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-kube-api-access-74mps") pod "payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" (UID: "b13acecb-716d-4f9f-b00e-ccd80316a7ad") : failed to fetch token: serviceaccounts "payload-processing-external-model-e2e-extmod-a-68685b" not found Oct 30 20:08:46.328776 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.328385 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:46.351283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.351249 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" podUID="b66210f4-c998-4be7-b46a-9b907fe77610" containerName="payload-processing" containerID="cri-o://639dc31cdd193781d4ffabd7a30b259d83c9ad97aa34bee7f32dfd94ac8e7f21" gracePeriod=25 Oct 30 20:08:46.444608 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.444526 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f77f10e-35b0-4413-88a9-0252cc4cdd86" path="/var/lib/kubelet/pods/7f77f10e-35b0-4413-88a9-0252cc4cdd86/volumes" Oct 30 20:08:46.651625 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.651592 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:46.657255 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.657230 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:46.669640 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.669618 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-plugins-config-volume\") pod \"b66210f4-c998-4be7-b46a-9b907fe77610\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " Oct 30 20:08:46.669772 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.669696 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-routing-overlay-volume\") pod \"b66210f4-c998-4be7-b46a-9b907fe77610\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " Oct 30 20:08:46.669901 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.669883 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-provider-credentials\") pod \"b66210f4-c998-4be7-b46a-9b907fe77610\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " Oct 30 20:08:46.669974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.669940 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-kube-api-access-gnsr9\" (UniqueName: \"kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-kube-api-access-gnsr9\") pod \"b66210f4-c998-4be7-b46a-9b907fe77610\" (UID: \"b66210f4-c998-4be7-b46a-9b907fe77610\") " Oct 30 20:08:46.670987 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.670128 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-routing-overlay-volume" pod "b66210f4-c998-4be7-b46a-9b907fe77610" (UID: "b66210f4-c998-4be7-b46a-9b907fe77610"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:46.670987 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.670206 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-plugins-config-volume" pod "b66210f4-c998-4be7-b46a-9b907fe77610" (UID: "b66210f4-c998-4be7-b46a-9b907fe77610"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:46.670987 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.670707 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:46.670987 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.670934 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:46.670987 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.670961 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b66210f4-c998-4be7-b46a-9b907fe77610-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:46.671405 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.671023 5091 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-plugins-e2e-extmod-a-68685b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-68685b" not found Oct 30 20:08:46.671405 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.671080 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-plugins-config-volume podName:b13acecb-716d-4f9f-b00e-ccd80316a7ad nodeName:}" failed. No retries permitted until 2026-10-30 20:08:47.671062023 +0000 UTC m=+2247.366269976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" (UID: "b13acecb-716d-4f9f-b00e-ccd80316a7ad") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-68685b" not found Oct 30 20:08:46.672858 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.672823 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-provider-credentials" pod "b66210f4-c998-4be7-b46a-9b907fe77610" (UID: "b66210f4-c998-4be7-b46a-9b907fe77610"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:46.673072 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.673056 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-kube-api-access-gnsr9" pod "b66210f4-c998-4be7-b46a-9b907fe77610" (UID: "b66210f4-c998-4be7-b46a-9b907fe77610"). InnerVolumeSpecName "kube-api-access-gnsr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:08:46.757556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.757518 5091 generic.go:356] "Generic (PLEG): container finished" podID="b66210f4-c998-4be7-b46a-9b907fe77610" containerID="639dc31cdd193781d4ffabd7a30b259d83c9ad97aa34bee7f32dfd94ac8e7f21" exitCode=0 Oct 30 20:08:46.757810 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.757569 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" Oct 30 20:08:46.757810 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.757597 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" event={"ID":"b66210f4-c998-4be7-b46a-9b907fe77610","Type":"ContainerDied","Data":"639dc31cdd193781d4ffabd7a30b259d83c9ad97aa34bee7f32dfd94ac8e7f21"} Oct 30 20:08:46.757810 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.757669 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr" event={"ID":"b66210f4-c998-4be7-b46a-9b907fe77610","Type":"ContainerDied","Data":"f113cb9846dbb66c4d55a6f3c2e3659ee032a58fa7a5411de54994bf9a09cad0"} Oct 30 20:08:46.757810 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.757691 5091 scope.go:122] "RemoveContainer" containerID="639dc31cdd193781d4ffabd7a30b259d83c9ad97aa34bee7f32dfd94ac8e7f21" Oct 30 20:08:46.759159 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.759125 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:46.772781 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.772760 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:46.772914 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.772864 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74mps\" (UniqueName: \"kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-kube-api-access-74mps\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:46.773228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.773208 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:46.773349 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.773338 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-68685b/agc-cred-8f19d92ff4bd: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:08:46.773410 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.773352 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h: secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:08:46.773410 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.773354 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:46.773410 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.773378 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnsr9\" (UniqueName: \"kubernetes.io/projected/b66210f4-c998-4be7-b46a-9b907fe77610-kube-api-access-gnsr9\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:46.773410 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.773394 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-provider-credentials podName:b13acecb-716d-4f9f-b00e-ccd80316a7ad nodeName:}" failed. No retries permitted until 2026-10-30 20:08:47.773381568 +0000 UTC m=+2247.468589504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" (UID: "b13acecb-716d-4f9f-b00e-ccd80316a7ad") : secret "agc-cred-8f19d92ff4bd" not found Oct 30 20:08:46.774586 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.774564 5091 scope.go:122] "RemoveContainer" containerID="639dc31cdd193781d4ffabd7a30b259d83c9ad97aa34bee7f32dfd94ac8e7f21" Oct 30 20:08:46.774868 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.774847 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"639dc31cdd193781d4ffabd7a30b259d83c9ad97aa34bee7f32dfd94ac8e7f21\": container with ID starting with 639dc31cdd193781d4ffabd7a30b259d83c9ad97aa34bee7f32dfd94ac8e7f21 not found: ID does not exist" containerID="639dc31cdd193781d4ffabd7a30b259d83c9ad97aa34bee7f32dfd94ac8e7f21" Oct 30 20:08:46.774944 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.774880 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"639dc31cdd193781d4ffabd7a30b259d83c9ad97aa34bee7f32dfd94ac8e7f21"} err="failed to get container status \"639dc31cdd193781d4ffabd7a30b259d83c9ad97aa34bee7f32dfd94ac8e7f21\": rpc error: code = NotFound desc = could not find container \"639dc31cdd193781d4ffabd7a30b259d83c9ad97aa34bee7f32dfd94ac8e7f21\": container with ID starting with 639dc31cdd193781d4ffabd7a30b259d83c9ad97aa34bee7f32dfd94ac8e7f21 not found: ID does not exist" Oct 30 20:08:46.779131 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.779112 5091 projected.go:196] Error preparing data for projected volume kube-api-access-74mps for pod ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h: failed to fetch token: serviceaccounts "payload-processing-external-model-e2e-extmod-a-68685b" not found Oct 30 20:08:46.779246 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:46.779189 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-kube-api-access-74mps podName:b13acecb-716d-4f9f-b00e-ccd80316a7ad nodeName:}" failed. No retries permitted until 2026-10-30 20:08:47.779171767 +0000 UTC m=+2247.474379716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-74mps" (UniqueName: "kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-kube-api-access-74mps") pod "payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" (UID: "b13acecb-716d-4f9f-b00e-ccd80316a7ad") : failed to fetch token: serviceaccounts "payload-processing-external-model-e2e-extmod-a-68685b" not found Oct 30 20:08:46.847211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.847128 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr"] Oct 30 20:08:46.851785 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.851762 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-517027/payload-processing-external-model-e2e-extmod-a-517027-cb58h6nqr"] Oct 30 20:08:46.875243 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.875210 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-routing-overlay-volume\") pod \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " Oct 30 20:08:46.875650 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.875619 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-routing-overlay-volume" pod "b13acecb-716d-4f9f-b00e-ccd80316a7ad" (UID: "b13acecb-716d-4f9f-b00e-ccd80316a7ad"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:08:46.875986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.875966 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:46.962731 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:46.962673 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:47.689009 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:47.688963 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:47.689542 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:47.689078 5091 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-plugins-e2e-extmod-a-68685b: configmap "payload-processing-external-model-plugins-e2e-extmod-a-68685b" not found Oct 30 20:08:47.689542 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:47.689158 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-plugins-config-volume podName:b13acecb-716d-4f9f-b00e-ccd80316a7ad nodeName:}" failed. No retries permitted until 2026-10-30 20:08:49.689132286 +0000 UTC m=+2249.384340223 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" (UID: "b13acecb-716d-4f9f-b00e-ccd80316a7ad") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-68685b" not found Oct 30 20:08:47.770861 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:47.770825 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:47.790139 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:47.790102 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:47.790329 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:47.790217 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74mps\" (UniqueName: \"kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-kube-api-access-74mps\") pod \"payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h\" (UID: \"b13acecb-716d-4f9f-b00e-ccd80316a7ad\") " pod="ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" Oct 30 20:08:47.790329 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:47.790248 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-68685b/agc-cred-8f19d92ff4bd: object "ai-tenant-e2e-extmod-a-68685b"/"agc-cred-8f19d92ff4bd" not registered Oct 30 20:08:47.790329 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:47.790268 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h: object "ai-tenant-e2e-extmod-a-68685b"/"agc-cred-8f19d92ff4bd" not registered Oct 30 20:08:47.790329 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:47.790320 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-provider-credentials podName:b13acecb-716d-4f9f-b00e-ccd80316a7ad nodeName:}" failed. No retries permitted until 2026-10-30 20:08:49.790305045 +0000 UTC m=+2249.485512986 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" (UID: "b13acecb-716d-4f9f-b00e-ccd80316a7ad") : object "ai-tenant-e2e-extmod-a-68685b"/"agc-cred-8f19d92ff4bd" not registered Oct 30 20:08:47.795700 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:47.795676 5091 projected.go:291] Couldn't get configMap ai-tenant-e2e-extmod-a-68685b/kube-root-ca.crt: object "ai-tenant-e2e-extmod-a-68685b"/"kube-root-ca.crt" not registered Oct 30 20:08:47.795700 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:47.795698 5091 projected.go:291] Couldn't get configMap ai-tenant-e2e-extmod-a-68685b/openshift-service-ca.crt: object "ai-tenant-e2e-extmod-a-68685b"/"openshift-service-ca.crt" not registered Oct 30 20:08:47.795901 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:47.795709 5091 projected.go:196] Error preparing data for projected volume kube-api-access-74mps for pod ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h: [failed to fetch token: serviceaccounts "payload-processing-external-model-e2e-extmod-a-68685b" not found, object "ai-tenant-e2e-extmod-a-68685b"/"kube-root-ca.crt" not registered, object "ai-tenant-e2e-extmod-a-68685b"/"openshift-service-ca.crt" not registered] Oct 30 20:08:47.795901 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:08:47.795760 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-kube-api-access-74mps podName:b13acecb-716d-4f9f-b00e-ccd80316a7ad nodeName:}" failed. No retries permitted until 2026-10-30 20:08:49.795744834 +0000 UTC m=+2249.490952783 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-74mps" (UniqueName: "kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-kube-api-access-74mps") pod "payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h" (UID: "b13acecb-716d-4f9f-b00e-ccd80316a7ad") : [failed to fetch token: serviceaccounts "payload-processing-external-model-e2e-extmod-a-68685b" not found, object "ai-tenant-e2e-extmod-a-68685b"/"kube-root-ca.crt" not registered, object "ai-tenant-e2e-extmod-a-68685b"/"openshift-service-ca.crt" not registered] Oct 30 20:08:47.850017 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:47.849981 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h"] Oct 30 20:08:47.863463 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:47.863416 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-68685b/payload-processing-external-model-e2e-extmod-a-68685b-6dd4s2t6h"] Oct 30 20:08:47.891917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:47.891875 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74mps\" (UniqueName: \"kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-kube-api-access-74mps\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:47.891917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:47.891906 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b13acecb-716d-4f9f-b00e-ccd80316a7ad-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:47.891917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:47.891915 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b13acecb-716d-4f9f-b00e-ccd80316a7ad-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:08:48.135050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:48.135010 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:48.443444 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:48.443365 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b13acecb-716d-4f9f-b00e-ccd80316a7ad" path="/var/lib/kubelet/pods/b13acecb-716d-4f9f-b00e-ccd80316a7ad/volumes" Oct 30 20:08:48.443757 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:48.443742 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66210f4-c998-4be7-b46a-9b907fe77610" path="/var/lib/kubelet/pods/b66210f4-c998-4be7-b46a-9b907fe77610/volumes" Oct 30 20:08:49.053021 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:49.052980 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:49.941722 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:49.941683 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:50.138197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:50.134089 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:51.009168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:51.006663 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:51.937695 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:51.937627 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:52.848104 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:52.848070 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:53.968725 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:53.968687 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:54.910390 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:54.910346 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:55.256222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:55.255062 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:55.600208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:55.600078 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:56.507603 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:56.507558 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:56.909028 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:56.908938 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j"] Oct 30 20:08:56.909794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:56.909775 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b66210f4-c998-4be7-b46a-9b907fe77610" containerName="payload-processing" Oct 30 20:08:56.909852 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:56.909796 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66210f4-c998-4be7-b46a-9b907fe77610" containerName="payload-processing" Oct 30 20:08:56.909935 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:56.909926 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66210f4-c998-4be7-b46a-9b907fe77610" containerName="payload-processing" Oct 30 20:08:56.915085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:56.915062 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" Oct 30 20:08:56.923784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:56.923756 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:08:56.924030 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:56.923886 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:08:56.933450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:56.933420 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j"] Oct 30 20:08:56.965741 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:56.965701 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vx45\" (UniqueName: \"kubernetes.io/projected/6e168a95-a7a1-469f-8222-d8437ed038ed-kube-api-access-6vx45\") pod \"maas-api-revoke-sub-80fb8b78-6f58j\" (UID: \"6e168a95-a7a1-469f-8222-d8437ed038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" Oct 30 20:08:56.965917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:56.965868 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e168a95-a7a1-469f-8222-d8437ed038ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80fb8b78-6f58j\" (UID: \"6e168a95-a7a1-469f-8222-d8437ed038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" Oct 30 20:08:56.965917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:56.965903 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e168a95-a7a1-469f-8222-d8437ed038ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80fb8b78-6f58j\" (UID: \"6e168a95-a7a1-469f-8222-d8437ed038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" Oct 30 20:08:57.067986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.067938 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vx45\" (UniqueName: \"kubernetes.io/projected/6e168a95-a7a1-469f-8222-d8437ed038ed-kube-api-access-6vx45\") pod \"maas-api-revoke-sub-80fb8b78-6f58j\" (UID: \"6e168a95-a7a1-469f-8222-d8437ed038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" Oct 30 20:08:57.068194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.068072 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e168a95-a7a1-469f-8222-d8437ed038ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80fb8b78-6f58j\" (UID: \"6e168a95-a7a1-469f-8222-d8437ed038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" Oct 30 20:08:57.068194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.068109 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e168a95-a7a1-469f-8222-d8437ed038ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80fb8b78-6f58j\" (UID: \"6e168a95-a7a1-469f-8222-d8437ed038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" Oct 30 20:08:57.068789 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.068759 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e168a95-a7a1-469f-8222-d8437ed038ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80fb8b78-6f58j\" (UID: \"6e168a95-a7a1-469f-8222-d8437ed038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" Oct 30 20:08:57.070827 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.070796 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e168a95-a7a1-469f-8222-d8437ed038ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80fb8b78-6f58j\" (UID: \"6e168a95-a7a1-469f-8222-d8437ed038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" Oct 30 20:08:57.084312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.084285 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vx45\" (UniqueName: \"kubernetes.io/projected/6e168a95-a7a1-469f-8222-d8437ed038ed-kube-api-access-6vx45\") pod \"maas-api-revoke-sub-80fb8b78-6f58j\" (UID: \"6e168a95-a7a1-469f-8222-d8437ed038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" Oct 30 20:08:57.235769 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.235681 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" Oct 30 20:08:57.576369 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.576334 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j"] Oct 30 20:08:57.750746 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.750706 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz"] Oct 30 20:08:57.757022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.757001 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" Oct 30 20:08:57.775059 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.775028 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:57.782356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.782327 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spdjw\" (UniqueName: \"kubernetes.io/projected/acc72fb2-ae86-4dbd-989a-44f79e345dfb-kube-api-access-spdjw\") pod \"maas-api-revoke-sub-1015ba60-zdvbz\" (UID: \"acc72fb2-ae86-4dbd-989a-44f79e345dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" Oct 30 20:08:57.782506 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.782400 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc72fb2-ae86-4dbd-989a-44f79e345dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1015ba60-zdvbz\" (UID: \"acc72fb2-ae86-4dbd-989a-44f79e345dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" Oct 30 20:08:57.782615 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.782591 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc72fb2-ae86-4dbd-989a-44f79e345dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1015ba60-zdvbz\" (UID: \"acc72fb2-ae86-4dbd-989a-44f79e345dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" Oct 30 20:08:57.783028 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.783004 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz"] Oct 30 20:08:57.884643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.884604 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spdjw\" (UniqueName: \"kubernetes.io/projected/acc72fb2-ae86-4dbd-989a-44f79e345dfb-kube-api-access-spdjw\") pod \"maas-api-revoke-sub-1015ba60-zdvbz\" (UID: \"acc72fb2-ae86-4dbd-989a-44f79e345dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" Oct 30 20:08:57.884796 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.884668 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc72fb2-ae86-4dbd-989a-44f79e345dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1015ba60-zdvbz\" (UID: \"acc72fb2-ae86-4dbd-989a-44f79e345dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" Oct 30 20:08:57.884796 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.884774 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc72fb2-ae86-4dbd-989a-44f79e345dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1015ba60-zdvbz\" (UID: \"acc72fb2-ae86-4dbd-989a-44f79e345dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" Oct 30 20:08:57.885519 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.885497 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc72fb2-ae86-4dbd-989a-44f79e345dfb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1015ba60-zdvbz\" (UID: \"acc72fb2-ae86-4dbd-989a-44f79e345dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" Oct 30 20:08:57.887051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.887029 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc72fb2-ae86-4dbd-989a-44f79e345dfb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1015ba60-zdvbz\" (UID: \"acc72fb2-ae86-4dbd-989a-44f79e345dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" Oct 30 20:08:57.903848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.903811 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spdjw\" (UniqueName: \"kubernetes.io/projected/acc72fb2-ae86-4dbd-989a-44f79e345dfb-kube-api-access-spdjw\") pod \"maas-api-revoke-sub-1015ba60-zdvbz\" (UID: \"acc72fb2-ae86-4dbd-989a-44f79e345dfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" Oct 30 20:08:57.939940 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:57.939907 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" event={"ID":"6e168a95-a7a1-469f-8222-d8437ed038ed","Type":"ContainerStarted","Data":"a4e88db2cf493a2a04c827023bd171246f861fbd774307c0e9a5801498bb92b4"} Oct 30 20:08:58.071686 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:58.071658 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" Oct 30 20:08:58.370577 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:58.370544 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz"] Oct 30 20:08:58.372970 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:08:58.372937 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacc72fb2_ae86_4dbd_989a_44f79e345dfb.slice/crio-1674848c142686a9d79669ac0928dc0c3f40c3692c46cf2ac4572d0d12856e31 WatchSource:0}: Error finding container 1674848c142686a9d79669ac0928dc0c3f40c3692c46cf2ac4572d0d12856e31: Status 404 returned error can't find the container with id 1674848c142686a9d79669ac0928dc0c3f40c3692c46cf2ac4572d0d12856e31 Oct 30 20:08:58.759941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:58.759907 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:58.951848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:58.951816 5091 generic.go:356] "Generic (PLEG): container finished" podID="6e168a95-a7a1-469f-8222-d8437ed038ed" containerID="b0b24d1654175ca81f696fd899e53055d69a9620bea62f9f5b6e1c8ecfdb7ae9" exitCode=0 Oct 30 20:08:58.952047 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:58.951900 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" event={"ID":"6e168a95-a7a1-469f-8222-d8437ed038ed","Type":"ContainerDied","Data":"b0b24d1654175ca81f696fd899e53055d69a9620bea62f9f5b6e1c8ecfdb7ae9"} Oct 30 20:08:58.954007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:58.953980 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" event={"ID":"acc72fb2-ae86-4dbd-989a-44f79e345dfb","Type":"ContainerStarted","Data":"2dbd901459f8c7013342894d59e0ad5def8614b3194e3b75da2df62f80f2b574"} Oct 30 20:08:58.954125 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:58.954031 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" event={"ID":"acc72fb2-ae86-4dbd-989a-44f79e345dfb","Type":"ContainerStarted","Data":"1674848c142686a9d79669ac0928dc0c3f40c3692c46cf2ac4572d0d12856e31"} Oct 30 20:08:59.044731 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:59.044674 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" podStartSLOduration=2.044659668 podStartE2EDuration="2.044659668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:08:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:08:59.038647576 +0000 UTC m=+2258.733855533" watchObservedRunningTime="2026-10-30 20:08:59.044659668 +0000 UTC m=+2258.739867626" Oct 30 20:08:59.912656 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:59.912616 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:08:59.967709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:59.967673 5091 generic.go:356] "Generic (PLEG): container finished" podID="acc72fb2-ae86-4dbd-989a-44f79e345dfb" containerID="2dbd901459f8c7013342894d59e0ad5def8614b3194e3b75da2df62f80f2b574" exitCode=0 Oct 30 20:08:59.967910 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:08:59.967757 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" event={"ID":"acc72fb2-ae86-4dbd-989a-44f79e345dfb","Type":"ContainerDied","Data":"2dbd901459f8c7013342894d59e0ad5def8614b3194e3b75da2df62f80f2b574"} Oct 30 20:09:00.165403 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:00.165332 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" Oct 30 20:09:00.271611 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:00.271570 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e168a95-a7a1-469f-8222-d8437ed038ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e168a95-a7a1-469f-8222-d8437ed038ed-maas-api-openshift-ca\") pod \"6e168a95-a7a1-469f-8222-d8437ed038ed\" (UID: \"6e168a95-a7a1-469f-8222-d8437ed038ed\") " Oct 30 20:09:00.271801 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:00.271624 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e168a95-a7a1-469f-8222-d8437ed038ed-kube-api-access-6vx45\" (UniqueName: \"kubernetes.io/projected/6e168a95-a7a1-469f-8222-d8437ed038ed-kube-api-access-6vx45\") pod \"6e168a95-a7a1-469f-8222-d8437ed038ed\" (UID: \"6e168a95-a7a1-469f-8222-d8437ed038ed\") " Oct 30 20:09:00.271801 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:00.271703 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e168a95-a7a1-469f-8222-d8437ed038ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e168a95-a7a1-469f-8222-d8437ed038ed-maas-api-xks-ca\") pod \"6e168a95-a7a1-469f-8222-d8437ed038ed\" (UID: \"6e168a95-a7a1-469f-8222-d8437ed038ed\") " Oct 30 20:09:00.272037 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:00.272009 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e168a95-a7a1-469f-8222-d8437ed038ed-maas-api-openshift-ca" pod "6e168a95-a7a1-469f-8222-d8437ed038ed" (UID: "6e168a95-a7a1-469f-8222-d8437ed038ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:00.272465 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:00.272440 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e168a95-a7a1-469f-8222-d8437ed038ed-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:00.273988 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:00.273964 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e168a95-a7a1-469f-8222-d8437ed038ed-kube-api-access-6vx45" pod "6e168a95-a7a1-469f-8222-d8437ed038ed" (UID: "6e168a95-a7a1-469f-8222-d8437ed038ed"). InnerVolumeSpecName "kube-api-access-6vx45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:00.274242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:00.274210 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e168a95-a7a1-469f-8222-d8437ed038ed-maas-api-xks-ca" pod "6e168a95-a7a1-469f-8222-d8437ed038ed" (UID: "6e168a95-a7a1-469f-8222-d8437ed038ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:09:00.373842 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:00.373808 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vx45\" (UniqueName: \"kubernetes.io/projected/6e168a95-a7a1-469f-8222-d8437ed038ed-kube-api-access-6vx45\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:00.373842 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:00.373840 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e168a95-a7a1-469f-8222-d8437ed038ed-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:00.619078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:00.619027 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:00.981354 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:00.981322 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" Oct 30 20:09:00.981354 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:00.981343 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j" event={"ID":"6e168a95-a7a1-469f-8222-d8437ed038ed","Type":"ContainerDied","Data":"a4e88db2cf493a2a04c827023bd171246f861fbd774307c0e9a5801498bb92b4"} Oct 30 20:09:00.981867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:00.981390 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4e88db2cf493a2a04c827023bd171246f861fbd774307c0e9a5801498bb92b4" Oct 30 20:09:01.039284 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:01.039246 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:01.251128 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:01.251099 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" Oct 30 20:09:01.340091 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:01.340061 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acc72fb2-ae86-4dbd-989a-44f79e345dfb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc72fb2-ae86-4dbd-989a-44f79e345dfb-maas-api-openshift-ca\") pod \"acc72fb2-ae86-4dbd-989a-44f79e345dfb\" (UID: \"acc72fb2-ae86-4dbd-989a-44f79e345dfb\") " Oct 30 20:09:01.340309 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:01.340132 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acc72fb2-ae86-4dbd-989a-44f79e345dfb-kube-api-access-spdjw\" (UniqueName: \"kubernetes.io/projected/acc72fb2-ae86-4dbd-989a-44f79e345dfb-kube-api-access-spdjw\") pod \"acc72fb2-ae86-4dbd-989a-44f79e345dfb\" (UID: \"acc72fb2-ae86-4dbd-989a-44f79e345dfb\") " Oct 30 20:09:01.340402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:01.340388 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acc72fb2-ae86-4dbd-989a-44f79e345dfb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc72fb2-ae86-4dbd-989a-44f79e345dfb-maas-api-xks-ca\") pod \"acc72fb2-ae86-4dbd-989a-44f79e345dfb\" (UID: \"acc72fb2-ae86-4dbd-989a-44f79e345dfb\") " Oct 30 20:09:01.340575 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:01.340541 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acc72fb2-ae86-4dbd-989a-44f79e345dfb-maas-api-openshift-ca" pod "acc72fb2-ae86-4dbd-989a-44f79e345dfb" (UID: "acc72fb2-ae86-4dbd-989a-44f79e345dfb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:01.340994 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:01.340974 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acc72fb2-ae86-4dbd-989a-44f79e345dfb-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:01.342592 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:01.342564 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acc72fb2-ae86-4dbd-989a-44f79e345dfb-kube-api-access-spdjw" pod "acc72fb2-ae86-4dbd-989a-44f79e345dfb" (UID: "acc72fb2-ae86-4dbd-989a-44f79e345dfb"). InnerVolumeSpecName "kube-api-access-spdjw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:01.342847 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:01.342818 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acc72fb2-ae86-4dbd-989a-44f79e345dfb-maas-api-xks-ca" pod "acc72fb2-ae86-4dbd-989a-44f79e345dfb" (UID: "acc72fb2-ae86-4dbd-989a-44f79e345dfb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:09:01.442031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:01.441987 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acc72fb2-ae86-4dbd-989a-44f79e345dfb-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:01.442031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:01.442025 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spdjw\" (UniqueName: \"kubernetes.io/projected/acc72fb2-ae86-4dbd-989a-44f79e345dfb-kube-api-access-spdjw\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:01.539470 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:01.539377 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:01.960255 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:01.951485 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:02.066639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:02.066026 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" event={"ID":"acc72fb2-ae86-4dbd-989a-44f79e345dfb","Type":"ContainerDied","Data":"1674848c142686a9d79669ac0928dc0c3f40c3692c46cf2ac4572d0d12856e31"} Oct 30 20:09:02.066639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:02.066094 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1674848c142686a9d79669ac0928dc0c3f40c3692c46cf2ac4572d0d12856e31" Oct 30 20:09:02.066639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:02.066210 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz" Oct 30 20:09:02.575850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:02.575809 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:02.856257 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:02.855870 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:03.946947 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:03.946908 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:04.982256 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:04.982217 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:05.504496 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:05.504451 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:05.687794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:05.687751 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:06.262520 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:06.262484 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:06.553220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:06.538570 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:07.639740 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:07.639699 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:08.032493 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:08.032454 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c"] Oct 30 20:09:08.188711 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:08.188675 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:08.414228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:08.410399 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:08.624703 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:08.624663 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:09.430842 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:09.430807 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:09.854676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:09.854633 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:10.047165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.038798 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:10.431638 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.431588 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6976477d7d-88x4l"] Oct 30 20:09:10.432673 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.432646 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6e168a95-a7a1-469f-8222-d8437ed038ed" containerName="revoke-keys" Oct 30 20:09:10.432673 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.432672 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e168a95-a7a1-469f-8222-d8437ed038ed" containerName="revoke-keys" Oct 30 20:09:10.432856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.432687 5091 cpu_manager.go:414] "containerMap: removing container" podUID="acc72fb2-ae86-4dbd-989a-44f79e345dfb" containerName="revoke-keys" Oct 30 20:09:10.432856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.432695 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acc72fb2-ae86-4dbd-989a-44f79e345dfb" containerName="revoke-keys" Oct 30 20:09:10.432969 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.432900 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e168a95-a7a1-469f-8222-d8437ed038ed" containerName="revoke-keys" Oct 30 20:09:10.432969 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.432916 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="acc72fb2-ae86-4dbd-989a-44f79e345dfb" containerName="revoke-keys" Oct 30 20:09:10.439076 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.439042 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:10.455647 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.455619 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9d45aadaaa09\"" Oct 30 20:09:10.457835 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.457812 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-08c2badb5904\"" Oct 30 20:09:10.457999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.457842 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c06e1d87a941\"" Oct 30 20:09:10.457999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.457842 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-mqtgc\"" Oct 30 20:09:10.457999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.457921 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 20:09:10.458235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.457942 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 20:09:10.458235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.457954 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 20:09:10.458235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.457967 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 20:09:10.459140 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.459117 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6976477d7d-88x4l"] Oct 30 20:09:10.494833 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.494792 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efa6cde-c023-40c7-a04b-a73b1605e9c9-plugins-config-volume\") pod \"payload-processing-external-model-6976477d7d-88x4l\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:10.495167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.495121 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-999vt\" (UniqueName: \"kubernetes.io/projected/4efa6cde-c023-40c7-a04b-a73b1605e9c9-kube-api-access-999vt\") pod \"payload-processing-external-model-6976477d7d-88x4l\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:10.497448 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.497419 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4efa6cde-c023-40c7-a04b-a73b1605e9c9-routing-overlay-volume\") pod \"payload-processing-external-model-6976477d7d-88x4l\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:10.497724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.497700 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4efa6cde-c023-40c7-a04b-a73b1605e9c9-provider-credentials\") pod \"payload-processing-external-model-6976477d7d-88x4l\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:10.599263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.599232 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4efa6cde-c023-40c7-a04b-a73b1605e9c9-routing-overlay-volume\") pod \"payload-processing-external-model-6976477d7d-88x4l\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:10.599427 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.599305 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4efa6cde-c023-40c7-a04b-a73b1605e9c9-provider-credentials\") pod \"payload-processing-external-model-6976477d7d-88x4l\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:10.599496 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.599465 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efa6cde-c023-40c7-a04b-a73b1605e9c9-plugins-config-volume\") pod \"payload-processing-external-model-6976477d7d-88x4l\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:10.599538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.599500 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-999vt\" (UniqueName: \"kubernetes.io/projected/4efa6cde-c023-40c7-a04b-a73b1605e9c9-kube-api-access-999vt\") pod \"payload-processing-external-model-6976477d7d-88x4l\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:10.600049 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.600017 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4efa6cde-c023-40c7-a04b-a73b1605e9c9-routing-overlay-volume\") pod \"payload-processing-external-model-6976477d7d-88x4l\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:10.600218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.600188 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efa6cde-c023-40c7-a04b-a73b1605e9c9-plugins-config-volume\") pod \"payload-processing-external-model-6976477d7d-88x4l\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:10.602018 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.601996 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4efa6cde-c023-40c7-a04b-a73b1605e9c9-provider-credentials\") pod \"payload-processing-external-model-6976477d7d-88x4l\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:10.617603 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.617583 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-999vt\" (UniqueName: \"kubernetes.io/projected/4efa6cde-c023-40c7-a04b-a73b1605e9c9-kube-api-access-999vt\") pod \"payload-processing-external-model-6976477d7d-88x4l\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:10.760951 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.760895 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:10.867720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:10.865430 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:11.107243 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:11.101100 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:11.255289 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:11.254954 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk"] Oct 30 20:09:11.309799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:11.307812 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6976477d7d-88x4l"] Oct 30 20:09:11.328082 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:11.325759 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:11.600230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:11.600196 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:12.247074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:12.247035 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:12.262336 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:12.262306 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" event={"ID":"4efa6cde-c023-40c7-a04b-a73b1605e9c9","Type":"ContainerStarted","Data":"dbd3fa913e10304756a70c3ef7fc22861b674090864e442ff410993f69e8d049"} Oct 30 20:09:12.262490 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:12.262399 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:12.262490 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:12.262426 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" event={"ID":"4efa6cde-c023-40c7-a04b-a73b1605e9c9","Type":"ContainerStarted","Data":"ca5c5fb1fddac6c47787207b375c408bb145c3dccf97a5fc5977aafb5b7400c0"} Oct 30 20:09:12.319821 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:12.319738 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" podStartSLOduration=2.3197200159999998 podStartE2EDuration="2.319720016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:12.316317301 +0000 UTC m=+2272.011525259" watchObservedRunningTime="2026-10-30 20:09:12.319720016 +0000 UTC m=+2272.014927975" Oct 30 20:09:12.921063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:12.921019 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:13.032773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.032733 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" podUID="81323fbd-decc-4170-bcfc-87351d7e8cca" containerName="payload-processing" containerID="cri-o://fce07879cec2c6c434be0b62c74b6b82364d2e35c9111899e8b9bbf7637b875d" gracePeriod=25 Oct 30 20:09:13.160282 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:09:13.160255 5091 logging.go:55] [core] [Channel #6152 SubChannel #6153]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.51:50052", ServerName: "10.132.1.51:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.51:50052: connect: connection refused" Oct 30 20:09:13.276835 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.276802 5091 generic.go:356] "Generic (PLEG): container finished" podID="81323fbd-decc-4170-bcfc-87351d7e8cca" containerID="fce07879cec2c6c434be0b62c74b6b82364d2e35c9111899e8b9bbf7637b875d" exitCode=0 Oct 30 20:09:13.277025 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.276867 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" event={"ID":"81323fbd-decc-4170-bcfc-87351d7e8cca","Type":"ContainerDied","Data":"fce07879cec2c6c434be0b62c74b6b82364d2e35c9111899e8b9bbf7637b875d"} Oct 30 20:09:13.337429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.337404 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:09:13.434500 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.434464 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81323fbd-decc-4170-bcfc-87351d7e8cca-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81323fbd-decc-4170-bcfc-87351d7e8cca-routing-overlay-volume\") pod \"81323fbd-decc-4170-bcfc-87351d7e8cca\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " Oct 30 20:09:13.434678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.434518 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81323fbd-decc-4170-bcfc-87351d7e8cca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81323fbd-decc-4170-bcfc-87351d7e8cca-plugins-config-volume\") pod \"81323fbd-decc-4170-bcfc-87351d7e8cca\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " Oct 30 20:09:13.434678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.434588 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81323fbd-decc-4170-bcfc-87351d7e8cca-kube-api-access-cb2rx\" (UniqueName: \"kubernetes.io/projected/81323fbd-decc-4170-bcfc-87351d7e8cca-kube-api-access-cb2rx\") pod \"81323fbd-decc-4170-bcfc-87351d7e8cca\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " Oct 30 20:09:13.434799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.434712 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81323fbd-decc-4170-bcfc-87351d7e8cca-provider-credentials\" (UniqueName: \"kubernetes.io/projected/81323fbd-decc-4170-bcfc-87351d7e8cca-provider-credentials\") pod \"81323fbd-decc-4170-bcfc-87351d7e8cca\" (UID: \"81323fbd-decc-4170-bcfc-87351d7e8cca\") " Oct 30 20:09:13.434891 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.434862 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81323fbd-decc-4170-bcfc-87351d7e8cca-routing-overlay-volume" pod "81323fbd-decc-4170-bcfc-87351d7e8cca" (UID: "81323fbd-decc-4170-bcfc-87351d7e8cca"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:13.435044 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.435018 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81323fbd-decc-4170-bcfc-87351d7e8cca-plugins-config-volume" pod "81323fbd-decc-4170-bcfc-87351d7e8cca" (UID: "81323fbd-decc-4170-bcfc-87351d7e8cca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:13.435664 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.435634 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/81323fbd-decc-4170-bcfc-87351d7e8cca-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:13.435791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.435666 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81323fbd-decc-4170-bcfc-87351d7e8cca-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:13.437139 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.437110 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81323fbd-decc-4170-bcfc-87351d7e8cca-kube-api-access-cb2rx" pod "81323fbd-decc-4170-bcfc-87351d7e8cca" (UID: "81323fbd-decc-4170-bcfc-87351d7e8cca"). InnerVolumeSpecName "kube-api-access-cb2rx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:13.437980 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.437960 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81323fbd-decc-4170-bcfc-87351d7e8cca-provider-credentials" pod "81323fbd-decc-4170-bcfc-87351d7e8cca" (UID: "81323fbd-decc-4170-bcfc-87351d7e8cca"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:13.474780 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.474708 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:13.537022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.536992 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb2rx\" (UniqueName: \"kubernetes.io/projected/81323fbd-decc-4170-bcfc-87351d7e8cca-kube-api-access-cb2rx\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:13.537206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:13.537027 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/81323fbd-decc-4170-bcfc-87351d7e8cca-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:14.159985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:14.159941 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" podUID="81323fbd-decc-4170-bcfc-87351d7e8cca" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.51:50052\" within 1s: context deadline exceeded" Oct 30 20:09:14.287010 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:14.286978 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" Oct 30 20:09:14.287245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:14.287012 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c" event={"ID":"81323fbd-decc-4170-bcfc-87351d7e8cca","Type":"ContainerDied","Data":"4a0ff4c33d1feec3799c9b415b12bb26d13624701224dcd82128c7e04021f986"} Oct 30 20:09:14.287245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:14.287087 5091 scope.go:122] "RemoveContainer" containerID="fce07879cec2c6c434be0b62c74b6b82364d2e35c9111899e8b9bbf7637b875d" Oct 30 20:09:14.338850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:14.338817 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c"] Oct 30 20:09:14.344296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:14.344270 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-8c8241/payload-processing-external-model-e2e-extmod-a-8c8241-68586jf5c"] Oct 30 20:09:14.492539 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:14.487443 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:14.510296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:14.506108 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81323fbd-decc-4170-bcfc-87351d7e8cca" path="/var/lib/kubelet/pods/81323fbd-decc-4170-bcfc-87351d7e8cca/volumes" Oct 30 20:09:14.851727 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:14.851688 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:15.540382 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:15.540336 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:15.979031 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:09:15.978930 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:09:16.269012 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:16.268974 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" podUID="4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4" containerName="payload-processing" containerID="cri-o://d17a0d3c6e7399b78f4f001c3465334b2c8cd6b5535385ae6c0edd6c8c882f75" gracePeriod=25 Oct 30 20:09:16.282728 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:09:16.282707 5091 logging.go:55] [core] [Channel #6169 SubChannel #6170]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.52:50052", ServerName: "10.132.1.52:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.52:50052: connect: connection refused" Oct 30 20:09:16.576460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:16.576427 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:09:16.633920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:16.633471 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-kube-api-access-rcntm\" (UniqueName: \"kubernetes.io/projected/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-kube-api-access-rcntm\") pod \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " Oct 30 20:09:16.633920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:16.633540 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-routing-overlay-volume\") pod \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " Oct 30 20:09:16.633920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:16.633594 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-provider-credentials\") pod \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " Oct 30 20:09:16.633920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:16.633679 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-plugins-config-volume\") pod \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\" (UID: \"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4\") " Oct 30 20:09:16.635852 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:16.635801 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-plugins-config-volume" pod "4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4" (UID: "4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:16.636169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:16.636120 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-routing-overlay-volume" pod "4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4" (UID: "4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:16.668087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:16.646054 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:16.668087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:16.646088 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:16.668087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:16.667949 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-provider-credentials" pod "4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4" (UID: "4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:16.670236 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:16.669703 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-kube-api-access-rcntm" pod "4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4" (UID: "4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4"). InnerVolumeSpecName "kube-api-access-rcntm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:16.694377 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:16.694345 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:16.747316 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:16.747282 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcntm\" (UniqueName: \"kubernetes.io/projected/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-kube-api-access-rcntm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:16.747316 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:16.747312 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:17.282084 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:17.282040 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" podUID="4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.52:50052\" within 1s: context deadline exceeded" Oct 30 20:09:17.321353 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:17.321314 5091 generic.go:356] "Generic (PLEG): container finished" podID="4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4" containerID="d17a0d3c6e7399b78f4f001c3465334b2c8cd6b5535385ae6c0edd6c8c882f75" exitCode=0 Oct 30 20:09:17.321541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:17.321359 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" Oct 30 20:09:17.321541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:17.321393 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" event={"ID":"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4","Type":"ContainerDied","Data":"d17a0d3c6e7399b78f4f001c3465334b2c8cd6b5535385ae6c0edd6c8c882f75"} Oct 30 20:09:17.321541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:17.321449 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk" event={"ID":"4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4","Type":"ContainerDied","Data":"a2f287c9f7950ab4756e3872868b469b75f7baef8072ec4853a0f40bc12147a9"} Oct 30 20:09:17.321541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:17.321474 5091 scope.go:122] "RemoveContainer" containerID="d17a0d3c6e7399b78f4f001c3465334b2c8cd6b5535385ae6c0edd6c8c882f75" Oct 30 20:09:17.333255 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:17.333236 5091 scope.go:122] "RemoveContainer" containerID="d17a0d3c6e7399b78f4f001c3465334b2c8cd6b5535385ae6c0edd6c8c882f75" Oct 30 20:09:17.333518 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:09:17.333495 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d17a0d3c6e7399b78f4f001c3465334b2c8cd6b5535385ae6c0edd6c8c882f75\": container with ID starting with d17a0d3c6e7399b78f4f001c3465334b2c8cd6b5535385ae6c0edd6c8c882f75 not found: ID does not exist" containerID="d17a0d3c6e7399b78f4f001c3465334b2c8cd6b5535385ae6c0edd6c8c882f75" Oct 30 20:09:17.333578 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:17.333530 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d17a0d3c6e7399b78f4f001c3465334b2c8cd6b5535385ae6c0edd6c8c882f75"} err="failed to get container status \"d17a0d3c6e7399b78f4f001c3465334b2c8cd6b5535385ae6c0edd6c8c882f75\": rpc error: code = NotFound desc = could not find container \"d17a0d3c6e7399b78f4f001c3465334b2c8cd6b5535385ae6c0edd6c8c882f75\": container with ID starting with d17a0d3c6e7399b78f4f001c3465334b2c8cd6b5535385ae6c0edd6c8c882f75 not found: ID does not exist" Oct 30 20:09:17.369218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:17.369192 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk"] Oct 30 20:09:17.375563 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:17.375539 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-67559ppbk"] Oct 30 20:09:17.896379 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:17.896336 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:18.444060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:18.444026 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4" path="/var/lib/kubelet/pods/4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4/volumes" Oct 30 20:09:19.114297 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:19.114265 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:20.440336 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:20.440275 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:21.378464 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:21.378325 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:21.681736 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:21.681632 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:21.995930 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:21.995838 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:22.894498 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:22.894460 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:23.280953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:23.280921 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:09:23.955305 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:23.955264 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:24.926175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:24.925724 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:25.848125 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:25.848087 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:27.100766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:27.100727 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:28.648224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:28.648134 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:29.949178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:29.949130 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:31.251651 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:31.251409 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:32.521172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:32.511864 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:32.790363 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:32.790273 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:33.644654 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:33.644614 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:33.880134 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:33.880097 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:34.110873 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.110832 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:34.233129 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.233092 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5"] Oct 30 20:09:34.233883 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.233867 5091 cpu_manager.go:414] "containerMap: removing container" podUID="81323fbd-decc-4170-bcfc-87351d7e8cca" containerName="payload-processing" Oct 30 20:09:34.233945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.233884 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81323fbd-decc-4170-bcfc-87351d7e8cca" containerName="payload-processing" Oct 30 20:09:34.233945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.233926 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4" containerName="payload-processing" Oct 30 20:09:34.233945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.233932 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4" containerName="payload-processing" Oct 30 20:09:34.234060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.234034 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe22a50-9a48-41bb-b837-7b3eb3b2b6a4" containerName="payload-processing" Oct 30 20:09:34.234060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.234047 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="81323fbd-decc-4170-bcfc-87351d7e8cca" containerName="payload-processing" Oct 30 20:09:34.239901 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.239879 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:34.267455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.267430 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5"] Oct 30 20:09:34.305974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.305946 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b1d704898dc4\"" Oct 30 20:09:34.367961 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.367872 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f637512-1495-4d6a-a7b5-c6d2228fc55b-plugins-config-volume\") pod \"payload-processing-external-model-7b74489dbd-k59r5\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:34.368129 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.368022 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-provider-credentials\") pod \"payload-processing-external-model-7b74489dbd-k59r5\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:34.368244 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.368176 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn5r4\" (UniqueName: \"kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-kube-api-access-wn5r4\") pod \"payload-processing-external-model-7b74489dbd-k59r5\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:34.368325 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.368307 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4f637512-1495-4d6a-a7b5-c6d2228fc55b-routing-overlay-volume\") pod \"payload-processing-external-model-7b74489dbd-k59r5\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:34.470223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.470184 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn5r4\" (UniqueName: \"kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-kube-api-access-wn5r4\") pod \"payload-processing-external-model-7b74489dbd-k59r5\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:34.470401 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.470312 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4f637512-1495-4d6a-a7b5-c6d2228fc55b-routing-overlay-volume\") pod \"payload-processing-external-model-7b74489dbd-k59r5\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:34.470466 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.470429 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f637512-1495-4d6a-a7b5-c6d2228fc55b-plugins-config-volume\") pod \"payload-processing-external-model-7b74489dbd-k59r5\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:34.470530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.470514 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-provider-credentials\") pod \"payload-processing-external-model-7b74489dbd-k59r5\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:34.471051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.471001 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4f637512-1495-4d6a-a7b5-c6d2228fc55b-routing-overlay-volume\") pod \"payload-processing-external-model-7b74489dbd-k59r5\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:34.471222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.471176 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f637512-1495-4d6a-a7b5-c6d2228fc55b-plugins-config-volume\") pod \"payload-processing-external-model-7b74489dbd-k59r5\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:34.473585 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.473555 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-provider-credentials\") pod \"payload-processing-external-model-7b74489dbd-k59r5\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:34.486743 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.486716 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn5r4\" (UniqueName: \"kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-kube-api-access-wn5r4\") pod \"payload-processing-external-model-7b74489dbd-k59r5\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:34.555113 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.555079 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:34.852726 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:09:34.852684 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f637512_1495_4d6a_a7b5_c6d2228fc55b.slice/crio-731a25d760e7a11780de09c7fd9db277ac84ecf3670d8575358683f74baa1f67 WatchSource:0}: Error finding container 731a25d760e7a11780de09c7fd9db277ac84ecf3670d8575358683f74baa1f67: Status 404 returned error can't find the container with id 731a25d760e7a11780de09c7fd9db277ac84ecf3670d8575358683f74baa1f67 Oct 30 20:09:34.855988 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:34.855959 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5"] Oct 30 20:09:35.006276 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:35.006224 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:35.166230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:35.158198 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5"] Oct 30 20:09:35.234170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:35.231332 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:35.679243 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:35.679208 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" event={"ID":"4f637512-1495-4d6a-a7b5-c6d2228fc55b","Type":"ContainerStarted","Data":"17419d50ba493e2bec799cdd769a1f361cd4c15560ba68719b9a182f31f9a78b"} Oct 30 20:09:35.679454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:35.679265 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" event={"ID":"4f637512-1495-4d6a-a7b5-c6d2228fc55b","Type":"ContainerStarted","Data":"731a25d760e7a11780de09c7fd9db277ac84ecf3670d8575358683f74baa1f67"} Oct 30 20:09:35.679454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:35.679311 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:35.733358 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:35.733307 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" podStartSLOduration=1.733293711 podStartE2EDuration="1.733293711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:35.728871538 +0000 UTC m=+2295.424079494" watchObservedRunningTime="2026-10-30 20:09:35.733293711 +0000 UTC m=+2295.428501668" Oct 30 20:09:35.817293 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:09:35.817240 5091 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b1d704898dc4: secret "agc-cred-b1d704898dc4" not found Oct 30 20:09:35.817293 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:09:35.817279 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5: secret "agc-cred-b1d704898dc4" not found Oct 30 20:09:35.817572 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:09:35.817349 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-provider-credentials podName:4f637512-1495-4d6a-a7b5-c6d2228fc55b nodeName:}" failed. No retries permitted until 2026-10-30 20:09:36.317332003 +0000 UTC m=+2296.012539942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-provider-credentials") pod "payload-processing-external-model-7b74489dbd-k59r5" (UID: "4f637512-1495-4d6a-a7b5-c6d2228fc55b") : secret "agc-cred-b1d704898dc4" not found Oct 30 20:09:36.081367 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.081329 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c"] Oct 30 20:09:36.086714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.086691 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.105438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.105409 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c"] Oct 30 20:09:36.227684 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.227648 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.227870 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.227709 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.227870 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.227728 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.227968 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.227907 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.228071 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.228053 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7458q\" (UniqueName: \"kubernetes.io/projected/92b34b40-9e41-4d49-b6b3-d8fb78227451-kube-api-access-7458q\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.228121 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.228108 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.228240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.228223 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.228352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.228339 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.228391 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.228378 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92b34b40-9e41-4d49-b6b3-d8fb78227451-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.330251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.330210 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.330440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.330297 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.330440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.330323 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.330440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.330374 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.330624 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.330503 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7458q\" (UniqueName: \"kubernetes.io/projected/92b34b40-9e41-4d49-b6b3-d8fb78227451-kube-api-access-7458q\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.330624 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.330571 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.330624 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.330585 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.330811 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.330663 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.330811 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.330663 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.330811 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.330779 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.330966 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.330824 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92b34b40-9e41-4d49-b6b3-d8fb78227451-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.330966 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:09:36.330857 5091 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b1d704898dc4: secret "agc-cred-b1d704898dc4" not found Oct 30 20:09:36.330966 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:09:36.330875 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5: secret "agc-cred-b1d704898dc4" not found Oct 30 20:09:36.330966 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.330899 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.330966 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:09:36.330940 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-provider-credentials podName:4f637512-1495-4d6a-a7b5-c6d2228fc55b nodeName:}" failed. No retries permitted until 2026-10-30 20:09:37.330922652 +0000 UTC m=+2297.026130591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-provider-credentials") pod "payload-processing-external-model-7b74489dbd-k59r5" (UID: "4f637512-1495-4d6a-a7b5-c6d2228fc55b") : secret "agc-cred-b1d704898dc4" not found Oct 30 20:09:36.331274 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.331031 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.331633 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.331566 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92b34b40-9e41-4d49-b6b3-d8fb78227451-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.333021 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.333001 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.333615 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.333591 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.349622 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.349579 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.377034 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.376822 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:36.381402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.381368 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7458q\" (UniqueName: \"kubernetes.io/projected/92b34b40-9e41-4d49-b6b3-d8fb78227451-kube-api-access-7458q\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-rgs5c\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.415521 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.415469 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:36.756635 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:09:36.756588 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b34b40_9e41_4d49_b6b3_d8fb78227451.slice/crio-6d5e56a2b2315574ab44a520cce7839a22c8a823647f33af513ae265f76efa57 WatchSource:0}: Error finding container 6d5e56a2b2315574ab44a520cce7839a22c8a823647f33af513ae265f76efa57: Status 404 returned error can't find the container with id 6d5e56a2b2315574ab44a520cce7839a22c8a823647f33af513ae265f76efa57 Oct 30 20:09:36.757617 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.757532 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c"] Oct 30 20:09:36.759922 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.759890 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:09:36.760019 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.759954 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:09:36.760019 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:36.759997 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:09:37.350584 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:09:37.350519 5091 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b1d704898dc4: secret "agc-cred-b1d704898dc4" not found Oct 30 20:09:37.350584 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:09:37.350560 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5: secret "agc-cred-b1d704898dc4" not found Oct 30 20:09:37.351365 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:09:37.350637 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-provider-credentials podName:4f637512-1495-4d6a-a7b5-c6d2228fc55b nodeName:}" failed. No retries permitted until 2026-10-30 20:09:39.350616932 +0000 UTC m=+2299.045824916 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-provider-credentials") pod "payload-processing-external-model-7b74489dbd-k59r5" (UID: "4f637512-1495-4d6a-a7b5-c6d2228fc55b") : secret "agc-cred-b1d704898dc4" not found Oct 30 20:09:37.570220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:37.569848 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:37.697730 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:37.697625 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" event={"ID":"92b34b40-9e41-4d49-b6b3-d8fb78227451","Type":"ContainerStarted","Data":"7139afc699f0abf99339334ee3f417fb15fbd38a1ec7240cfc082fad9e54d6c8"} Oct 30 20:09:37.697730 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:37.697690 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" event={"ID":"92b34b40-9e41-4d49-b6b3-d8fb78227451","Type":"ContainerStarted","Data":"6d5e56a2b2315574ab44a520cce7839a22c8a823647f33af513ae265f76efa57"} Oct 30 20:09:37.732404 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:37.732335 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" podStartSLOduration=1.7323143669999999 podStartE2EDuration="1.732314367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:37.727738672 +0000 UTC m=+2297.422946631" watchObservedRunningTime="2026-10-30 20:09:37.732314367 +0000 UTC m=+2297.427522327" Oct 30 20:09:38.040296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:38.040260 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:38.355268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:38.354674 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:38.416766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:38.416730 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:38.418515 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:38.418486 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body= Oct 30 20:09:38.418642 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:38.418538 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" podUID="92b34b40-9e41-4d49-b6b3-d8fb78227451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" Oct 30 20:09:38.516626 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:38.516568 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:38.744888 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:38.744798 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:39.094171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:39.093600 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:39.314533 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:39.314480 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:39.408989 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:09:39.408903 5091 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-b1d704898dc4: secret "agc-cred-b1d704898dc4" not found Oct 30 20:09:39.408989 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:09:39.408933 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5: secret "agc-cred-b1d704898dc4" not found Oct 30 20:09:39.409550 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:09:39.409001 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-provider-credentials podName:4f637512-1495-4d6a-a7b5-c6d2228fc55b nodeName:}" failed. No retries permitted until 2026-10-30 20:09:43.408980877 +0000 UTC m=+2303.104188830 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-provider-credentials") pod "payload-processing-external-model-7b74489dbd-k59r5" (UID: "4f637512-1495-4d6a-a7b5-c6d2228fc55b") : secret "agc-cred-b1d704898dc4" not found Oct 30 20:09:39.416419 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:39.416392 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body= Oct 30 20:09:39.416551 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:39.416445 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" podUID="92b34b40-9e41-4d49-b6b3-d8fb78227451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" Oct 30 20:09:39.541168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:39.541007 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:40.139170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:40.130868 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:40.518172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:40.517987 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body= Oct 30 20:09:40.518172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:40.518056 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" podUID="92b34b40-9e41-4d49-b6b3-d8fb78227451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" Oct 30 20:09:40.538172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:40.519388 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:41.402519 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:41.398310 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:41.433596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:41.432095 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body= Oct 30 20:09:41.433596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:41.432174 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" podUID="92b34b40-9e41-4d49-b6b3-d8fb78227451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" Oct 30 20:09:41.687927 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:41.687818 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" podUID="4f637512-1495-4d6a-a7b5-c6d2228fc55b" containerName="payload-processing" containerID="cri-o://17419d50ba493e2bec799cdd769a1f361cd4c15560ba68719b9a182f31f9a78b" gracePeriod=25 Oct 30 20:09:41.840038 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:41.832012 5091 generic.go:356] "Generic (PLEG): container finished" podID="4f637512-1495-4d6a-a7b5-c6d2228fc55b" containerID="17419d50ba493e2bec799cdd769a1f361cd4c15560ba68719b9a182f31f9a78b" exitCode=0 Oct 30 20:09:41.840038 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:41.832101 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" event={"ID":"4f637512-1495-4d6a-a7b5-c6d2228fc55b","Type":"ContainerDied","Data":"17419d50ba493e2bec799cdd769a1f361cd4c15560ba68719b9a182f31f9a78b"} Oct 30 20:09:42.138675 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.138645 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:42.284288 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.284232 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f637512-1495-4d6a-a7b5-c6d2228fc55b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f637512-1495-4d6a-a7b5-c6d2228fc55b-plugins-config-volume\") pod \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " Oct 30 20:09:42.284288 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.284283 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f637512-1495-4d6a-a7b5-c6d2228fc55b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4f637512-1495-4d6a-a7b5-c6d2228fc55b-routing-overlay-volume\") pod \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " Oct 30 20:09:42.284567 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.284556 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-provider-credentials\") pod \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " Oct 30 20:09:42.284677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.284655 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-kube-api-access-wn5r4\" (UniqueName: \"kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-kube-api-access-wn5r4\") pod \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\" (UID: \"4f637512-1495-4d6a-a7b5-c6d2228fc55b\") " Oct 30 20:09:42.286791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.285681 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f637512-1495-4d6a-a7b5-c6d2228fc55b-routing-overlay-volume" pod "4f637512-1495-4d6a-a7b5-c6d2228fc55b" (UID: "4f637512-1495-4d6a-a7b5-c6d2228fc55b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:42.286791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.286079 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f637512-1495-4d6a-a7b5-c6d2228fc55b-plugins-config-volume" pod "4f637512-1495-4d6a-a7b5-c6d2228fc55b" (UID: "4f637512-1495-4d6a-a7b5-c6d2228fc55b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:09:42.294434 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.294404 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-provider-credentials" pod "4f637512-1495-4d6a-a7b5-c6d2228fc55b" (UID: "4f637512-1495-4d6a-a7b5-c6d2228fc55b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:42.302331 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.302287 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-kube-api-access-wn5r4" pod "4f637512-1495-4d6a-a7b5-c6d2228fc55b" (UID: "4f637512-1495-4d6a-a7b5-c6d2228fc55b"). InnerVolumeSpecName "kube-api-access-wn5r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:09:42.414016 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.413922 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn5r4\" (UniqueName: \"kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-kube-api-access-wn5r4\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:42.415890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.415859 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f637512-1495-4d6a-a7b5-c6d2228fc55b-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:42.417071 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.416411 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body= Oct 30 20:09:42.417071 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.416466 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" podUID="92b34b40-9e41-4d49-b6b3-d8fb78227451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" Oct 30 20:09:42.417513 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.416121 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4f637512-1495-4d6a-a7b5-c6d2228fc55b-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:42.417513 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.417363 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4f637512-1495-4d6a-a7b5-c6d2228fc55b-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:09:42.434271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.434247 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:42.867170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.862942 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" event={"ID":"4f637512-1495-4d6a-a7b5-c6d2228fc55b","Type":"ContainerDied","Data":"731a25d760e7a11780de09c7fd9db277ac84ecf3670d8575358683f74baa1f67"} Oct 30 20:09:42.867170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.863020 5091 scope.go:122] "RemoveContainer" containerID="17419d50ba493e2bec799cdd769a1f361cd4c15560ba68719b9a182f31f9a78b" Oct 30 20:09:42.867170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.863231 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5" Oct 30 20:09:42.895439 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.895412 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5"] Oct 30 20:09:42.901889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:42.901863 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7b74489dbd-k59r5"] Oct 30 20:09:43.396171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:43.391177 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:43.421171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:43.417575 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body= Oct 30 20:09:43.421171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:43.417654 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" podUID="92b34b40-9e41-4d49-b6b3-d8fb78227451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" Oct 30 20:09:44.234210 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:44.234114 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:44.416917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:44.416884 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" start-of-body= Oct 30 20:09:44.417106 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:44.416938 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" podUID="92b34b40-9e41-4d49-b6b3-d8fb78227451" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.64:15021/healthz/ready\": dial tcp 10.132.1.64:15021: connect: connection refused" Oct 30 20:09:44.448739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:44.448703 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f637512-1495-4d6a-a7b5-c6d2228fc55b" path="/var/lib/kubelet/pods/4f637512-1495-4d6a-a7b5-c6d2228fc55b/volumes" Oct 30 20:09:45.028795 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:45.028695 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:45.985952 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:09:45.985908 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:09:46.011500 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:46.011459 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:46.295359 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:46.295327 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:46.296022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:46.295993 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:46.297050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:46.297014 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:09:46.997335 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:46.996897 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:47.916726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:47.916669 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:48.806278 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:48.806239 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:49.709097 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:49.709040 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:50.678954 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:50.677520 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:51.132189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.132135 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck"] Oct 30 20:09:51.133282 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.133251 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4f637512-1495-4d6a-a7b5-c6d2228fc55b" containerName="payload-processing" Oct 30 20:09:51.133493 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.133467 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f637512-1495-4d6a-a7b5-c6d2228fc55b" containerName="payload-processing" Oct 30 20:09:51.133804 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.133786 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f637512-1495-4d6a-a7b5-c6d2228fc55b" containerName="payload-processing" Oct 30 20:09:51.138784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.138762 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.163545 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.163509 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck"] Oct 30 20:09:51.305990 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.305948 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.305990 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.305994 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.306316 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.306086 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvfhn\" (UniqueName: \"kubernetes.io/projected/961fed7b-2029-4b0e-886f-f4d6cda91f81-kube-api-access-fvfhn\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.306316 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.306240 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.306440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.306381 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.306440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.306415 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961fed7b-2029-4b0e-886f-f4d6cda91f81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.306584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.306560 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.306707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.306668 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.306901 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.306873 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.408207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.408120 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.408207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.408203 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.408455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.408230 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.408455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.408258 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvfhn\" (UniqueName: \"kubernetes.io/projected/961fed7b-2029-4b0e-886f-f4d6cda91f81-kube-api-access-fvfhn\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.408455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.408320 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.408455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.408377 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.408455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.408394 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961fed7b-2029-4b0e-886f-f4d6cda91f81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.408753 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.408476 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.408753 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.408678 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.408753 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.408696 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.408928 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.408793 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.408928 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.408857 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.409040 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.409018 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.409220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.409187 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961fed7b-2029-4b0e-886f-f4d6cda91f81-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.410526 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.410504 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.410850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.410829 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.423006 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.422974 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.423884 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.423861 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvfhn\" (UniqueName: \"kubernetes.io/projected/961fed7b-2029-4b0e-886f-f4d6cda91f81-kube-api-access-fvfhn\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-d5bck\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.468833 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.468804 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:51.676250 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.670283 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:51.768837 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:09:51.768805 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961fed7b_2029_4b0e_886f_f4d6cda91f81.slice/crio-a5ea39cf719d694b3293bef9fe5134a886b589ef9bf80044de969600e87a3bb9 WatchSource:0}: Error finding container a5ea39cf719d694b3293bef9fe5134a886b589ef9bf80044de969600e87a3bb9: Status 404 returned error can't find the container with id a5ea39cf719d694b3293bef9fe5134a886b589ef9bf80044de969600e87a3bb9 Oct 30 20:09:51.772565 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.772530 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:09:51.772703 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.772598 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:09:51.772703 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.772636 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:09:51.772960 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:51.772932 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck"] Oct 30 20:09:52.012888 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:52.012840 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" event={"ID":"961fed7b-2029-4b0e-886f-f4d6cda91f81","Type":"ContainerStarted","Data":"ba8a5e01a21350edbb0dbc00c606c6bd3bc8a8e5f343d9c302902bf75f45ba95"} Oct 30 20:09:52.013088 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:52.012948 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" event={"ID":"961fed7b-2029-4b0e-886f-f4d6cda91f81","Type":"ContainerStarted","Data":"a5ea39cf719d694b3293bef9fe5134a886b589ef9bf80044de969600e87a3bb9"} Oct 30 20:09:52.469639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:52.469528 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:09:52.471125 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:52.471094 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d5bck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Oct 30 20:09:52.471291 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:52.471192 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" podUID="961fed7b-2029-4b0e-886f-f4d6cda91f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Oct 30 20:09:52.689424 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:52.689384 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:53.469609 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:53.469567 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d5bck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Oct 30 20:09:53.470291 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:53.470259 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" podUID="961fed7b-2029-4b0e-886f-f4d6cda91f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Oct 30 20:09:53.793869 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:53.793828 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:54.472242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:54.470588 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d5bck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Oct 30 20:09:54.472242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:54.470653 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" podUID="961fed7b-2029-4b0e-886f-f4d6cda91f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Oct 30 20:09:55.042289 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:55.042126 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:55.471841 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:55.471742 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d5bck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Oct 30 20:09:55.472226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:55.472123 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" podUID="961fed7b-2029-4b0e-886f-f4d6cda91f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Oct 30 20:09:56.217561 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:56.212029 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:56.470017 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:56.469489 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d5bck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Oct 30 20:09:56.470017 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:56.469564 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" podUID="961fed7b-2029-4b0e-886f-f4d6cda91f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Oct 30 20:09:57.472169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:57.471522 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d5bck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Oct 30 20:09:57.472169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:57.471606 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" podUID="961fed7b-2029-4b0e-886f-f4d6cda91f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Oct 30 20:09:57.494170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:57.493990 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:58.470201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:58.470160 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d5bck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Oct 30 20:09:58.470391 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:58.470238 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" podUID="961fed7b-2029-4b0e-886f-f4d6cda91f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Oct 30 20:09:58.956554 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:58.956516 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:09:59.365824 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.365765 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" podStartSLOduration=8.365744771 podStartE2EDuration="8.365744771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:09:52.049682178 +0000 UTC m=+2311.744890137" watchObservedRunningTime="2026-10-30 20:09:59.365744771 +0000 UTC m=+2319.060952729" Oct 30 20:09:59.368191 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.368133 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29"] Oct 30 20:09:59.373216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.373191 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" Oct 30 20:09:59.385965 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.385939 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:09:59.386313 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.386292 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:09:59.397812 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.397741 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29"] Oct 30 20:09:59.462198 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.461644 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af5285a-eea9-4f07-a464-7a3925cd78a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce5f3aef-j2r29\" (UID: \"1af5285a-eea9-4f07-a464-7a3925cd78a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" Oct 30 20:09:59.462198 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.461705 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrv75\" (UniqueName: \"kubernetes.io/projected/1af5285a-eea9-4f07-a464-7a3925cd78a7-kube-api-access-jrv75\") pod \"maas-api-revoke-sub-ce5f3aef-j2r29\" (UID: \"1af5285a-eea9-4f07-a464-7a3925cd78a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" Oct 30 20:09:59.462198 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.461833 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af5285a-eea9-4f07-a464-7a3925cd78a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce5f3aef-j2r29\" (UID: \"1af5285a-eea9-4f07-a464-7a3925cd78a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" Oct 30 20:09:59.470114 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.470079 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-d5bck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" start-of-body= Oct 30 20:09:59.470314 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.470168 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" podUID="961fed7b-2029-4b0e-886f-f4d6cda91f81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.65:15021/healthz/ready\": dial tcp 10.132.1.65:15021: connect: connection refused" Oct 30 20:09:59.563854 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.563811 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af5285a-eea9-4f07-a464-7a3925cd78a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce5f3aef-j2r29\" (UID: \"1af5285a-eea9-4f07-a464-7a3925cd78a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" Oct 30 20:09:59.564062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.563863 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrv75\" (UniqueName: \"kubernetes.io/projected/1af5285a-eea9-4f07-a464-7a3925cd78a7-kube-api-access-jrv75\") pod \"maas-api-revoke-sub-ce5f3aef-j2r29\" (UID: \"1af5285a-eea9-4f07-a464-7a3925cd78a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" Oct 30 20:09:59.564062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.564016 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af5285a-eea9-4f07-a464-7a3925cd78a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce5f3aef-j2r29\" (UID: \"1af5285a-eea9-4f07-a464-7a3925cd78a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" Oct 30 20:09:59.564708 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.564677 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af5285a-eea9-4f07-a464-7a3925cd78a7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce5f3aef-j2r29\" (UID: \"1af5285a-eea9-4f07-a464-7a3925cd78a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" Oct 30 20:09:59.566712 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.566692 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af5285a-eea9-4f07-a464-7a3925cd78a7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce5f3aef-j2r29\" (UID: \"1af5285a-eea9-4f07-a464-7a3925cd78a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" Oct 30 20:09:59.583856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.583831 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrv75\" (UniqueName: \"kubernetes.io/projected/1af5285a-eea9-4f07-a464-7a3925cd78a7-kube-api-access-jrv75\") pod \"maas-api-revoke-sub-ce5f3aef-j2r29\" (UID: \"1af5285a-eea9-4f07-a464-7a3925cd78a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" Oct 30 20:09:59.698901 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.698802 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" Oct 30 20:09:59.790090 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:09:59.790024 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:00.084338 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:00.084303 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29"] Oct 30 20:10:00.084823 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:10:00.084795 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af5285a_eea9_4f07_a464_7a3925cd78a7.slice/crio-883fc669c35116eba038dbb8cde984ac57f38f009db6ee6c7c8f26bf5542dba1 WatchSource:0}: Error finding container 883fc669c35116eba038dbb8cde984ac57f38f009db6ee6c7c8f26bf5542dba1: Status 404 returned error can't find the container with id 883fc669c35116eba038dbb8cde984ac57f38f009db6ee6c7c8f26bf5542dba1 Oct 30 20:10:00.172374 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:00.172338 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" event={"ID":"1af5285a-eea9-4f07-a464-7a3925cd78a7","Type":"ContainerStarted","Data":"883fc669c35116eba038dbb8cde984ac57f38f009db6ee6c7c8f26bf5542dba1"} Oct 30 20:10:00.931538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:00.929963 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:01.187396 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:01.187128 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" event={"ID":"1af5285a-eea9-4f07-a464-7a3925cd78a7","Type":"ContainerStarted","Data":"cf7dbbcce71b02fc1961cb32691c13c9f94e8dbed35688aa87d48db10a3a5538"} Oct 30 20:10:01.218465 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:01.218423 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:10:01.219246 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:01.218845 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:10:01.220613 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:01.220586 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:10:01.237485 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:01.237419 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" podStartSLOduration=2.23739939 podStartE2EDuration="2.23739939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:09:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:01.233419544 +0000 UTC m=+2320.928627505" watchObservedRunningTime="2026-10-30 20:10:01.23739939 +0000 UTC m=+2320.932607348" Oct 30 20:10:01.937406 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:01.937364 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:02.199290 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:02.199199 5091 generic.go:356] "Generic (PLEG): container finished" podID="1af5285a-eea9-4f07-a464-7a3925cd78a7" containerID="cf7dbbcce71b02fc1961cb32691c13c9f94e8dbed35688aa87d48db10a3a5538" exitCode=0 Oct 30 20:10:02.199290 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:02.199262 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" event={"ID":"1af5285a-eea9-4f07-a464-7a3925cd78a7","Type":"ContainerDied","Data":"cf7dbbcce71b02fc1961cb32691c13c9f94e8dbed35688aa87d48db10a3a5538"} Oct 30 20:10:02.887167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:02.886903 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:03.395812 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:03.395785 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" Oct 30 20:10:03.478048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:03.477962 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1af5285a-eea9-4f07-a464-7a3925cd78a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af5285a-eea9-4f07-a464-7a3925cd78a7-maas-api-openshift-ca\") pod \"1af5285a-eea9-4f07-a464-7a3925cd78a7\" (UID: \"1af5285a-eea9-4f07-a464-7a3925cd78a7\") " Oct 30 20:10:03.478253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:03.478201 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1af5285a-eea9-4f07-a464-7a3925cd78a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af5285a-eea9-4f07-a464-7a3925cd78a7-maas-api-xks-ca\") pod \"1af5285a-eea9-4f07-a464-7a3925cd78a7\" (UID: \"1af5285a-eea9-4f07-a464-7a3925cd78a7\") " Oct 30 20:10:03.478253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:03.478237 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af5285a-eea9-4f07-a464-7a3925cd78a7-kube-api-access-jrv75\" (UniqueName: \"kubernetes.io/projected/1af5285a-eea9-4f07-a464-7a3925cd78a7-kube-api-access-jrv75\") pod \"1af5285a-eea9-4f07-a464-7a3925cd78a7\" (UID: \"1af5285a-eea9-4f07-a464-7a3925cd78a7\") " Oct 30 20:10:03.479018 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:03.478987 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1af5285a-eea9-4f07-a464-7a3925cd78a7-maas-api-openshift-ca" pod "1af5285a-eea9-4f07-a464-7a3925cd78a7" (UID: "1af5285a-eea9-4f07-a464-7a3925cd78a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:10:03.479297 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:03.479270 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1af5285a-eea9-4f07-a464-7a3925cd78a7-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:03.481120 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:03.481082 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1af5285a-eea9-4f07-a464-7a3925cd78a7-maas-api-xks-ca" pod "1af5285a-eea9-4f07-a464-7a3925cd78a7" (UID: "1af5285a-eea9-4f07-a464-7a3925cd78a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:10:03.481742 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:03.481715 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af5285a-eea9-4f07-a464-7a3925cd78a7-kube-api-access-jrv75" pod "1af5285a-eea9-4f07-a464-7a3925cd78a7" (UID: "1af5285a-eea9-4f07-a464-7a3925cd78a7"). InnerVolumeSpecName "kube-api-access-jrv75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:10:03.580859 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:03.580814 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1af5285a-eea9-4f07-a464-7a3925cd78a7-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:03.580859 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:03.580853 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrv75\" (UniqueName: \"kubernetes.io/projected/1af5285a-eea9-4f07-a464-7a3925cd78a7-kube-api-access-jrv75\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:10:03.744228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:03.732441 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:04.218109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:04.218079 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" Oct 30 20:10:04.218355 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:04.218122 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29" event={"ID":"1af5285a-eea9-4f07-a464-7a3925cd78a7","Type":"ContainerDied","Data":"883fc669c35116eba038dbb8cde984ac57f38f009db6ee6c7c8f26bf5542dba1"} Oct 30 20:10:04.218355 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:04.218195 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="883fc669c35116eba038dbb8cde984ac57f38f009db6ee6c7c8f26bf5542dba1" Oct 30 20:10:04.738621 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:04.738521 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:05.606168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:05.603579 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:06.611303 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:06.611243 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:07.488284 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:07.484919 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:08.369177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:08.366796 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:09.217663 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:09.217623 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:10.445249 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:10.445208 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:10.661940 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:10.661897 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:10.850182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:10.848471 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:11.314957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:11.314923 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:11.720766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:11.720676 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:12.641977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:12.639604 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:13.481421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:13.481383 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:14.397130 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:14.395890 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:14.544432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:14.544392 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:15.299073 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:15.299022 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:15.980986 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:10:15.980950 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:10:16.321774 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:16.321738 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:17.191344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:17.191303 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:18.141274 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:18.141238 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:18.985048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:18.984982 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:19.234026 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:19.233988 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:19.416637 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:19.416592 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:19.901165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:19.899287 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:20.970032 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:20.969987 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:21.921167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:21.912946 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:22.880187 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:22.876404 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:23.793877 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:23.793790 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:24.832167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:24.830633 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:25.123483 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:25.123303 5091 scope.go:122] "RemoveContainer" containerID="7ba9a28061d1b343d1ac6f301b21f7757e94553749a6ad7b389e038fca168473" Oct 30 20:10:25.834007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:25.833955 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:27.069171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:27.063048 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:27.969180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:27.968279 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:10:28.696873 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:28.696834 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:28.962290 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:28.962190 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:29.190414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:29.190377 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:29.918942 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:29.817288 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:29.935166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:29.923193 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6"] Oct 30 20:10:29.952209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:29.952175 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1af5285a-eea9-4f07-a464-7a3925cd78a7" containerName="revoke-keys" Oct 30 20:10:29.953435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:29.953406 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1af5285a-eea9-4f07-a464-7a3925cd78a7" containerName="revoke-keys" Oct 30 20:10:29.953964 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:29.953943 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1af5285a-eea9-4f07-a464-7a3925cd78a7" containerName="revoke-keys" Oct 30 20:10:30.211729 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.211635 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:30.232606 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.232575 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:30.243278 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.243248 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"openshift-service-ca.crt\"" Oct 30 20:10:30.243760 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.243741 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"routing-overlay\"" Oct 30 20:10:30.244077 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.244061 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4d5758\"" Oct 30 20:10:30.246720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.246695 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"kube-root-ca.crt\"" Oct 30 20:10:30.246887 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.246867 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"agc-cred-4a17ee2c812f\"" Oct 30 20:10:30.254859 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.254835 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6"] Oct 30 20:10:30.319423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.319381 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0675912-3f28-4aa1-99bf-c1903edee0dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:30.338938 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.338215 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvc5d\" (UniqueName: \"kubernetes.io/projected/c0675912-3f28-4aa1-99bf-c1903edee0dc-kube-api-access-bvc5d\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:30.338938 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.338301 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0675912-3f28-4aa1-99bf-c1903edee0dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:30.338938 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.338362 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0675912-3f28-4aa1-99bf-c1903edee0dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:30.368238 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.368198 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf"] Oct 30 20:10:30.436584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.423015 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:30.436584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.427928 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf"] Oct 30 20:10:30.436584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.428045 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:30.452168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.442939 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0675912-3f28-4aa1-99bf-c1903edee0dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:30.465209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.456503 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvc5d\" (UniqueName: \"kubernetes.io/projected/c0675912-3f28-4aa1-99bf-c1903edee0dc-kube-api-access-bvc5d\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:30.465487 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.465461 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0675912-3f28-4aa1-99bf-c1903edee0dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:30.465666 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.465646 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0675912-3f28-4aa1-99bf-c1903edee0dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:30.492460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.471285 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0675912-3f28-4aa1-99bf-c1903edee0dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:30.492460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.472461 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"agc-cred-c8bb93f01837\"" Oct 30 20:10:30.492460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.472579 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"payload-processing-external-model-e2e--8e4491f3-dockercfg-rdtdr\"" Oct 30 20:10:30.492460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.472654 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"payload-processing-external-model-plugins-e2e-extmod-a-222480\"" Oct 30 20:10:30.492460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.472792 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"openshift-service-ca.crt\"" Oct 30 20:10:30.492460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.472854 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"routing-overlay\"" Oct 30 20:10:30.492460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.472906 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"kube-root-ca.crt\"" Oct 30 20:10:30.492460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.479356 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0675912-3f28-4aa1-99bf-c1903edee0dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:30.535166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.529909 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvc5d\" (UniqueName: \"kubernetes.io/projected/c0675912-3f28-4aa1-99bf-c1903edee0dc-kube-api-access-bvc5d\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:30.587170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.585797 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b066923b-fa1a-4e9d-a960-18cd1854dad3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f9468qf\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:30.587170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.585970 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b066923b-fa1a-4e9d-a960-18cd1854dad3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f9468qf\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:30.587170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.586055 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrxx8\" (UniqueName: \"kubernetes.io/projected/b066923b-fa1a-4e9d-a960-18cd1854dad3-kube-api-access-vrxx8\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f9468qf\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:30.587170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.586119 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b066923b-fa1a-4e9d-a960-18cd1854dad3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f9468qf\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:30.601385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.588737 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0675912-3f28-4aa1-99bf-c1903edee0dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:30.646973 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.646937 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:30.688216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.688184 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b066923b-fa1a-4e9d-a960-18cd1854dad3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f9468qf\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:30.688350 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.688306 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b066923b-fa1a-4e9d-a960-18cd1854dad3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f9468qf\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:30.688350 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.688347 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrxx8\" (UniqueName: \"kubernetes.io/projected/b066923b-fa1a-4e9d-a960-18cd1854dad3-kube-api-access-vrxx8\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f9468qf\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:30.688437 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.688384 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b066923b-fa1a-4e9d-a960-18cd1854dad3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f9468qf\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:30.689264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.689203 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b066923b-fa1a-4e9d-a960-18cd1854dad3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f9468qf\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:30.689420 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.689395 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b066923b-fa1a-4e9d-a960-18cd1854dad3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f9468qf\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:30.691074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.691051 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b066923b-fa1a-4e9d-a960-18cd1854dad3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f9468qf\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:30.704111 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.704072 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrxx8\" (UniqueName: \"kubernetes.io/projected/b066923b-fa1a-4e9d-a960-18cd1854dad3-kube-api-access-vrxx8\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f9468qf\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:30.836946 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.836910 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:30.944761 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:10:30.944726 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0675912_3f28_4aa1_99bf_c1903edee0dc.slice/crio-59cccdda8fb1fe4b4671c5e00356973600ae2c71fecda0ebe0a39955140618cb WatchSource:0}: Error finding container 59cccdda8fb1fe4b4671c5e00356973600ae2c71fecda0ebe0a39955140618cb: Status 404 returned error can't find the container with id 59cccdda8fb1fe4b4671c5e00356973600ae2c71fecda0ebe0a39955140618cb Oct 30 20:10:30.948439 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:30.948411 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6"] Oct 30 20:10:31.135740 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:10:31.135709 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb066923b_fa1a_4e9d_a960_18cd1854dad3.slice/crio-ebdbcfbeb87dfaf0be945c2bc235b1cf2926ba0453cf1235a08c0c982f7f6d14 WatchSource:0}: Error finding container ebdbcfbeb87dfaf0be945c2bc235b1cf2926ba0453cf1235a08c0c982f7f6d14: Status 404 returned error can't find the container with id ebdbcfbeb87dfaf0be945c2bc235b1cf2926ba0453cf1235a08c0c982f7f6d14 Oct 30 20:10:31.137615 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:31.137467 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf"] Oct 30 20:10:31.322168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:31.315935 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:31.762252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:31.762208 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" event={"ID":"b066923b-fa1a-4e9d-a960-18cd1854dad3","Type":"ContainerStarted","Data":"a193703e7ef70b22c0826d8b28e900113b0169423f28216d914b42be76761909"} Oct 30 20:10:31.762455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:31.762276 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" event={"ID":"b066923b-fa1a-4e9d-a960-18cd1854dad3","Type":"ContainerStarted","Data":"ebdbcfbeb87dfaf0be945c2bc235b1cf2926ba0453cf1235a08c0c982f7f6d14"} Oct 30 20:10:31.762455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:31.762304 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:31.763871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:31.763844 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" event={"ID":"c0675912-3f28-4aa1-99bf-c1903edee0dc","Type":"ContainerStarted","Data":"dc9f38aebc0d88383614a040792d829b71bd6793e4e52d04701cc179116c5861"} Oct 30 20:10:31.764007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:31.763894 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" event={"ID":"c0675912-3f28-4aa1-99bf-c1903edee0dc","Type":"ContainerStarted","Data":"59cccdda8fb1fe4b4671c5e00356973600ae2c71fecda0ebe0a39955140618cb"} Oct 30 20:10:31.764007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:31.763948 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:31.798470 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:31.798419 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" podStartSLOduration=1.798404546 podStartE2EDuration="1.798404546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:31.792971246 +0000 UTC m=+2351.488179199" watchObservedRunningTime="2026-10-30 20:10:31.798404546 +0000 UTC m=+2351.493612521" Oct 30 20:10:31.825023 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:31.824971 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" podStartSLOduration=2.8249611850000003 podStartE2EDuration="2.824961185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:31.824509532 +0000 UTC m=+2351.519717490" watchObservedRunningTime="2026-10-30 20:10:31.824961185 +0000 UTC m=+2351.520169143" Oct 30 20:10:32.265178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:32.264103 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:33.212930 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:33.212873 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:34.230420 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:34.230379 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:35.057167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:35.043123 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:36.032051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.031971 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:36.218859 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.218822 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb"] Oct 30 20:10:36.224109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.224089 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.245051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.245018 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb"] Oct 30 20:10:36.261899 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.261863 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxngd\" (UniqueName: \"kubernetes.io/projected/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-kube-api-access-dxngd\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.262183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.261927 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.262183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.261990 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.262183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.262039 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.262183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.262174 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.262518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.262249 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.262518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.262309 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.262518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.262354 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.262692 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.262542 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.364773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.364374 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.364773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.364466 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.364773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.364523 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.364773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.364577 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.364773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.364658 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.364773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.364721 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.364773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.364770 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.365354 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.364892 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-workload-certs\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.365354 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.364967 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.365354 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.365107 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxngd\" (UniqueName: \"kubernetes.io/projected/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-kube-api-access-dxngd\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.365531 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.365378 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.365931 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.365776 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-credential-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.366165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.365996 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-workload-socket\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.366371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.366350 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-istio-data\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.368067 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.368043 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-istio-envoy\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.368910 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.368880 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.383457 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.383405 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-istio-token\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.383968 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.383943 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxngd\" (UniqueName: \"kubernetes.io/projected/d0dfebf4-1e1b-46a5-86aa-cb488de34ca2-kube-api-access-dxngd\") pod \"maas-default-gateway-openshift-default-5db77fbd9f-84zhb\" (UID: \"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.542173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.542116 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:36.865381 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.865345 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:10:36.865496 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.865415 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:10:36.865496 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.865446 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:10:36.871237 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:10:36.862704 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0dfebf4_1e1b_46a5_86aa_cb488de34ca2.slice/crio-464155d01aab9f45f25cb40ac034b9b0928c5488f6117032153b45a9430c1dca WatchSource:0}: Error finding container 464155d01aab9f45f25cb40ac034b9b0928c5488f6117032153b45a9430c1dca: Status 404 returned error can't find the container with id 464155d01aab9f45f25cb40ac034b9b0928c5488f6117032153b45a9430c1dca Oct 30 20:10:36.871725 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:36.871700 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb"] Oct 30 20:10:37.173897 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:37.173801 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:37.818876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:37.818837 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" event={"ID":"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2","Type":"ContainerStarted","Data":"35a7f89f123bf30ef68f28450fcb366fec0753b75bfc8d5becfed2e299a4df99"} Oct 30 20:10:37.819071 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:37.818928 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" event={"ID":"d0dfebf4-1e1b-46a5-86aa-cb488de34ca2","Type":"ContainerStarted","Data":"464155d01aab9f45f25cb40ac034b9b0928c5488f6117032153b45a9430c1dca"} Oct 30 20:10:37.892219 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:37.887958 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" podStartSLOduration=1.88794554 podStartE2EDuration="1.88794554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:10:37.887620577 +0000 UTC m=+2357.582828535" watchObservedRunningTime="2026-10-30 20:10:37.88794554 +0000 UTC m=+2357.583153547" Oct 30 20:10:38.142299 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:38.141030 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:38.542970 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:38.542938 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:38.544412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:38.544388 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-84zhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 30 20:10:38.544539 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:38.544436 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" podUID="d0dfebf4-1e1b-46a5-86aa-cb488de34ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 30 20:10:39.111782 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:39.111741 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:39.542842 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:39.542803 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-84zhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 30 20:10:39.543040 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:39.542852 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" podUID="d0dfebf4-1e1b-46a5-86aa-cb488de34ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 30 20:10:40.015518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:39.997825 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:40.543431 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:40.543389 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-84zhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 30 20:10:40.543920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:40.543453 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" podUID="d0dfebf4-1e1b-46a5-86aa-cb488de34ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 30 20:10:40.909237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:40.905505 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:41.542981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:41.542949 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-84zhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 30 20:10:41.543207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:41.542996 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" podUID="d0dfebf4-1e1b-46a5-86aa-cb488de34ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 30 20:10:41.988370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:41.986759 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:42.542486 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:42.542450 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-84zhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 30 20:10:42.542690 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:42.542505 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" podUID="d0dfebf4-1e1b-46a5-86aa-cb488de34ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 30 20:10:42.780310 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:42.779544 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:10:42.804236 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:42.796223 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:10:42.990173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:42.973890 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:43.543123 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:43.543079 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-84zhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 30 20:10:43.543354 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:43.543167 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" podUID="d0dfebf4-1e1b-46a5-86aa-cb488de34ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 30 20:10:43.773105 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:43.773066 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:44.543199 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:44.543138 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-84zhb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" start-of-body= Oct 30 20:10:44.543673 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:44.543232 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" podUID="d0dfebf4-1e1b-46a5-86aa-cb488de34ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.69:15021/healthz/ready\": dial tcp 10.132.1.69:15021: connect: connection refused" Oct 30 20:10:44.576166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:44.575445 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:45.414454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:45.414414 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:45.983165 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:10:45.982750 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:10:46.012559 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:46.012523 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:46.014404 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:46.014365 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:46.014958 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:46.014939 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-84zhb" Oct 30 20:10:46.339169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:46.339118 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:47.343510 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:47.343474 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:48.524209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:48.520107 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:49.802207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:49.802159 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:50.220566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:50.220456 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:50.525884 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:50.525783 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:50.915658 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:50.915559 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:51.124181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:51.122226 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:52.088215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:52.088172 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:53.001824 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:53.001730 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:53.911168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:53.905970 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:54.758060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:54.758023 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:55.537760 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:55.537715 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:56.605611 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:56.605549 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:57.279167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:57.278133 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:58.198309 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:58.198265 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:58.626423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:58.626387 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:59.075006 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:59.068591 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:10:59.962177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:10:59.958719 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:00.811875 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:00.811834 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:01.689992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:01.689956 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:02.614984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:02.614943 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:03.478310 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:03.478268 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:04.413920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:04.413876 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:05.249768 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:05.249725 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:06.448169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:06.440895 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:07.390902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:07.390863 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:08.297654 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:08.295966 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:09.170952 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:09.170836 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:10.019440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:10.019328 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:10.727394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:10.727348 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:10.930172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:10.924115 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:11.792720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:11.792680 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:12.702300 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:12.702261 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:13.558672 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:13.558630 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:14.471063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:14.471007 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:14.615084 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:14.615027 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:15.530380 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:15.530337 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:15.980767 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:11:15.980660 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:11:16.290743 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:16.290698 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:17.032015 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:17.031978 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw"] Oct 30 20:11:17.039543 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:17.039518 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ea0d3b9-xw6bw"] Oct 30 20:11:17.255821 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:17.255785 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:18.181959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:18.181912 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:18.443285 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:18.443211 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0c7bc95-72c8-4b53-9a9e-c0f069783697" path="/var/lib/kubelet/pods/c0c7bc95-72c8-4b53-9a9e-c0f069783697/volumes" Oct 30 20:11:19.055829 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:19.055791 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:19.932245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:19.927715 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:20.055374 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:20.055341 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f"] Oct 30 20:11:20.060187 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:20.060139 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89cb5bcb-k6s5f"] Oct 30 20:11:20.444487 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:20.444453 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90e48ff-e0d8-4495-b447-ccdd7fb3935c" path="/var/lib/kubelet/pods/e90e48ff-e0d8-4495-b447-ccdd7fb3935c/volumes" Oct 30 20:11:20.995226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:20.995190 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:21.374432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:21.374335 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 20:11:21.377923 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:21.377863 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 20:11:21.819922 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:21.819868 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:22.667318 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:22.667105 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:23.517026 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:23.516989 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:24.606748 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:24.606706 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:25.322157 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:25.322106 5091 scope.go:122] "RemoveContainer" containerID="429e936c9e7860d43805f089020c786a596c6182f882491aab14e72ce4d823b8" Oct 30 20:11:25.579266 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:25.574362 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:26.509302 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:26.505467 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:27.427781 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:27.427742 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:28.257171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:28.253134 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:29.196483 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:29.196444 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:29.975247 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:29.975210 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:30.885683 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:30.885631 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:31.693172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:31.689403 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:32.063356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:32.063322 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5"] Oct 30 20:11:32.071180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:32.071131 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dcd17e7c-9n6p5"] Oct 30 20:11:32.445115 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:32.445029 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500fdafc-51b5-4c27-9615-14899c5cc65a" path="/var/lib/kubelet/pods/500fdafc-51b5-4c27-9615-14899c5cc65a/volumes" Oct 30 20:11:32.661722 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:32.661658 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:33.587503 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:33.578765 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:34.411407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:34.411371 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:35.122459 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:35.122424 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk"] Oct 30 20:11:35.122955 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:35.122674 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" podUID="d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" containerName="istio-proxy" containerID="cri-o://2e6cc51cbe80b1206d91fe7e6c24f43e07ef13f8ee924d3443cf4a5c0470ddf4" gracePeriod=30 Oct 30 20:11:35.305130 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:35.303260 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:36.201060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:36.201012 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:37.200735 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:37.200691 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:38.018877 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:38.018835 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:39.007388 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:39.007352 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:39.890089 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:39.890048 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:40.440460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.440429 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 20:11:40.572696 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.572658 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istiod-ca-cert\") pod \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " Oct 30 20:11:40.572868 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.572724 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-kube-api-access-2dv85\" (UniqueName: \"kubernetes.io/projected/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-kube-api-access-2dv85\") pod \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " Oct 30 20:11:40.572868 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.572768 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-podinfo\") pod \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " Oct 30 20:11:40.572868 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.572799 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-workload-socket\") pod \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " Oct 30 20:11:40.573084 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.572899 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-token\" (UniqueName: \"kubernetes.io/projected/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-token\") pod \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " Oct 30 20:11:40.573084 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.573034 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-credential-socket\") pod \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " Oct 30 20:11:40.573084 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.573049 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istiod-ca-cert" pod "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" (UID: "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:11:40.573332 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.573108 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-data\") pod \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " Oct 30 20:11:40.573332 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.573234 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-envoy\") pod \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " Oct 30 20:11:40.573332 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.573275 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-workload-certs\") pod \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\" (UID: \"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1\") " Oct 30 20:11:40.573493 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.573330 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-workload-socket" pod "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" (UID: "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:40.573556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.573538 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-data" pod "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" (UID: "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:40.573632 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.573551 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-credential-socket" pod "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" (UID: "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:40.573753 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.573730 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-workload-certs" pod "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" (UID: "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:40.574186 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.574136 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:40.574739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.574191 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:40.574739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.574205 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:40.574739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.574218 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:40.574739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.574231 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:40.575299 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.575278 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-token" pod "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" (UID: "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:40.575383 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.575322 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-kube-api-access-2dv85" pod "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" (UID: "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1"). InnerVolumeSpecName "kube-api-access-2dv85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:11:40.575534 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.575516 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-podinfo" pod "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" (UID: "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:11:40.576307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.576287 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-envoy" pod "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" (UID: "d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:11:40.675831 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.675752 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:40.675831 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.675781 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:40.675831 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.675791 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dv85\" (UniqueName: \"kubernetes.io/projected/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-kube-api-access-2dv85\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:40.675831 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.675802 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:11:40.775170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.770544 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:40.849249 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.849216 5091 generic.go:356] "Generic (PLEG): container finished" podID="d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" containerID="2e6cc51cbe80b1206d91fe7e6c24f43e07ef13f8ee924d3443cf4a5c0470ddf4" exitCode=0 Oct 30 20:11:40.849460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.849280 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" Oct 30 20:11:40.849460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.849300 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" event={"ID":"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1","Type":"ContainerDied","Data":"2e6cc51cbe80b1206d91fe7e6c24f43e07ef13f8ee924d3443cf4a5c0470ddf4"} Oct 30 20:11:40.849460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.849364 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk" event={"ID":"d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1","Type":"ContainerDied","Data":"b57626616209c94135724a96552112046d628c34c3c2cc66bd5cd98a853feb72"} Oct 30 20:11:40.849460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.849398 5091 scope.go:122] "RemoveContainer" containerID="2e6cc51cbe80b1206d91fe7e6c24f43e07ef13f8ee924d3443cf4a5c0470ddf4" Oct 30 20:11:40.862778 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.862759 5091 scope.go:122] "RemoveContainer" containerID="2e6cc51cbe80b1206d91fe7e6c24f43e07ef13f8ee924d3443cf4a5c0470ddf4" Oct 30 20:11:40.863081 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:11:40.863057 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e6cc51cbe80b1206d91fe7e6c24f43e07ef13f8ee924d3443cf4a5c0470ddf4\": container with ID starting with 2e6cc51cbe80b1206d91fe7e6c24f43e07ef13f8ee924d3443cf4a5c0470ddf4 not found: ID does not exist" containerID="2e6cc51cbe80b1206d91fe7e6c24f43e07ef13f8ee924d3443cf4a5c0470ddf4" Oct 30 20:11:40.863186 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.863092 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e6cc51cbe80b1206d91fe7e6c24f43e07ef13f8ee924d3443cf4a5c0470ddf4"} err="failed to get container status \"2e6cc51cbe80b1206d91fe7e6c24f43e07ef13f8ee924d3443cf4a5c0470ddf4\": rpc error: code = NotFound desc = could not find container \"2e6cc51cbe80b1206d91fe7e6c24f43e07ef13f8ee924d3443cf4a5c0470ddf4\": container with ID starting with 2e6cc51cbe80b1206d91fe7e6c24f43e07ef13f8ee924d3443cf4a5c0470ddf4 not found: ID does not exist" Oct 30 20:11:40.883132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.883104 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk"] Oct 30 20:11:40.888537 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:40.888515 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-r7dlk"] Oct 30 20:11:41.603669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:41.603619 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:42.448032 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:42.447994 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" path="/var/lib/kubelet/pods/d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1/volumes" Oct 30 20:11:42.513845 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:42.513803 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:43.496165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:43.495788 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:44.353506 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:44.353460 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:45.244169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:45.240129 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:45.979471 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:11:45.979427 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:11:46.305493 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:46.305458 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:47.203839 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:47.203796 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:48.363327 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:48.363285 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:49.169000 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:49.168957 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:50.002170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:49.997802 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:50.789933 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:50.789897 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:51.735099 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:51.735045 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:52.526031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:52.525988 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:53.501253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:53.501217 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:54.033564 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:54.033520 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5"] Oct 30 20:11:54.038451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:54.038420 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbcf75d5-tc7v5"] Oct 30 20:11:54.444518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:54.444438 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9de2d614-d949-43a8-805d-4c828da19cc6" path="/var/lib/kubelet/pods/9de2d614-d949-43a8-805d-4c828da19cc6/volumes" Oct 30 20:11:54.450927 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:54.450889 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:55.564467 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:55.564414 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:56.433168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:56.431218 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:57.344072 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.344038 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls"] Oct 30 20:11:57.345179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.345139 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" containerName="istio-proxy" Oct 30 20:11:57.345179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.345178 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" containerName="istio-proxy" Oct 30 20:11:57.345378 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.345344 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8fca0c1-9b52-4e2d-9e1f-266cf303ffe1" containerName="istio-proxy" Oct 30 20:11:57.349529 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.349508 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" Oct 30 20:11:57.363399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.363373 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:11:57.363500 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.363420 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:11:57.371216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.371187 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls"] Oct 30 20:11:57.489321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.487305 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ce6a42-7jjls\" (UID: \"0e42c2d6-2cfc-4012-9c93-1a612809a5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" Oct 30 20:11:57.489321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.487483 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ce6a42-7jjls\" (UID: \"0e42c2d6-2cfc-4012-9c93-1a612809a5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" Oct 30 20:11:57.489321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.487578 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6wt2\" (UniqueName: \"kubernetes.io/projected/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-kube-api-access-t6wt2\") pod \"maas-api-revoke-sub-59ce6a42-7jjls\" (UID: \"0e42c2d6-2cfc-4012-9c93-1a612809a5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" Oct 30 20:11:57.574446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.574405 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:57.589195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.589169 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ce6a42-7jjls\" (UID: \"0e42c2d6-2cfc-4012-9c93-1a612809a5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" Oct 30 20:11:57.589691 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.589403 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6wt2\" (UniqueName: \"kubernetes.io/projected/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-kube-api-access-t6wt2\") pod \"maas-api-revoke-sub-59ce6a42-7jjls\" (UID: \"0e42c2d6-2cfc-4012-9c93-1a612809a5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" Oct 30 20:11:57.589691 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.589553 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ce6a42-7jjls\" (UID: \"0e42c2d6-2cfc-4012-9c93-1a612809a5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" Oct 30 20:11:57.589937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.589917 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59ce6a42-7jjls\" (UID: \"0e42c2d6-2cfc-4012-9c93-1a612809a5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" Oct 30 20:11:57.592090 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.592060 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59ce6a42-7jjls\" (UID: \"0e42c2d6-2cfc-4012-9c93-1a612809a5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" Oct 30 20:11:57.607442 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.607386 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6wt2\" (UniqueName: \"kubernetes.io/projected/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-kube-api-access-t6wt2\") pod \"maas-api-revoke-sub-59ce6a42-7jjls\" (UID: \"0e42c2d6-2cfc-4012-9c93-1a612809a5fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" Oct 30 20:11:57.664970 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.664950 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" Oct 30 20:11:57.975770 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:11:57.975736 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e42c2d6_2cfc_4012_9c93_1a612809a5fb.slice/crio-4e48a006de8167fa124529a8ddecef9c71aabb0a2cb16555cbe5385ea39d8076 WatchSource:0}: Error finding container 4e48a006de8167fa124529a8ddecef9c71aabb0a2cb16555cbe5385ea39d8076: Status 404 returned error can't find the container with id 4e48a006de8167fa124529a8ddecef9c71aabb0a2cb16555cbe5385ea39d8076 Oct 30 20:11:57.976399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:57.976374 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls"] Oct 30 20:11:58.119685 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:58.119649 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" event={"ID":"0e42c2d6-2cfc-4012-9c93-1a612809a5fb","Type":"ContainerStarted","Data":"4e48a006de8167fa124529a8ddecef9c71aabb0a2cb16555cbe5385ea39d8076"} Oct 30 20:11:58.470186 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:58.467886 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:11:59.134434 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:59.134400 5091 generic.go:356] "Generic (PLEG): container finished" podID="0e42c2d6-2cfc-4012-9c93-1a612809a5fb" containerID="dc4cdb8a18358da0f1946858cda5835fa1bef7609b6dac631b5fcf502b5128db" exitCode=0 Oct 30 20:11:59.134616 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:59.134483 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" event={"ID":"0e42c2d6-2cfc-4012-9c93-1a612809a5fb","Type":"ContainerDied","Data":"dc4cdb8a18358da0f1946858cda5835fa1bef7609b6dac631b5fcf502b5128db"} Oct 30 20:11:59.293026 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:11:59.292990 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:00.354108 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:00.353767 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:00.418743 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:00.418295 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" Oct 30 20:12:00.555851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:00.555818 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-kube-api-access-t6wt2\" (UniqueName: \"kubernetes.io/projected/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-kube-api-access-t6wt2\") pod \"0e42c2d6-2cfc-4012-9c93-1a612809a5fb\" (UID: \"0e42c2d6-2cfc-4012-9c93-1a612809a5fb\") " Oct 30 20:12:00.556051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:00.556035 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-maas-api-xks-ca\") pod \"0e42c2d6-2cfc-4012-9c93-1a612809a5fb\" (UID: \"0e42c2d6-2cfc-4012-9c93-1a612809a5fb\") " Oct 30 20:12:00.556135 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:00.556069 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-maas-api-openshift-ca\") pod \"0e42c2d6-2cfc-4012-9c93-1a612809a5fb\" (UID: \"0e42c2d6-2cfc-4012-9c93-1a612809a5fb\") " Oct 30 20:12:00.556539 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:00.556509 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-maas-api-openshift-ca" pod "0e42c2d6-2cfc-4012-9c93-1a612809a5fb" (UID: "0e42c2d6-2cfc-4012-9c93-1a612809a5fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:00.556789 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:00.556768 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:00.558210 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:00.558175 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-kube-api-access-t6wt2" pod "0e42c2d6-2cfc-4012-9c93-1a612809a5fb" (UID: "0e42c2d6-2cfc-4012-9c93-1a612809a5fb"). InnerVolumeSpecName "kube-api-access-t6wt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:00.558319 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:00.558205 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-maas-api-xks-ca" pod "0e42c2d6-2cfc-4012-9c93-1a612809a5fb" (UID: "0e42c2d6-2cfc-4012-9c93-1a612809a5fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:00.658325 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:00.658285 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6wt2\" (UniqueName: \"kubernetes.io/projected/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-kube-api-access-t6wt2\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:00.658325 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:00.658322 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e42c2d6-2cfc-4012-9c93-1a612809a5fb-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:01.154731 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:01.154703 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" Oct 30 20:12:01.154961 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:01.154700 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls" event={"ID":"0e42c2d6-2cfc-4012-9c93-1a612809a5fb","Type":"ContainerDied","Data":"4e48a006de8167fa124529a8ddecef9c71aabb0a2cb16555cbe5385ea39d8076"} Oct 30 20:12:01.154961 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:01.154833 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e48a006de8167fa124529a8ddecef9c71aabb0a2cb16555cbe5385ea39d8076" Oct 30 20:12:01.413930 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:01.413819 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:02.460384 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:02.460344 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:03.434398 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:03.433745 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:04.032263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:04.032230 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb"] Oct 30 20:12:04.036996 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:04.036967 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00b66701-rnqdb"] Oct 30 20:12:04.303224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:04.302727 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:04.442753 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:04.442721 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a86cae-73d3-44a4-8a42-c7bcddaf557d" path="/var/lib/kubelet/pods/f5a86cae-73d3-44a4-8a42-c7bcddaf557d/volumes" Oct 30 20:12:05.049371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:05.049335 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt"] Oct 30 20:12:05.054127 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:05.054095 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-081b6330-h66dt"] Oct 30 20:12:05.213256 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:05.213217 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:05.788462 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:05.788425 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:05.997176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:05.996957 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:06.444375 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:06.444347 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f10798fc-4b1c-464d-88a0-0849293cf3ec" path="/var/lib/kubelet/pods/f10798fc-4b1c-464d-88a0-0849293cf3ec/volumes" Oct 30 20:12:07.003642 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:07.003602 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:07.918850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:07.918804 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:08.530981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:08.530941 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:08.690731 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:08.690691 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:09.043270 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.043230 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:09.380312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.380231 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg"] Oct 30 20:12:09.381106 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.381088 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0e42c2d6-2cfc-4012-9c93-1a612809a5fb" containerName="revoke-keys" Oct 30 20:12:09.381174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.381107 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e42c2d6-2cfc-4012-9c93-1a612809a5fb" containerName="revoke-keys" Oct 30 20:12:09.381235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.381225 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e42c2d6-2cfc-4012-9c93-1a612809a5fb" containerName="revoke-keys" Oct 30 20:12:09.386536 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.386493 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" Oct 30 20:12:09.395681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.395653 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg"] Oct 30 20:12:09.396430 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.396395 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:12:09.396549 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.396514 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:12:09.556810 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.556770 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49a5707-dc8c-4371-835f-8c9f99a4cc94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f19b333-hggmg\" (UID: \"a49a5707-dc8c-4371-835f-8c9f99a4cc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" Oct 30 20:12:09.557426 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.557401 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49a5707-dc8c-4371-835f-8c9f99a4cc94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f19b333-hggmg\" (UID: \"a49a5707-dc8c-4371-835f-8c9f99a4cc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" Oct 30 20:12:09.557757 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.557732 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrzqm\" (UniqueName: \"kubernetes.io/projected/a49a5707-dc8c-4371-835f-8c9f99a4cc94-kube-api-access-rrzqm\") pod \"maas-api-revoke-sub-4f19b333-hggmg\" (UID: \"a49a5707-dc8c-4371-835f-8c9f99a4cc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" Oct 30 20:12:09.659682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.659591 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49a5707-dc8c-4371-835f-8c9f99a4cc94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f19b333-hggmg\" (UID: \"a49a5707-dc8c-4371-835f-8c9f99a4cc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" Oct 30 20:12:09.659853 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.659742 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrzqm\" (UniqueName: \"kubernetes.io/projected/a49a5707-dc8c-4371-835f-8c9f99a4cc94-kube-api-access-rrzqm\") pod \"maas-api-revoke-sub-4f19b333-hggmg\" (UID: \"a49a5707-dc8c-4371-835f-8c9f99a4cc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" Oct 30 20:12:09.659853 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.659799 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49a5707-dc8c-4371-835f-8c9f99a4cc94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f19b333-hggmg\" (UID: \"a49a5707-dc8c-4371-835f-8c9f99a4cc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" Oct 30 20:12:09.660695 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.660666 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49a5707-dc8c-4371-835f-8c9f99a4cc94-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f19b333-hggmg\" (UID: \"a49a5707-dc8c-4371-835f-8c9f99a4cc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" Oct 30 20:12:09.662218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.662188 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49a5707-dc8c-4371-835f-8c9f99a4cc94-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f19b333-hggmg\" (UID: \"a49a5707-dc8c-4371-835f-8c9f99a4cc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" Oct 30 20:12:09.674846 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.674820 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrzqm\" (UniqueName: \"kubernetes.io/projected/a49a5707-dc8c-4371-835f-8c9f99a4cc94-kube-api-access-rrzqm\") pod \"maas-api-revoke-sub-4f19b333-hggmg\" (UID: \"a49a5707-dc8c-4371-835f-8c9f99a4cc94\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" Oct 30 20:12:09.708614 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:09.708585 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" Oct 30 20:12:10.008939 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:12:10.008909 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda49a5707_dc8c_4371_835f_8c9f99a4cc94.slice/crio-5b7a3541fb44996cb24553a680ac067545c6449ea0e6806ec7f8603861823448 WatchSource:0}: Error finding container 5b7a3541fb44996cb24553a680ac067545c6449ea0e6806ec7f8603861823448: Status 404 returned error can't find the container with id 5b7a3541fb44996cb24553a680ac067545c6449ea0e6806ec7f8603861823448 Oct 30 20:12:10.010299 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:10.010276 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg"] Oct 30 20:12:10.110472 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:10.110382 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:10.275665 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:10.275618 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" event={"ID":"a49a5707-dc8c-4371-835f-8c9f99a4cc94","Type":"ContainerStarted","Data":"5b7a3541fb44996cb24553a680ac067545c6449ea0e6806ec7f8603861823448"} Oct 30 20:12:10.886208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:10.883751 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:11.027941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:11.027904 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5"] Oct 30 20:12:11.028133 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:11.028102 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" podUID="8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" containerName="istio-proxy" containerID="cri-o://d32608cb95951f9df5895e80a0aee54dfb7c209619e81dae4c80c27701d78af1" gracePeriod=30 Oct 30 20:12:11.286401 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:11.286362 5091 generic.go:356] "Generic (PLEG): container finished" podID="a49a5707-dc8c-4371-835f-8c9f99a4cc94" containerID="fbb75e3e5d98b78438aaeb6aef83c661f79a7c3432067bb38ef11af13a58c431" exitCode=0 Oct 30 20:12:11.286857 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:11.286453 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" event={"ID":"a49a5707-dc8c-4371-835f-8c9f99a4cc94","Type":"ContainerDied","Data":"fbb75e3e5d98b78438aaeb6aef83c661f79a7c3432067bb38ef11af13a58c431"} Oct 30 20:12:11.933187 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:11.932336 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:12.496243 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:12.496210 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" Oct 30 20:12:12.637929 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:12.637845 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a49a5707-dc8c-4371-835f-8c9f99a4cc94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49a5707-dc8c-4371-835f-8c9f99a4cc94-maas-api-openshift-ca\") pod \"a49a5707-dc8c-4371-835f-8c9f99a4cc94\" (UID: \"a49a5707-dc8c-4371-835f-8c9f99a4cc94\") " Oct 30 20:12:12.638094 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:12.638003 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a49a5707-dc8c-4371-835f-8c9f99a4cc94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49a5707-dc8c-4371-835f-8c9f99a4cc94-maas-api-xks-ca\") pod \"a49a5707-dc8c-4371-835f-8c9f99a4cc94\" (UID: \"a49a5707-dc8c-4371-835f-8c9f99a4cc94\") " Oct 30 20:12:12.638287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:12.638263 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49a5707-dc8c-4371-835f-8c9f99a4cc94-maas-api-openshift-ca" pod "a49a5707-dc8c-4371-835f-8c9f99a4cc94" (UID: "a49a5707-dc8c-4371-835f-8c9f99a4cc94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:12.638437 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:12.638266 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49a5707-dc8c-4371-835f-8c9f99a4cc94-kube-api-access-rrzqm\" (UniqueName: \"kubernetes.io/projected/a49a5707-dc8c-4371-835f-8c9f99a4cc94-kube-api-access-rrzqm\") pod \"a49a5707-dc8c-4371-835f-8c9f99a4cc94\" (UID: \"a49a5707-dc8c-4371-835f-8c9f99a4cc94\") " Oct 30 20:12:12.639010 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:12.638985 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a49a5707-dc8c-4371-835f-8c9f99a4cc94-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:12.640421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:12.640393 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49a5707-dc8c-4371-835f-8c9f99a4cc94-kube-api-access-rrzqm" pod "a49a5707-dc8c-4371-835f-8c9f99a4cc94" (UID: "a49a5707-dc8c-4371-835f-8c9f99a4cc94"). InnerVolumeSpecName "kube-api-access-rrzqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:12.640656 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:12.640632 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a49a5707-dc8c-4371-835f-8c9f99a4cc94-maas-api-xks-ca" pod "a49a5707-dc8c-4371-835f-8c9f99a4cc94" (UID: "a49a5707-dc8c-4371-835f-8c9f99a4cc94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:12:12.740321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:12.740278 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrzqm\" (UniqueName: \"kubernetes.io/projected/a49a5707-dc8c-4371-835f-8c9f99a4cc94-kube-api-access-rrzqm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:12.740321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:12.740310 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a49a5707-dc8c-4371-835f-8c9f99a4cc94-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:12.908248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:12.900522 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:13.306813 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:13.306780 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" Oct 30 20:12:13.306813 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:13.306799 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg" event={"ID":"a49a5707-dc8c-4371-835f-8c9f99a4cc94","Type":"ContainerDied","Data":"5b7a3541fb44996cb24553a680ac067545c6449ea0e6806ec7f8603861823448"} Oct 30 20:12:13.307053 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:13.306854 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b7a3541fb44996cb24553a680ac067545c6449ea0e6806ec7f8603861823448" Oct 30 20:12:13.986181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:13.985830 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:14.615601 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:14.615561 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:15.117271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:15.117241 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:15.494470 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:15.494390 5091 patch_prober.go:29] interesting pod/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:12:15.494470 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:15.494455 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" podUID="8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:12:15.981249 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:12:15.981209 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:12:16.031080 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.031044 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r"] Oct 30 20:12:16.037340 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.037311 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-059fd74b-hsq5r"] Oct 30 20:12:16.123561 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.123525 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:16.338805 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.338776 5091 generic.go:356] "Generic (PLEG): container finished" podID="8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" containerID="d32608cb95951f9df5895e80a0aee54dfb7c209619e81dae4c80c27701d78af1" exitCode=0 Oct 30 20:12:16.338984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.338850 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" event={"ID":"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6","Type":"ContainerDied","Data":"d32608cb95951f9df5895e80a0aee54dfb7c209619e81dae4c80c27701d78af1"} Oct 30 20:12:16.338984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.338946 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" event={"ID":"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6","Type":"ContainerDied","Data":"c7057fd72f869a7c6c6fb4ef29c48af5303a06e442288ecc58bac4708ea21e1c"} Oct 30 20:12:16.338984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.338970 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7057fd72f869a7c6c6fb4ef29c48af5303a06e442288ecc58bac4708ea21e1c" Oct 30 20:12:16.359398 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.359373 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 20:12:16.423333 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.423297 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istiod-ca-cert\") pod \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " Oct 30 20:12:16.423540 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.423384 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-credential-socket\") pod \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " Oct 30 20:12:16.423540 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.423416 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-token\" (UniqueName: \"kubernetes.io/projected/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-token\") pod \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " Oct 30 20:12:16.423540 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.423448 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-kube-api-access-dzrn2\" (UniqueName: \"kubernetes.io/projected/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-kube-api-access-dzrn2\") pod \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " Oct 30 20:12:16.423540 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.423486 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-data\") pod \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " Oct 30 20:12:16.423789 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.423571 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-podinfo\") pod \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " Oct 30 20:12:16.423789 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.423748 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istiod-ca-cert" pod "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" (UID: "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:12:16.423905 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.423816 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-workload-certs\") pod \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " Oct 30 20:12:16.423999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.423979 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-workload-socket\") pod \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " Oct 30 20:12:16.424077 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.424061 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-envoy\") pod \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\" (UID: \"8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6\") " Oct 30 20:12:16.425174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.424538 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-workload-certs" pod "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" (UID: "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:16.425174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.424946 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:16.425174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.424971 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:16.425496 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.425466 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-workload-socket" pod "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" (UID: "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:16.425750 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.425727 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-data" pod "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" (UID: "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:16.425988 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.425965 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-credential-socket" pod "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" (UID: "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:16.426490 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.426462 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-token" pod "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" (UID: "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:16.426927 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.426905 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-envoy" pod "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" (UID: "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:12:16.427136 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.427087 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-podinfo" pod "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" (UID: "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:12:16.427863 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.427839 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-kube-api-access-dzrn2" pod "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" (UID: "8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6"). InnerVolumeSpecName "kube-api-access-dzrn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:12:16.450175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.450132 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f848a7a0-fafe-4b2b-b8f7-e97ecc993e37" path="/var/lib/kubelet/pods/f848a7a0-fafe-4b2b-b8f7-e97ecc993e37/volumes" Oct 30 20:12:16.526594 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.526559 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:16.526594 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.526592 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:16.526808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.526604 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:16.526808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.526614 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:16.526808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.526622 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:16.526808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.526631 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzrn2\" (UniqueName: \"kubernetes.io/projected/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-kube-api-access-dzrn2\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:16.526808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:16.526640 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:12:17.078169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:17.076124 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:17.104166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:17.097928 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj"] Oct 30 20:12:17.124583 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:17.124375 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13a45d69-4ljnj"] Oct 30 20:12:17.348383 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:17.348287 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5" Oct 30 20:12:17.377808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:17.377773 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5"] Oct 30 20:12:17.382314 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:17.382285 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-znfv5"] Oct 30 20:12:17.991172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:17.990163 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:18.444204 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:18.444167 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" path="/var/lib/kubelet/pods/8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6/volumes" Oct 30 20:12:18.444792 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:18.444774 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997a1e67-2b6d-4322-b333-04eb9b498bd3" path="/var/lib/kubelet/pods/997a1e67-2b6d-4322-b333-04eb9b498bd3/volumes" Oct 30 20:12:18.988896 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:18.988856 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:20.000170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:19.993258 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:20.519039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:20.518996 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:21.063607 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:21.063530 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:21.230209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:21.230167 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:21.424159 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:21.424058 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:22.383225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:22.383183 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:23.180895 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:23.180857 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:24.284159 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:24.284108 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:25.048428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:25.048334 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:25.357574 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:25.357494 5091 scope.go:122] "RemoveContainer" containerID="3d5f60e466b766deeb8390068f3ec4c2a81e27ede581672ab3c4a6a60a2b0774" Oct 30 20:12:25.371583 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:25.371550 5091 scope.go:122] "RemoveContainer" containerID="8e58312d8d89251ef24821fedc5273fbae2bc1c830ade9e479823318e05e054e" Oct 30 20:12:25.382976 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:25.382953 5091 scope.go:122] "RemoveContainer" containerID="d32608cb95951f9df5895e80a0aee54dfb7c209619e81dae4c80c27701d78af1" Oct 30 20:12:25.942170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:25.936905 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:26.862167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:26.856109 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:27.860705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:27.860663 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:28.616069 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:28.616028 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:28.795816 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:28.795777 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:28.983222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:28.978696 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:29.180167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:29.179742 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:29.413009 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:29.412953 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:29.738730 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:29.738642 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:30.003712 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:30.003673 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:30.181601 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:30.181554 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:30.707604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:30.707567 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:30.911658 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:30.911259 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:31.234207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:31.210844 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:31.596217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:31.596052 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:31.807353 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:31.807311 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:32.539190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:32.539125 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:33.616032 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:33.615975 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:34.625164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:34.624487 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:35.894237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:35.893770 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:36.935166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:36.923525 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:37.672453 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:37.672409 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:38.542237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:38.540007 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:39.399364 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:39.399332 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:40.260081 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:40.260044 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:41.068096 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:41.068055 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:41.941040 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:41.941001 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:42.813926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:42.813880 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:43.711025 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:43.710926 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:44.537846 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:44.537806 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:45.370714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:45.370675 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:45.984008 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:12:45.983960 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:12:46.386809 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:46.386760 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:46.686777 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:46.686681 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:47.350641 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:47.350598 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:48.320169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:48.318823 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:49.158167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:49.158016 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:50.029230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:50.026888 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:50.890380 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:50.890340 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:51.831161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:51.831105 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:52.697183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:52.693766 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:53.556298 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:53.556261 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:54.672121 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:54.672081 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:55.556679 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:55.555244 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:56.416895 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:56.416850 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:57.315132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:57.315081 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:58.306440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:58.306383 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:12:59.358643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:12:59.358606 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:00.139166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:00.138768 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:01.010174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:00.998247 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:01.888373 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:01.888328 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:02.906246 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:02.906062 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:03.795188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:03.792454 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:04.699177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:04.699127 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:05.894218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:05.894109 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:06.822178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:06.822063 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:07.854547 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:07.854471 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:08.887769 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:08.887728 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:09.791100 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:09.791057 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:10.068899 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:10.068799 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:10.225855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:10.225812 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:11.236180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:11.236121 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:12.227372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:12.227330 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:13.080365 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:13.080322 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:14.017823 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:14.017789 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:14.575694 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:14.575640 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:15.109213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:15.109103 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:15.891328 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:15.891275 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:16.008714 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:13:16.008675 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:13:16.822408 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:16.822371 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:17.929198 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:17.924826 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:18.932171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:18.930228 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:19.030820 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:19.030785 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2"] Oct 30 20:13:19.035175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:19.035138 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a1a1863-tzhc2"] Oct 30 20:13:19.665799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:19.665754 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:20.443962 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:20.443932 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29893af6-a166-4973-b38d-55ab6177a749" path="/var/lib/kubelet/pods/29893af6-a166-4973-b38d-55ab6177a749/volumes" Oct 30 20:13:20.729859 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:20.728682 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:21.626428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:21.626298 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:22.589465 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:22.589424 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6"] Oct 30 20:13:22.659335 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:22.659281 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:23.544041 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:23.542930 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:24.523177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:24.522229 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:24.649826 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:24.649788 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf"] Oct 30 20:13:25.483392 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:25.483359 5091 scope.go:122] "RemoveContainer" containerID="08ebc5f6f64877e818f1c034d72f4897838460f3b64725df95d96cbcea7853d3" Oct 30 20:13:25.596177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:25.593085 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:25.639689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:25.636575 5091 scope.go:122] "RemoveContainer" containerID="4194747c3e019e86a87163b95c772085ddcf60063ab4de329f6f3f775b1b7f96" Oct 30 20:13:25.780446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:25.780414 5091 scope.go:122] "RemoveContainer" containerID="4bcaacd129c95f8d0807d8760238ff7f12f8dda32595631b3662b94e7354160c" Oct 30 20:13:25.894891 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:25.894769 5091 scope.go:122] "RemoveContainer" containerID="6e63b4367765a78312b10a1593e9e9b85154afbbcf75160a5fde7406a8302002" Oct 30 20:13:25.935456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:25.935429 5091 scope.go:122] "RemoveContainer" containerID="f068d12cc015ffc759062dc68e539004f98e04cabd12fd143aeb880ff7045ee4" Oct 30 20:13:25.952709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:25.952686 5091 scope.go:122] "RemoveContainer" containerID="59011d31bdbeb2af7a8742305171428df448ece644feb3832c2b1b30b77302a8" Oct 30 20:13:26.043748 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:26.043713 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5"] Oct 30 20:13:26.048254 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:26.048217 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr"] Oct 30 20:13:26.052294 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:26.052265 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-299e8d76-92nlr"] Oct 30 20:13:26.056054 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:26.056027 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f8abbcd9-zklq5"] Oct 30 20:13:26.507694 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:26.505776 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:26.517468 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:26.517216 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67" path="/var/lib/kubelet/pods/138e64ef-eb2f-4a4a-a074-0dc9e3aa8e67/volumes" Oct 30 20:13:26.518107 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:26.518079 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be037c9f-4faf-42a4-b160-1b92dc24dcd0" path="/var/lib/kubelet/pods/be037c9f-4faf-42a4-b160-1b92dc24dcd0/volumes" Oct 30 20:13:27.330911 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:27.329750 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:27.590518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:27.590429 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" podUID="c0675912-3f28-4aa1-99bf-c1903edee0dc" containerName="payload-processing" containerID="cri-o://dc9f38aebc0d88383614a040792d829b71bd6793e4e52d04701cc179116c5861" gracePeriod=25 Oct 30 20:13:27.909941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:27.909915 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:13:27.951963 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:27.951929 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0675912-3f28-4aa1-99bf-c1903edee0dc-kube-api-access-bvc5d\" (UniqueName: \"kubernetes.io/projected/c0675912-3f28-4aa1-99bf-c1903edee0dc-kube-api-access-bvc5d\") pod \"c0675912-3f28-4aa1-99bf-c1903edee0dc\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " Oct 30 20:13:27.952121 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:27.952104 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0675912-3f28-4aa1-99bf-c1903edee0dc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0675912-3f28-4aa1-99bf-c1903edee0dc-routing-overlay-volume\") pod \"c0675912-3f28-4aa1-99bf-c1903edee0dc\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " Oct 30 20:13:27.952377 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:27.952358 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0675912-3f28-4aa1-99bf-c1903edee0dc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0675912-3f28-4aa1-99bf-c1903edee0dc-provider-credentials\") pod \"c0675912-3f28-4aa1-99bf-c1903edee0dc\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " Oct 30 20:13:27.952469 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:27.952448 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0675912-3f28-4aa1-99bf-c1903edee0dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0675912-3f28-4aa1-99bf-c1903edee0dc-plugins-config-volume\") pod \"c0675912-3f28-4aa1-99bf-c1903edee0dc\" (UID: \"c0675912-3f28-4aa1-99bf-c1903edee0dc\") " Oct 30 20:13:27.952534 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:27.952502 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0675912-3f28-4aa1-99bf-c1903edee0dc-routing-overlay-volume" pod "c0675912-3f28-4aa1-99bf-c1903edee0dc" (UID: "c0675912-3f28-4aa1-99bf-c1903edee0dc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:27.952924 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:27.952893 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0675912-3f28-4aa1-99bf-c1903edee0dc-plugins-config-volume" pod "c0675912-3f28-4aa1-99bf-c1903edee0dc" (UID: "c0675912-3f28-4aa1-99bf-c1903edee0dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:27.953368 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:27.953343 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c0675912-3f28-4aa1-99bf-c1903edee0dc-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:27.953530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:27.953373 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0675912-3f28-4aa1-99bf-c1903edee0dc-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:27.955123 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:27.955096 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0675912-3f28-4aa1-99bf-c1903edee0dc-kube-api-access-bvc5d" pod "c0675912-3f28-4aa1-99bf-c1903edee0dc" (UID: "c0675912-3f28-4aa1-99bf-c1903edee0dc"). InnerVolumeSpecName "kube-api-access-bvc5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:27.955662 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:27.955636 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0675912-3f28-4aa1-99bf-c1903edee0dc-provider-credentials" pod "c0675912-3f28-4aa1-99bf-c1903edee0dc" (UID: "c0675912-3f28-4aa1-99bf-c1903edee0dc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:28.055109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:28.055070 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c0675912-3f28-4aa1-99bf-c1903edee0dc-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.055109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:28.055110 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvc5d\" (UniqueName: \"kubernetes.io/projected/c0675912-3f28-4aa1-99bf-c1903edee0dc-kube-api-access-bvc5d\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:28.308190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:28.305527 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:28.584209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:28.584076 5091 generic.go:356] "Generic (PLEG): container finished" podID="c0675912-3f28-4aa1-99bf-c1903edee0dc" containerID="dc9f38aebc0d88383614a040792d829b71bd6793e4e52d04701cc179116c5861" exitCode=0 Oct 30 20:13:28.584689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:28.584220 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" event={"ID":"c0675912-3f28-4aa1-99bf-c1903edee0dc","Type":"ContainerDied","Data":"dc9f38aebc0d88383614a040792d829b71bd6793e4e52d04701cc179116c5861"} Oct 30 20:13:28.584689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:28.584333 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" event={"ID":"c0675912-3f28-4aa1-99bf-c1903edee0dc","Type":"ContainerDied","Data":"59cccdda8fb1fe4b4671c5e00356973600ae2c71fecda0ebe0a39955140618cb"} Oct 30 20:13:28.584689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:28.584374 5091 scope.go:122] "RemoveContainer" containerID="dc9f38aebc0d88383614a040792d829b71bd6793e4e52d04701cc179116c5861" Oct 30 20:13:28.584689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:28.584243 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6" Oct 30 20:13:28.598370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:28.598107 5091 scope.go:122] "RemoveContainer" containerID="dc9f38aebc0d88383614a040792d829b71bd6793e4e52d04701cc179116c5861" Oct 30 20:13:28.598441 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:13:28.598421 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc9f38aebc0d88383614a040792d829b71bd6793e4e52d04701cc179116c5861\": container with ID starting with dc9f38aebc0d88383614a040792d829b71bd6793e4e52d04701cc179116c5861 not found: ID does not exist" containerID="dc9f38aebc0d88383614a040792d829b71bd6793e4e52d04701cc179116c5861" Oct 30 20:13:28.598487 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:28.598444 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc9f38aebc0d88383614a040792d829b71bd6793e4e52d04701cc179116c5861"} err="failed to get container status \"dc9f38aebc0d88383614a040792d829b71bd6793e4e52d04701cc179116c5861\": rpc error: code = NotFound desc = could not find container \"dc9f38aebc0d88383614a040792d829b71bd6793e4e52d04701cc179116c5861\": container with ID starting with dc9f38aebc0d88383614a040792d829b71bd6793e4e52d04701cc179116c5861 not found: ID does not exist" Oct 30 20:13:28.611500 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:28.611476 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6"] Oct 30 20:13:28.618635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:28.618613 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b699fq6"] Oct 30 20:13:29.203222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:29.201900 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:29.651394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:29.651323 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" podUID="b066923b-fa1a-4e9d-a960-18cd1854dad3" containerName="payload-processing" containerID="cri-o://a193703e7ef70b22c0826d8b28e900113b0169423f28216d914b42be76761909" gracePeriod=25 Oct 30 20:13:29.990036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:29.989381 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:13:30.044493 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.044453 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr"] Oct 30 20:13:30.051695 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.051659 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14a6ab4-vddjr"] Oct 30 20:13:30.094416 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.094366 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b066923b-fa1a-4e9d-a960-18cd1854dad3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b066923b-fa1a-4e9d-a960-18cd1854dad3-routing-overlay-volume\") pod \"b066923b-fa1a-4e9d-a960-18cd1854dad3\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " Oct 30 20:13:30.094618 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.094505 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b066923b-fa1a-4e9d-a960-18cd1854dad3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b066923b-fa1a-4e9d-a960-18cd1854dad3-plugins-config-volume\") pod \"b066923b-fa1a-4e9d-a960-18cd1854dad3\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " Oct 30 20:13:30.094764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.094746 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b066923b-fa1a-4e9d-a960-18cd1854dad3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b066923b-fa1a-4e9d-a960-18cd1854dad3-provider-credentials\") pod \"b066923b-fa1a-4e9d-a960-18cd1854dad3\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " Oct 30 20:13:30.094835 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.094783 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b066923b-fa1a-4e9d-a960-18cd1854dad3-kube-api-access-vrxx8\" (UniqueName: \"kubernetes.io/projected/b066923b-fa1a-4e9d-a960-18cd1854dad3-kube-api-access-vrxx8\") pod \"b066923b-fa1a-4e9d-a960-18cd1854dad3\" (UID: \"b066923b-fa1a-4e9d-a960-18cd1854dad3\") " Oct 30 20:13:30.094835 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.094813 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b066923b-fa1a-4e9d-a960-18cd1854dad3-routing-overlay-volume" pod "b066923b-fa1a-4e9d-a960-18cd1854dad3" (UID: "b066923b-fa1a-4e9d-a960-18cd1854dad3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:30.094960 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.094857 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b066923b-fa1a-4e9d-a960-18cd1854dad3-plugins-config-volume" pod "b066923b-fa1a-4e9d-a960-18cd1854dad3" (UID: "b066923b-fa1a-4e9d-a960-18cd1854dad3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:30.095709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.095654 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b066923b-fa1a-4e9d-a960-18cd1854dad3-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:30.095857 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.095713 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b066923b-fa1a-4e9d-a960-18cd1854dad3-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:30.097887 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.097865 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b066923b-fa1a-4e9d-a960-18cd1854dad3-provider-credentials" pod "b066923b-fa1a-4e9d-a960-18cd1854dad3" (UID: "b066923b-fa1a-4e9d-a960-18cd1854dad3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:30.098005 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.097896 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b066923b-fa1a-4e9d-a960-18cd1854dad3-kube-api-access-vrxx8" pod "b066923b-fa1a-4e9d-a960-18cd1854dad3" (UID: "b066923b-fa1a-4e9d-a960-18cd1854dad3"). InnerVolumeSpecName "kube-api-access-vrxx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:30.206228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.205776 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b066923b-fa1a-4e9d-a960-18cd1854dad3-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:30.206228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.205816 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrxx8\" (UniqueName: \"kubernetes.io/projected/b066923b-fa1a-4e9d-a960-18cd1854dad3-kube-api-access-vrxx8\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:30.247170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.242972 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:30.444932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.444894 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="659451eb-0ad3-439c-8a9d-02fa6a9431d9" path="/var/lib/kubelet/pods/659451eb-0ad3-439c-8a9d-02fa6a9431d9/volumes" Oct 30 20:13:30.445520 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.445487 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0675912-3f28-4aa1-99bf-c1903edee0dc" path="/var/lib/kubelet/pods/c0675912-3f28-4aa1-99bf-c1903edee0dc/volumes" Oct 30 20:13:30.604584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.604549 5091 generic.go:356] "Generic (PLEG): container finished" podID="b066923b-fa1a-4e9d-a960-18cd1854dad3" containerID="a193703e7ef70b22c0826d8b28e900113b0169423f28216d914b42be76761909" exitCode=0 Oct 30 20:13:30.604784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.604594 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" Oct 30 20:13:30.604784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.604640 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" event={"ID":"b066923b-fa1a-4e9d-a960-18cd1854dad3","Type":"ContainerDied","Data":"a193703e7ef70b22c0826d8b28e900113b0169423f28216d914b42be76761909"} Oct 30 20:13:30.604784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.604700 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf" event={"ID":"b066923b-fa1a-4e9d-a960-18cd1854dad3","Type":"ContainerDied","Data":"ebdbcfbeb87dfaf0be945c2bc235b1cf2926ba0453cf1235a08c0c982f7f6d14"} Oct 30 20:13:30.604784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.604724 5091 scope.go:122] "RemoveContainer" containerID="a193703e7ef70b22c0826d8b28e900113b0169423f28216d914b42be76761909" Oct 30 20:13:30.616801 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.616773 5091 scope.go:122] "RemoveContainer" containerID="a193703e7ef70b22c0826d8b28e900113b0169423f28216d914b42be76761909" Oct 30 20:13:30.617201 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:13:30.617135 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a193703e7ef70b22c0826d8b28e900113b0169423f28216d914b42be76761909\": container with ID starting with a193703e7ef70b22c0826d8b28e900113b0169423f28216d914b42be76761909 not found: ID does not exist" containerID="a193703e7ef70b22c0826d8b28e900113b0169423f28216d914b42be76761909" Oct 30 20:13:30.617294 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.617212 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a193703e7ef70b22c0826d8b28e900113b0169423f28216d914b42be76761909"} err="failed to get container status \"a193703e7ef70b22c0826d8b28e900113b0169423f28216d914b42be76761909\": rpc error: code = NotFound desc = could not find container \"a193703e7ef70b22c0826d8b28e900113b0169423f28216d914b42be76761909\": container with ID starting with a193703e7ef70b22c0826d8b28e900113b0169423f28216d914b42be76761909 not found: ID does not exist" Oct 30 20:13:30.631299 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.631271 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf"] Oct 30 20:13:30.640190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:30.640158 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f9468qf"] Oct 30 20:13:31.122014 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:31.121886 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:31.992176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:31.986038 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:32.442734 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:32.442699 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b066923b-fa1a-4e9d-a960-18cd1854dad3" path="/var/lib/kubelet/pods/b066923b-fa1a-4e9d-a960-18cd1854dad3/volumes" Oct 30 20:13:32.886194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:32.885600 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:33.885802 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:33.885744 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:34.788345 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:34.788309 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:35.686180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:35.686063 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:36.528170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:36.519226 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:37.055086 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:37.055035 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp"] Oct 30 20:13:37.061848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:37.061812 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f983d766-dmvrp"] Oct 30 20:13:37.423301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:37.423201 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:38.348232 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:38.348192 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:38.443679 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:38.443645 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53568d0f-819b-4557-905f-72bdcc424d97" path="/var/lib/kubelet/pods/53568d0f-819b-4557-905f-72bdcc424d97/volumes" Oct 30 20:13:39.426181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:39.421449 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:40.396176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:40.392934 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:41.252179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:41.251676 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:42.119540 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:42.114782 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:42.988230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:42.987821 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:43.886358 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:43.886317 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:44.899436 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:44.899396 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:45.893198 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:45.893133 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:45.989816 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:13:45.989775 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:13:46.788877 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:46.788831 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:47.797993 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:47.797955 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:48.677904 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:48.677852 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:49.588174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:49.586081 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:50.497420 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:50.497379 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:51.285900 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:51.285856 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:52.058228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:52.058177 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:52.926842 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:52.926799 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:53.755854 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:53.755783 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:53.996563 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:53.996530 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn"] Oct 30 20:13:53.997074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:53.996797 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" podUID="1888b3fa-2409-4051-99ff-22d37072bd4f" containerName="istio-proxy" containerID="cri-o://6214aa32fc9e307705e938777fbd2c2575e52d296e1b1c41b91c5993c8df8c87" gracePeriod=30 Oct 30 20:13:54.723344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:54.720916 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:55.485726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:55.485691 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:56.300981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:56.300910 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:57.194361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:57.194303 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:58.092200 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:58.090945 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:58.947983 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:58.947943 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:59.315340 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.315310 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:13:59.480178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.480119 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1888b3fa-2409-4051-99ff-22d37072bd4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1888b3fa-2409-4051-99ff-22d37072bd4f-istiod-ca-cert\") pod \"1888b3fa-2409-4051-99ff-22d37072bd4f\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " Oct 30 20:13:59.480372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.480271 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-workload-socket\") pod \"1888b3fa-2409-4051-99ff-22d37072bd4f\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " Oct 30 20:13:59.480372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.480335 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-data\") pod \"1888b3fa-2409-4051-99ff-22d37072bd4f\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " Oct 30 20:13:59.480523 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.480431 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-token\" (UniqueName: \"kubernetes.io/projected/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-token\") pod \"1888b3fa-2409-4051-99ff-22d37072bd4f\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " Oct 30 20:13:59.480523 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.480469 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-podinfo\") pod \"1888b3fa-2409-4051-99ff-22d37072bd4f\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " Oct 30 20:13:59.480647 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.480548 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1888b3fa-2409-4051-99ff-22d37072bd4f-kube-api-access-fcxdm\" (UniqueName: \"kubernetes.io/projected/1888b3fa-2409-4051-99ff-22d37072bd4f-kube-api-access-fcxdm\") pod \"1888b3fa-2409-4051-99ff-22d37072bd4f\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " Oct 30 20:13:59.480647 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.480593 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-workload-socket" pod "1888b3fa-2409-4051-99ff-22d37072bd4f" (UID: "1888b3fa-2409-4051-99ff-22d37072bd4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:59.480647 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.480515 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1888b3fa-2409-4051-99ff-22d37072bd4f-istiod-ca-cert" pod "1888b3fa-2409-4051-99ff-22d37072bd4f" (UID: "1888b3fa-2409-4051-99ff-22d37072bd4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:13:59.480647 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.480642 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-envoy\") pod \"1888b3fa-2409-4051-99ff-22d37072bd4f\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " Oct 30 20:13:59.480886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.480684 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-credential-socket\") pod \"1888b3fa-2409-4051-99ff-22d37072bd4f\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " Oct 30 20:13:59.480886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.480866 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-workload-certs\") pod \"1888b3fa-2409-4051-99ff-22d37072bd4f\" (UID: \"1888b3fa-2409-4051-99ff-22d37072bd4f\") " Oct 30 20:13:59.481188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.481128 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-credential-socket" pod "1888b3fa-2409-4051-99ff-22d37072bd4f" (UID: "1888b3fa-2409-4051-99ff-22d37072bd4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:59.481404 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.481380 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-data" pod "1888b3fa-2409-4051-99ff-22d37072bd4f" (UID: "1888b3fa-2409-4051-99ff-22d37072bd4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:59.481492 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.481402 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-workload-certs" pod "1888b3fa-2409-4051-99ff-22d37072bd4f" (UID: "1888b3fa-2409-4051-99ff-22d37072bd4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:59.481849 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.481827 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:59.481905 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.481855 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:59.481905 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.481870 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1888b3fa-2409-4051-99ff-22d37072bd4f-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:59.481905 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.481883 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:59.481905 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.481898 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:59.483197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.483137 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-token" pod "1888b3fa-2409-4051-99ff-22d37072bd4f" (UID: "1888b3fa-2409-4051-99ff-22d37072bd4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:59.484423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.484397 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1888b3fa-2409-4051-99ff-22d37072bd4f-kube-api-access-fcxdm" pod "1888b3fa-2409-4051-99ff-22d37072bd4f" (UID: "1888b3fa-2409-4051-99ff-22d37072bd4f"). InnerVolumeSpecName "kube-api-access-fcxdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:13:59.484533 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.484400 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-envoy" pod "1888b3fa-2409-4051-99ff-22d37072bd4f" (UID: "1888b3fa-2409-4051-99ff-22d37072bd4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:13:59.484667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.484641 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-podinfo" pod "1888b3fa-2409-4051-99ff-22d37072bd4f" (UID: "1888b3fa-2409-4051-99ff-22d37072bd4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:13:59.583400 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.583302 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:59.583400 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.583335 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:59.583400 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.583348 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcxdm\" (UniqueName: \"kubernetes.io/projected/1888b3fa-2409-4051-99ff-22d37072bd4f-kube-api-access-fcxdm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:59.583400 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.583361 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1888b3fa-2409-4051-99ff-22d37072bd4f-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:13:59.820166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.816782 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:13:59.922236 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.922113 5091 generic.go:356] "Generic (PLEG): container finished" podID="1888b3fa-2409-4051-99ff-22d37072bd4f" containerID="6214aa32fc9e307705e938777fbd2c2575e52d296e1b1c41b91c5993c8df8c87" exitCode=0 Oct 30 20:13:59.922236 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.922197 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" Oct 30 20:13:59.922459 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.922198 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" event={"ID":"1888b3fa-2409-4051-99ff-22d37072bd4f","Type":"ContainerDied","Data":"6214aa32fc9e307705e938777fbd2c2575e52d296e1b1c41b91c5993c8df8c87"} Oct 30 20:13:59.922459 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.922318 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn" event={"ID":"1888b3fa-2409-4051-99ff-22d37072bd4f","Type":"ContainerDied","Data":"72bf252784400930fba67854bd575eabbef340e7eb69007f496dbf8c02f860d7"} Oct 30 20:13:59.922459 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.922341 5091 scope.go:122] "RemoveContainer" containerID="6214aa32fc9e307705e938777fbd2c2575e52d296e1b1c41b91c5993c8df8c87" Oct 30 20:13:59.935130 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.935107 5091 scope.go:122] "RemoveContainer" containerID="6a89dcf2a18e9415269f153f1b7a5ed5f61c7ce6c2408cbbd43281481255cfd6" Oct 30 20:13:59.945590 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.945566 5091 scope.go:122] "RemoveContainer" containerID="6214aa32fc9e307705e938777fbd2c2575e52d296e1b1c41b91c5993c8df8c87" Oct 30 20:13:59.945862 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:13:59.945845 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6214aa32fc9e307705e938777fbd2c2575e52d296e1b1c41b91c5993c8df8c87\": container with ID starting with 6214aa32fc9e307705e938777fbd2c2575e52d296e1b1c41b91c5993c8df8c87 not found: ID does not exist" containerID="6214aa32fc9e307705e938777fbd2c2575e52d296e1b1c41b91c5993c8df8c87" Oct 30 20:13:59.945929 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.945869 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6214aa32fc9e307705e938777fbd2c2575e52d296e1b1c41b91c5993c8df8c87"} err="failed to get container status \"6214aa32fc9e307705e938777fbd2c2575e52d296e1b1c41b91c5993c8df8c87\": rpc error: code = NotFound desc = could not find container \"6214aa32fc9e307705e938777fbd2c2575e52d296e1b1c41b91c5993c8df8c87\": container with ID starting with 6214aa32fc9e307705e938777fbd2c2575e52d296e1b1c41b91c5993c8df8c87 not found: ID does not exist" Oct 30 20:13:59.945929 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.945888 5091 scope.go:122] "RemoveContainer" containerID="6a89dcf2a18e9415269f153f1b7a5ed5f61c7ce6c2408cbbd43281481255cfd6" Oct 30 20:13:59.946125 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:13:59.946106 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a89dcf2a18e9415269f153f1b7a5ed5f61c7ce6c2408cbbd43281481255cfd6\": container with ID starting with 6a89dcf2a18e9415269f153f1b7a5ed5f61c7ce6c2408cbbd43281481255cfd6 not found: ID does not exist" containerID="6a89dcf2a18e9415269f153f1b7a5ed5f61c7ce6c2408cbbd43281481255cfd6" Oct 30 20:13:59.947009 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.946135 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a89dcf2a18e9415269f153f1b7a5ed5f61c7ce6c2408cbbd43281481255cfd6"} err="failed to get container status \"6a89dcf2a18e9415269f153f1b7a5ed5f61c7ce6c2408cbbd43281481255cfd6\": rpc error: code = NotFound desc = could not find container \"6a89dcf2a18e9415269f153f1b7a5ed5f61c7ce6c2408cbbd43281481255cfd6\": container with ID starting with 6a89dcf2a18e9415269f153f1b7a5ed5f61c7ce6c2408cbbd43281481255cfd6 not found: ID does not exist" Oct 30 20:13:59.955158 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.955125 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn"] Oct 30 20:13:59.961892 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:13:59.961867 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-2a2286-openshift-default-6d8868d969-h27mn"] Oct 30 20:14:00.443207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:00.443127 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1888b3fa-2409-4051-99ff-22d37072bd4f" path="/var/lib/kubelet/pods/1888b3fa-2409-4051-99ff-22d37072bd4f/volumes" Oct 30 20:14:00.700221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:00.694300 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:01.035787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:01.035752 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j"] Oct 30 20:14:01.039861 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:01.039837 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80fb8b78-6f58j"] Oct 30 20:14:01.509920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:01.505073 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:02.034278 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:02.034239 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz"] Oct 30 20:14:02.041105 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:02.041077 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1015ba60-zdvbz"] Oct 30 20:14:02.471132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:02.471048 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e168a95-a7a1-469f-8222-d8437ed038ed" path="/var/lib/kubelet/pods/6e168a95-a7a1-469f-8222-d8437ed038ed/volumes" Oct 30 20:14:02.472368 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:02.472127 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acc72fb2-ae86-4dbd-989a-44f79e345dfb" path="/var/lib/kubelet/pods/acc72fb2-ae86-4dbd-989a-44f79e345dfb/volumes" Oct 30 20:14:02.480380 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:02.479691 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:03.411641 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:03.411603 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:03.684361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:03.684280 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc"] Oct 30 20:14:03.684523 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:03.684497 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" podUID="7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" containerName="istio-proxy" containerID="cri-o://3628ea82aea5ca2b1f72f994b74b3cb30e8f5397dace04bc7a3080a56dd28af4" gracePeriod=30 Oct 30 20:14:04.268765 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:04.268717 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:05.135789 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:05.135706 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:05.459604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:05.459507 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:05.607443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:05.607412 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:06.609569 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:06.609527 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:07.154641 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:07.147057 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:07.342261 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:07.342225 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:07.589169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:07.587851 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:08.618494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:08.605194 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:08.818520 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:08.818487 5091 patch_prober.go:29] interesting pod/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Oct 30 20:14:08.818673 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:08.818563 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" podUID="7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Oct 30 20:14:08.973080 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:08.973048 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 20:14:09.057271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.057232 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-credential-socket\") pod \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " Oct 30 20:14:09.057473 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.057286 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-workload-certs\") pod \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " Oct 30 20:14:09.057552 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.057499 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-data\") pod \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " Oct 30 20:14:09.057552 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.057530 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-credential-socket" pod "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" (UID: "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:09.057667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.057584 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-kube-api-access-ghgt7\" (UniqueName: \"kubernetes.io/projected/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-kube-api-access-ghgt7\") pod \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " Oct 30 20:14:09.057763 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.057647 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-workload-certs" pod "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" (UID: "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:09.057763 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.057658 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-envoy\") pod \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " Oct 30 20:14:09.057763 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.057716 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-data" pod "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" (UID: "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:09.057932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.057806 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-podinfo\") pod \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " Oct 30 20:14:09.057932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.057849 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istiod-ca-cert\") pod \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " Oct 30 20:14:09.057932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.057894 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-token\" (UniqueName: \"kubernetes.io/projected/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-token\") pod \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " Oct 30 20:14:09.058090 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.058045 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-workload-socket\") pod \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\" (UID: \"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e\") " Oct 30 20:14:09.058760 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.058621 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-workload-socket" pod "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" (UID: "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:09.058760 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.058549 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istiod-ca-cert" pod "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" (UID: "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:09.058988 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.058935 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:09.058988 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.058958 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:09.058988 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.058975 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:09.059213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.058990 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:09.059213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.059003 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:09.060326 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.060292 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-envoy" pod "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" (UID: "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:09.060326 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.060313 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-podinfo" pod "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" (UID: "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:09.060793 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.060773 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-token" pod "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" (UID: "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:09.060865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.060835 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-kube-api-access-ghgt7" pod "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" (UID: "7dcda2f6-599d-4eea-b8d7-766b9a95cc2e"). InnerVolumeSpecName "kube-api-access-ghgt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:09.160336 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.160302 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghgt7\" (UniqueName: \"kubernetes.io/projected/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-kube-api-access-ghgt7\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:09.160336 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.160332 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:09.160336 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.160342 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:09.160578 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.160350 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:09.222785 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.222698 5091 generic.go:356] "Generic (PLEG): container finished" podID="7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" containerID="3628ea82aea5ca2b1f72f994b74b3cb30e8f5397dace04bc7a3080a56dd28af4" exitCode=0 Oct 30 20:14:09.222785 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.222747 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" Oct 30 20:14:09.222984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.222782 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" event={"ID":"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e","Type":"ContainerDied","Data":"3628ea82aea5ca2b1f72f994b74b3cb30e8f5397dace04bc7a3080a56dd28af4"} Oct 30 20:14:09.222984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.222848 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc" event={"ID":"7dcda2f6-599d-4eea-b8d7-766b9a95cc2e","Type":"ContainerDied","Data":"cf856da271bc49512ee73cfac24ec3234c778f0514260900d309c86bba608ebc"} Oct 30 20:14:09.222984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.222879 5091 scope.go:122] "RemoveContainer" containerID="3628ea82aea5ca2b1f72f994b74b3cb30e8f5397dace04bc7a3080a56dd28af4" Oct 30 20:14:09.237556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.237538 5091 scope.go:122] "RemoveContainer" containerID="3628ea82aea5ca2b1f72f994b74b3cb30e8f5397dace04bc7a3080a56dd28af4" Oct 30 20:14:09.237869 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:14:09.237844 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3628ea82aea5ca2b1f72f994b74b3cb30e8f5397dace04bc7a3080a56dd28af4\": container with ID starting with 3628ea82aea5ca2b1f72f994b74b3cb30e8f5397dace04bc7a3080a56dd28af4 not found: ID does not exist" containerID="3628ea82aea5ca2b1f72f994b74b3cb30e8f5397dace04bc7a3080a56dd28af4" Oct 30 20:14:09.237935 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.237878 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3628ea82aea5ca2b1f72f994b74b3cb30e8f5397dace04bc7a3080a56dd28af4"} err="failed to get container status \"3628ea82aea5ca2b1f72f994b74b3cb30e8f5397dace04bc7a3080a56dd28af4\": rpc error: code = NotFound desc = could not find container \"3628ea82aea5ca2b1f72f994b74b3cb30e8f5397dace04bc7a3080a56dd28af4\": container with ID starting with 3628ea82aea5ca2b1f72f994b74b3cb30e8f5397dace04bc7a3080a56dd28af4 not found: ID does not exist" Oct 30 20:14:09.253242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.253218 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc"] Oct 30 20:14:09.257784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.257764 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-7a745c-openshift-default-57b8b749-nlfbc"] Oct 30 20:14:09.643329 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:09.643289 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:10.443255 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:10.443224 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" path="/var/lib/kubelet/pods/7dcda2f6-599d-4eea-b8d7-766b9a95cc2e/volumes" Oct 30 20:14:10.526170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:10.524935 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:11.313173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:11.306797 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:12.281171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:12.277712 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:13.042341 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:13.042304 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:13.984169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:13.983716 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:14.645461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:14.645430 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:15.005169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:15.003136 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:15.844007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:15.841246 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:15.979427 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:14:15.979385 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:14:16.698168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:16.698114 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:17.241062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:17.241021 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g"] Oct 30 20:14:17.241487 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:17.241292 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" podUID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" containerName="istio-proxy" containerID="cri-o://6b06c26c1c600f57eb3fc7b0360f3b7795a1c2fd23f0a99dba456635cbe0bfb2" gracePeriod=30 Oct 30 20:14:17.994528 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:17.994479 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:18.620657 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:18.620620 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:18.944220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:18.942733 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:19.592853 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.592815 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.616473 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2"] Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.617979 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b066923b-fa1a-4e9d-a960-18cd1854dad3" containerName="payload-processing" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618008 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b066923b-fa1a-4e9d-a960-18cd1854dad3" containerName="payload-processing" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618039 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c0675912-3f28-4aa1-99bf-c1903edee0dc" containerName="payload-processing" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618048 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0675912-3f28-4aa1-99bf-c1903edee0dc" containerName="payload-processing" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618063 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a49a5707-dc8c-4371-835f-8c9f99a4cc94" containerName="revoke-keys" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618071 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49a5707-dc8c-4371-835f-8c9f99a4cc94" containerName="revoke-keys" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618091 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1888b3fa-2409-4051-99ff-22d37072bd4f" containerName="istio-proxy" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618098 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1888b3fa-2409-4051-99ff-22d37072bd4f" containerName="istio-proxy" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618112 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" containerName="istio-proxy" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618120 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" containerName="istio-proxy" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618165 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" containerName="istio-proxy" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618175 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" containerName="istio-proxy" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618211 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1888b3fa-2409-4051-99ff-22d37072bd4f" containerName="istio-proxy" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618218 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1888b3fa-2409-4051-99ff-22d37072bd4f" containerName="istio-proxy" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618424 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1888b3fa-2409-4051-99ff-22d37072bd4f" containerName="istio-proxy" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618447 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b066923b-fa1a-4e9d-a960-18cd1854dad3" containerName="payload-processing" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618460 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0675912-3f28-4aa1-99bf-c1903edee0dc" containerName="payload-processing" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618471 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49a5707-dc8c-4371-835f-8c9f99a4cc94" containerName="revoke-keys" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618481 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef8573b-3f96-4e5d-b9ff-7b0e3924b0a6" containerName="istio-proxy" Oct 30 20:14:19.622101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.618492 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dcda2f6-599d-4eea-b8d7-766b9a95cc2e" containerName="istio-proxy" Oct 30 20:14:19.720170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.712215 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:19.742165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.730136 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nj5xk\" (UniqueName: \"kubernetes.io/projected/04dd38d4-f223-4b01-8a58-ac3ef092a862-kube-api-access-nj5xk\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:19.742165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.730385 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04dd38d4-f223-4b01-8a58-ac3ef092a862-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:19.742165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.730561 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04dd38d4-f223-4b01-8a58-ac3ef092a862-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:19.742165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.730622 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04dd38d4-f223-4b01-8a58-ac3ef092a862-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:19.781164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.769956 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"agc-cred-4a17ee2c812f\"" Oct 30 20:14:19.781164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.770101 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"payload-processing-external-model-e2e--632181a7-dockercfg-dd2cn\"" Oct 30 20:14:19.781164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.770218 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"routing-overlay\"" Oct 30 20:14:19.781164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.770287 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"agc-cred-57fee2c1790f\"" Oct 30 20:14:19.781164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.770339 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"agc-cred-e787e8649e56\"" Oct 30 20:14:19.781164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.770430 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"agc-cred-20d4e7c9cc7c\"" Oct 30 20:14:19.781164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.770488 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"openshift-service-ca.crt\"" Oct 30 20:14:19.781164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.770543 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"payload-processing-external-model-plugins-e2e-extmod-b-4d5758\"" Oct 30 20:14:19.781164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.772267 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-4d5758\"/\"kube-root-ca.crt\"" Oct 30 20:14:19.800251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.798731 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2"] Oct 30 20:14:19.810397 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.810361 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk"] Oct 30 20:14:19.811908 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.811873 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1888b3fa-2409-4051-99ff-22d37072bd4f" containerName="istio-proxy" Oct 30 20:14:19.816824 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.816805 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:19.827471 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.827417 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"routing-overlay\"" Oct 30 20:14:19.827597 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.827521 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"kube-root-ca.crt\"" Oct 30 20:14:19.827744 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.827727 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"payload-processing-external-model-plugins-e2e-extmod-a-222480\"" Oct 30 20:14:19.827845 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.827786 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"openshift-service-ca.crt\"" Oct 30 20:14:19.828199 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.828174 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk"] Oct 30 20:14:19.831806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.831783 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04dd38d4-f223-4b01-8a58-ac3ef092a862-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:19.831918 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.831857 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04dd38d4-f223-4b01-8a58-ac3ef092a862-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:19.831988 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.831959 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nj5xk\" (UniqueName: \"kubernetes.io/projected/04dd38d4-f223-4b01-8a58-ac3ef092a862-kube-api-access-nj5xk\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:19.832207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.832185 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04dd38d4-f223-4b01-8a58-ac3ef092a862-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:19.832844 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.832824 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04dd38d4-f223-4b01-8a58-ac3ef092a862-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:19.832917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.832844 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04dd38d4-f223-4b01-8a58-ac3ef092a862-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:19.836341 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.836319 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04dd38d4-f223-4b01-8a58-ac3ef092a862-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:19.841957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.841938 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"agc-cred-ff68f10d845a\"" Oct 30 20:14:19.842119 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.842106 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"agc-cred-c8bb93f01837\"" Oct 30 20:14:19.842264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.842104 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"agc-cred-bd77348f7e09\"" Oct 30 20:14:19.842345 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.842137 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"agc-cred-7233bc96d635\"" Oct 30 20:14:19.842420 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.842263 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-222480\"/\"payload-processing-external-model-e2e--8e4491f3-dockercfg-rdtdr\"" Oct 30 20:14:19.861875 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.861805 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nj5xk\" (UniqueName: \"kubernetes.io/projected/04dd38d4-f223-4b01-8a58-ac3ef092a862-kube-api-access-nj5xk\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:19.933555 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.933521 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z7rs\" (UniqueName: \"kubernetes.io/projected/0ec79660-754a-446d-846c-9632c0b6b2e6-kube-api-access-6z7rs\") pod \"payload-processing-external-model-e2e-extmod-a-222480-9589btknk\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:19.933734 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.933626 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ec79660-754a-446d-846c-9632c0b6b2e6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-9589btknk\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:19.933734 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.933686 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec79660-754a-446d-846c-9632c0b6b2e6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-9589btknk\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:19.933826 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:19.933747 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ec79660-754a-446d-846c-9632c0b6b2e6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-9589btknk\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:20.035819 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.035774 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z7rs\" (UniqueName: \"kubernetes.io/projected/0ec79660-754a-446d-846c-9632c0b6b2e6-kube-api-access-6z7rs\") pod \"payload-processing-external-model-e2e-extmod-a-222480-9589btknk\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:20.035995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.035834 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ec79660-754a-446d-846c-9632c0b6b2e6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-9589btknk\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:20.035995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.035938 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec79660-754a-446d-846c-9632c0b6b2e6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-9589btknk\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:20.036128 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.036009 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ec79660-754a-446d-846c-9632c0b6b2e6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-9589btknk\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:20.036662 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.036632 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ec79660-754a-446d-846c-9632c0b6b2e6-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-9589btknk\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:20.036800 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.036753 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec79660-754a-446d-846c-9632c0b6b2e6-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-9589btknk\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:20.038822 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.038795 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ec79660-754a-446d-846c-9632c0b6b2e6-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-9589btknk\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:20.050223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.050194 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z7rs\" (UniqueName: \"kubernetes.io/projected/0ec79660-754a-446d-846c-9632c0b6b2e6-kube-api-access-6z7rs\") pod \"payload-processing-external-model-e2e-extmod-a-222480-9589btknk\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:20.110948 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.110914 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:20.136562 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.136350 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:20.407256 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.403114 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:20.460039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.460007 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2"] Oct 30 20:14:20.462564 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:14:20.462539 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04dd38d4_f223_4b01_8a58_ac3ef092a862.slice/crio-26e0b463b2e6578b5d88c0ab0f19dae8d810dccfa8f8a4856eeadda1c80653a5 WatchSource:0}: Error finding container 26e0b463b2e6578b5d88c0ab0f19dae8d810dccfa8f8a4856eeadda1c80653a5: Status 404 returned error can't find the container with id 26e0b463b2e6578b5d88c0ab0f19dae8d810dccfa8f8a4856eeadda1c80653a5 Oct 30 20:14:20.476352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.476322 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk"] Oct 30 20:14:20.478171 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:14:20.478131 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ec79660_754a_446d_846c_9632c0b6b2e6.slice/crio-88f840b273409095031d996cd1c1756f7478de0f0bd98bfbc70955b6581e349f WatchSource:0}: Error finding container 88f840b273409095031d996cd1c1756f7478de0f0bd98bfbc70955b6581e349f: Status 404 returned error can't find the container with id 88f840b273409095031d996cd1c1756f7478de0f0bd98bfbc70955b6581e349f Oct 30 20:14:20.668930 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.668818 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" event={"ID":"0ec79660-754a-446d-846c-9632c0b6b2e6","Type":"ContainerStarted","Data":"defaf3f23cd5a02404442104f3e76a6067f2d74dadbdb9d4bb256f6005ca8542"} Oct 30 20:14:20.668930 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.668885 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" event={"ID":"0ec79660-754a-446d-846c-9632c0b6b2e6","Type":"ContainerStarted","Data":"88f840b273409095031d996cd1c1756f7478de0f0bd98bfbc70955b6581e349f"} Oct 30 20:14:20.668930 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.668914 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:20.670834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.670806 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" event={"ID":"04dd38d4-f223-4b01-8a58-ac3ef092a862","Type":"ContainerStarted","Data":"2dc3f7c6ca81099eaa48c5e29a800a597e0562dd9518ccb0234062ee0a2d8e3f"} Oct 30 20:14:20.670981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.670854 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" event={"ID":"04dd38d4-f223-4b01-8a58-ac3ef092a862","Type":"ContainerStarted","Data":"26e0b463b2e6578b5d88c0ab0f19dae8d810dccfa8f8a4856eeadda1c80653a5"} Oct 30 20:14:20.670981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.670882 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:20.694882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.694835 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" podStartSLOduration=1.694820827 podStartE2EDuration="1.694820827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:20.692031664 +0000 UTC m=+2580.387239623" watchObservedRunningTime="2026-10-30 20:14:20.694820827 +0000 UTC m=+2580.390028805" Oct 30 20:14:20.720261 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:20.720213 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" podStartSLOduration=1.7202015149999998 podStartE2EDuration="1.720201515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:14:20.718545601 +0000 UTC m=+2580.413753559" watchObservedRunningTime="2026-10-30 20:14:20.720201515 +0000 UTC m=+2580.415409473" Oct 30 20:14:21.204073 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:21.199495 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:21.992182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:21.984601 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:22.543962 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.543937 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 20:14:22.578139 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.578103 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-token\" (UniqueName: \"kubernetes.io/projected/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-token\") pod \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " Oct 30 20:14:22.578380 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.578223 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0153a5c2-c41c-4dd5-9bc2-035455025bd1-kube-api-access-v57tr\" (UniqueName: \"kubernetes.io/projected/0153a5c2-c41c-4dd5-9bc2-035455025bd1-kube-api-access-v57tr\") pod \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " Oct 30 20:14:22.578380 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.578263 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-workload-socket\") pod \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " Oct 30 20:14:22.578380 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.578339 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-data\") pod \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " Oct 30 20:14:22.578570 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.578471 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-podinfo\") pod \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " Oct 30 20:14:22.578675 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.578622 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-envoy\") pod \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " Oct 30 20:14:22.578747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.578674 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-workload-certs\") pod \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " Oct 30 20:14:22.578747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.578733 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istiod-ca-cert\") pod \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " Oct 30 20:14:22.578855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.578798 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-credential-socket\") pod \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\" (UID: \"0153a5c2-c41c-4dd5-9bc2-035455025bd1\") " Oct 30 20:14:22.579579 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.579353 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-workload-certs" pod "0153a5c2-c41c-4dd5-9bc2-035455025bd1" (UID: "0153a5c2-c41c-4dd5-9bc2-035455025bd1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:22.579724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.579599 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-credential-socket" pod "0153a5c2-c41c-4dd5-9bc2-035455025bd1" (UID: "0153a5c2-c41c-4dd5-9bc2-035455025bd1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:22.579724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.579714 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:22.579954 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.579933 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-data" pod "0153a5c2-c41c-4dd5-9bc2-035455025bd1" (UID: "0153a5c2-c41c-4dd5-9bc2-035455025bd1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:22.580042 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.579968 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istiod-ca-cert" pod "0153a5c2-c41c-4dd5-9bc2-035455025bd1" (UID: "0153a5c2-c41c-4dd5-9bc2-035455025bd1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:22.580770 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.580742 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-workload-socket" pod "0153a5c2-c41c-4dd5-9bc2-035455025bd1" (UID: "0153a5c2-c41c-4dd5-9bc2-035455025bd1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:22.581852 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.581827 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-podinfo" pod "0153a5c2-c41c-4dd5-9bc2-035455025bd1" (UID: "0153a5c2-c41c-4dd5-9bc2-035455025bd1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:22.582190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.582155 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0153a5c2-c41c-4dd5-9bc2-035455025bd1-kube-api-access-v57tr" pod "0153a5c2-c41c-4dd5-9bc2-035455025bd1" (UID: "0153a5c2-c41c-4dd5-9bc2-035455025bd1"). InnerVolumeSpecName "kube-api-access-v57tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:22.583012 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.582987 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-token" pod "0153a5c2-c41c-4dd5-9bc2-035455025bd1" (UID: "0153a5c2-c41c-4dd5-9bc2-035455025bd1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:22.583333 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.583304 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-envoy" pod "0153a5c2-c41c-4dd5-9bc2-035455025bd1" (UID: "0153a5c2-c41c-4dd5-9bc2-035455025bd1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:22.680997 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.680962 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:22.681275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.681231 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:22.681275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.681261 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:22.681275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.681276 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:22.681531 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.681289 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:22.687127 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.681303 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0153a5c2-c41c-4dd5-9bc2-035455025bd1-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:22.687127 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.685609 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v57tr\" (UniqueName: \"kubernetes.io/projected/0153a5c2-c41c-4dd5-9bc2-035455025bd1-kube-api-access-v57tr\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:22.687127 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.685632 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0153a5c2-c41c-4dd5-9bc2-035455025bd1-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:22.702368 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.702337 5091 generic.go:356] "Generic (PLEG): container finished" podID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" containerID="6b06c26c1c600f57eb3fc7b0360f3b7795a1c2fd23f0a99dba456635cbe0bfb2" exitCode=0 Oct 30 20:14:22.702556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.702392 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" Oct 30 20:14:22.702556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.702420 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" event={"ID":"0153a5c2-c41c-4dd5-9bc2-035455025bd1","Type":"ContainerDied","Data":"6b06c26c1c600f57eb3fc7b0360f3b7795a1c2fd23f0a99dba456635cbe0bfb2"} Oct 30 20:14:22.702556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.702486 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g" event={"ID":"0153a5c2-c41c-4dd5-9bc2-035455025bd1","Type":"ContainerDied","Data":"88dc36fc2cca0d83c2605b2141b75cf42fa1cbb07967c3c11f752e504950be1d"} Oct 30 20:14:22.702556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.702516 5091 scope.go:122] "RemoveContainer" containerID="6b06c26c1c600f57eb3fc7b0360f3b7795a1c2fd23f0a99dba456635cbe0bfb2" Oct 30 20:14:22.719083 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.719060 5091 scope.go:122] "RemoveContainer" containerID="ccbd6495cf6cda2472174039f2a1ced45112c359f21843030002c9dec0ef3fe3" Oct 30 20:14:22.729885 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.729867 5091 scope.go:122] "RemoveContainer" containerID="6b06c26c1c600f57eb3fc7b0360f3b7795a1c2fd23f0a99dba456635cbe0bfb2" Oct 30 20:14:22.730221 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:14:22.730187 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b06c26c1c600f57eb3fc7b0360f3b7795a1c2fd23f0a99dba456635cbe0bfb2\": container with ID starting with 6b06c26c1c600f57eb3fc7b0360f3b7795a1c2fd23f0a99dba456635cbe0bfb2 not found: ID does not exist" containerID="6b06c26c1c600f57eb3fc7b0360f3b7795a1c2fd23f0a99dba456635cbe0bfb2" Oct 30 20:14:22.730291 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.730233 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b06c26c1c600f57eb3fc7b0360f3b7795a1c2fd23f0a99dba456635cbe0bfb2"} err="failed to get container status \"6b06c26c1c600f57eb3fc7b0360f3b7795a1c2fd23f0a99dba456635cbe0bfb2\": rpc error: code = NotFound desc = could not find container \"6b06c26c1c600f57eb3fc7b0360f3b7795a1c2fd23f0a99dba456635cbe0bfb2\": container with ID starting with 6b06c26c1c600f57eb3fc7b0360f3b7795a1c2fd23f0a99dba456635cbe0bfb2 not found: ID does not exist" Oct 30 20:14:22.730291 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.730258 5091 scope.go:122] "RemoveContainer" containerID="ccbd6495cf6cda2472174039f2a1ced45112c359f21843030002c9dec0ef3fe3" Oct 30 20:14:22.730510 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:14:22.730491 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbd6495cf6cda2472174039f2a1ced45112c359f21843030002c9dec0ef3fe3\": container with ID starting with ccbd6495cf6cda2472174039f2a1ced45112c359f21843030002c9dec0ef3fe3 not found: ID does not exist" containerID="ccbd6495cf6cda2472174039f2a1ced45112c359f21843030002c9dec0ef3fe3" Oct 30 20:14:22.730612 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.730517 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbd6495cf6cda2472174039f2a1ced45112c359f21843030002c9dec0ef3fe3"} err="failed to get container status \"ccbd6495cf6cda2472174039f2a1ced45112c359f21843030002c9dec0ef3fe3\": rpc error: code = NotFound desc = could not find container \"ccbd6495cf6cda2472174039f2a1ced45112c359f21843030002c9dec0ef3fe3\": container with ID starting with ccbd6495cf6cda2472174039f2a1ced45112c359f21843030002c9dec0ef3fe3 not found: ID does not exist" Oct 30 20:14:22.733259 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.733236 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g"] Oct 30 20:14:22.742193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.742172 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-7eaeed-openshift-default-694c55cd79-jp46g"] Oct 30 20:14:22.890226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:22.889357 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:23.977223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:23.976562 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:24.445871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:24.445832 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" path="/var/lib/kubelet/pods/0153a5c2-c41c-4dd5-9bc2-035455025bd1/volumes" Oct 30 20:14:24.921637 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:24.921470 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:25.855167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:25.853860 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:26.027007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:26.026968 5091 scope.go:122] "RemoveContainer" containerID="deef98030540979c2e1b8544522f1efd70e0aec8a8679d1e50259563e6aa8fc7" Oct 30 20:14:26.038417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:26.038393 5091 scope.go:122] "RemoveContainer" containerID="1c66aceccd78751682872ab78d7801ed10ceabd0d9ae599b6566ecf33fbcb5d3" Oct 30 20:14:26.048874 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:26.048850 5091 scope.go:122] "RemoveContainer" containerID="8d97b3d0fb05faf7e28561e8575eaf86a26e70b2ce241078f3c079d3d05c0889" Oct 30 20:14:26.878327 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:26.869815 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:27.848637 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:27.848593 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:28.581168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:28.570859 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:29.506173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:29.500857 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:30.317168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:30.316560 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:31.165442 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:31.165400 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl"] Oct 30 20:14:31.165875 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:31.165840 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" podUID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" containerName="istio-proxy" containerID="cri-o://3a44fe7d5fe6acc03144dca101f3f5928132b56d35fc7d5a8cc308327bd6ab82" gracePeriod=30 Oct 30 20:14:31.468996 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:31.468899 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:31.687911 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:31.687875 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:14:31.688552 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:31.688527 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:14:32.244197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:32.233090 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:33.085172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:33.083809 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:34.088825 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.088786 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:34.172666 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.172625 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp"] Oct 30 20:14:34.173869 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.173843 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" containerName="istio-proxy" Oct 30 20:14:34.173984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.173870 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" containerName="istio-proxy" Oct 30 20:14:34.174126 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.174112 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" containerName="istio-proxy" Oct 30 20:14:34.174210 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.174135 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" containerName="istio-proxy" Oct 30 20:14:34.180916 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.180895 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" Oct 30 20:14:34.189585 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.189555 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp"] Oct 30 20:14:34.191641 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.191429 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:14:34.191641 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.191476 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:14:34.293207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.293171 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1252f0d1-ngkjp\" (UID: \"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" Oct 30 20:14:34.293393 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.293317 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1252f0d1-ngkjp\" (UID: \"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" Oct 30 20:14:34.293393 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.293355 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whcnc\" (UniqueName: \"kubernetes.io/projected/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-kube-api-access-whcnc\") pod \"maas-api-revoke-sub-1252f0d1-ngkjp\" (UID: \"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" Oct 30 20:14:34.395202 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.395095 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whcnc\" (UniqueName: \"kubernetes.io/projected/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-kube-api-access-whcnc\") pod \"maas-api-revoke-sub-1252f0d1-ngkjp\" (UID: \"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" Oct 30 20:14:34.395414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.395396 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1252f0d1-ngkjp\" (UID: \"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" Oct 30 20:14:34.395530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.395514 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1252f0d1-ngkjp\" (UID: \"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" Oct 30 20:14:34.396162 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.396126 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1252f0d1-ngkjp\" (UID: \"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" Oct 30 20:14:34.397966 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.397940 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1252f0d1-ngkjp\" (UID: \"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" Oct 30 20:14:34.410041 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.410017 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whcnc\" (UniqueName: \"kubernetes.io/projected/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-kube-api-access-whcnc\") pod \"maas-api-revoke-sub-1252f0d1-ngkjp\" (UID: \"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" Oct 30 20:14:34.504225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.504197 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" Oct 30 20:14:34.796825 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:14:34.796779 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e26c2b6_aafc_4eec_a7ab_eb24d6c456de.slice/crio-7790c37886fa1ca0d34e1152ffc20dd51849a12c1eab70f3ab630e0da181098c WatchSource:0}: Error finding container 7790c37886fa1ca0d34e1152ffc20dd51849a12c1eab70f3ab630e0da181098c: Status 404 returned error can't find the container with id 7790c37886fa1ca0d34e1152ffc20dd51849a12c1eab70f3ab630e0da181098c Oct 30 20:14:34.797338 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.797315 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp"] Oct 30 20:14:34.931939 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:34.931901 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" event={"ID":"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de","Type":"ContainerStarted","Data":"7790c37886fa1ca0d34e1152ffc20dd51849a12c1eab70f3ab630e0da181098c"} Oct 30 20:14:35.021768 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:35.021503 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:35.994461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:35.993988 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:36.000629 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:35.995923 5091 generic.go:356] "Generic (PLEG): container finished" podID="1e26c2b6-aafc-4eec-a7ab-eb24d6c456de" containerID="9fe10b78c457562ffefcbbeb83cfe8024009c8a7eedc436ed4b5d3bae78df5b8" exitCode=0 Oct 30 20:14:36.000629 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:35.995986 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" event={"ID":"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de","Type":"ContainerDied","Data":"9fe10b78c457562ffefcbbeb83cfe8024009c8a7eedc436ed4b5d3bae78df5b8"} Oct 30 20:14:36.489832 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.489797 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 20:14:36.529806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.529771 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-workload-socket\") pod \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " Oct 30 20:14:36.529965 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.529863 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-envoy\") pod \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " Oct 30 20:14:36.530017 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.529960 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istiod-ca-cert\") pod \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " Oct 30 20:14:36.530062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.530006 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-workload-socket" pod "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" (UID: "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:36.530062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.530023 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-credential-socket\") pod \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " Oct 30 20:14:36.530155 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.530062 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-data\") pod \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " Oct 30 20:14:36.530155 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.530119 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-podinfo\") pod \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " Oct 30 20:14:36.530252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.530236 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-kube-api-access-7lgrx\" (UniqueName: \"kubernetes.io/projected/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-kube-api-access-7lgrx\") pod \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " Oct 30 20:14:36.530361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.530330 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-workload-certs\") pod \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " Oct 30 20:14:36.530455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.530381 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-data" pod "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" (UID: "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:36.530455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.530437 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-credential-socket" pod "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" (UID: "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:36.530591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.530441 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istiod-ca-cert" pod "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" (UID: "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:36.530591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.530441 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-token\" (UniqueName: \"kubernetes.io/projected/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-token\") pod \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\" (UID: \"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4\") " Oct 30 20:14:36.530710 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.530661 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-workload-certs" pod "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" (UID: "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:36.531682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.531552 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:36.531682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.531582 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:36.531682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.531597 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:36.531682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.531613 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:36.531682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.531627 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:36.532560 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.532532 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-envoy" pod "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" (UID: "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:14:36.532877 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.532854 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-token" pod "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" (UID: "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:36.532960 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.532943 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-podinfo" pod "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" (UID: "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:14:36.533401 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.533375 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-kube-api-access-7lgrx" pod "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" (UID: "a9241c43-7eee-4c68-a8e6-6d89c81ec4e4"). InnerVolumeSpecName "kube-api-access-7lgrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:36.637333 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.637296 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:36.637333 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.637330 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7lgrx\" (UniqueName: \"kubernetes.io/projected/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-kube-api-access-7lgrx\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:36.637333 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.637342 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:36.637636 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.637351 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:36.846573 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:36.846475 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:37.006299 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.006256 5091 generic.go:356] "Generic (PLEG): container finished" podID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" containerID="3a44fe7d5fe6acc03144dca101f3f5928132b56d35fc7d5a8cc308327bd6ab82" exitCode=0 Oct 30 20:14:37.006804 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.006300 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" event={"ID":"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4","Type":"ContainerDied","Data":"3a44fe7d5fe6acc03144dca101f3f5928132b56d35fc7d5a8cc308327bd6ab82"} Oct 30 20:14:37.006804 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.006354 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" Oct 30 20:14:37.006804 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.006402 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl" event={"ID":"a9241c43-7eee-4c68-a8e6-6d89c81ec4e4","Type":"ContainerDied","Data":"1a1803844b3dd6cd2a9ce3f33fa30f1cb175f635ba6da2f51944416888c6e291"} Oct 30 20:14:37.006804 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.006431 5091 scope.go:122] "RemoveContainer" containerID="3a44fe7d5fe6acc03144dca101f3f5928132b56d35fc7d5a8cc308327bd6ab82" Oct 30 20:14:37.020776 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.020750 5091 scope.go:122] "RemoveContainer" containerID="3a44fe7d5fe6acc03144dca101f3f5928132b56d35fc7d5a8cc308327bd6ab82" Oct 30 20:14:37.021108 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:14:37.021081 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a44fe7d5fe6acc03144dca101f3f5928132b56d35fc7d5a8cc308327bd6ab82\": container with ID starting with 3a44fe7d5fe6acc03144dca101f3f5928132b56d35fc7d5a8cc308327bd6ab82 not found: ID does not exist" containerID="3a44fe7d5fe6acc03144dca101f3f5928132b56d35fc7d5a8cc308327bd6ab82" Oct 30 20:14:37.021230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.021116 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a44fe7d5fe6acc03144dca101f3f5928132b56d35fc7d5a8cc308327bd6ab82"} err="failed to get container status \"3a44fe7d5fe6acc03144dca101f3f5928132b56d35fc7d5a8cc308327bd6ab82\": rpc error: code = NotFound desc = could not find container \"3a44fe7d5fe6acc03144dca101f3f5928132b56d35fc7d5a8cc308327bd6ab82\": container with ID starting with 3a44fe7d5fe6acc03144dca101f3f5928132b56d35fc7d5a8cc308327bd6ab82 not found: ID does not exist" Oct 30 20:14:37.045094 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.045048 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl"] Oct 30 20:14:37.051576 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.051541 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-0eeba5-openshift-default-7f66686885-8bmgl"] Oct 30 20:14:37.209983 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.209957 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" Oct 30 20:14:37.255265 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.253999 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-maas-api-openshift-ca\") pod \"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de\" (UID: \"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de\") " Oct 30 20:14:37.255265 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.254101 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-kube-api-access-whcnc\" (UniqueName: \"kubernetes.io/projected/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-kube-api-access-whcnc\") pod \"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de\" (UID: \"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de\") " Oct 30 20:14:37.255265 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.254216 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-maas-api-xks-ca\") pod \"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de\" (UID: \"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de\") " Oct 30 20:14:37.255265 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.254442 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-maas-api-openshift-ca" pod "1e26c2b6-aafc-4eec-a7ab-eb24d6c456de" (UID: "1e26c2b6-aafc-4eec-a7ab-eb24d6c456de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:14:37.255265 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.255224 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:37.258914 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.258883 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-maas-api-xks-ca" pod "1e26c2b6-aafc-4eec-a7ab-eb24d6c456de" (UID: "1e26c2b6-aafc-4eec-a7ab-eb24d6c456de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:14:37.259133 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.259113 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-kube-api-access-whcnc" pod "1e26c2b6-aafc-4eec-a7ab-eb24d6c456de" (UID: "1e26c2b6-aafc-4eec-a7ab-eb24d6c456de"). InnerVolumeSpecName "kube-api-access-whcnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:14:37.357044 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.357004 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whcnc\" (UniqueName: \"kubernetes.io/projected/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-kube-api-access-whcnc\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:37.357044 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.357040 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:14:37.813350 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:37.813311 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:38.016741 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:38.016711 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" Oct 30 20:14:38.016741 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:38.016720 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp" event={"ID":"1e26c2b6-aafc-4eec-a7ab-eb24d6c456de","Type":"ContainerDied","Data":"7790c37886fa1ca0d34e1152ffc20dd51849a12c1eab70f3ab630e0da181098c"} Oct 30 20:14:38.017355 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:38.016770 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7790c37886fa1ca0d34e1152ffc20dd51849a12c1eab70f3ab630e0da181098c" Oct 30 20:14:38.444248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:38.444215 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" path="/var/lib/kubelet/pods/a9241c43-7eee-4c68-a8e6-6d89c81ec4e4/volumes" Oct 30 20:14:38.793939 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:38.793832 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:39.640167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:39.637064 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:40.518168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:40.506876 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:41.635027 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:41.634980 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:42.442843 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:42.442802 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:43.299169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:43.289269 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:44.397776 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:44.397729 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:45.193289 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:45.193249 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:45.472240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:45.452312 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:45.823651 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:45.823614 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:46.020079 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:14:46.020043 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:14:46.805168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:46.794673 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:47.520556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:47.520518 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:48.101167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:48.084894 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:48.286489 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:48.286174 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:48.613929 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:48.613893 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:49.157338 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:49.157293 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:50.075797 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:50.075717 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:51.130406 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:51.130374 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:52.075036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:52.074994 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:52.498714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:52.498668 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:53.185961 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:53.185915 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:53.495340 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:53.495247 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:53.798167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:53.794753 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:54.295812 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:54.295201 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:55.468168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:55.466768 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:56.532006 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:56.515964 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:57.373467 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:57.373423 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6976477d7d-88x4l"] Oct 30 20:14:57.450409 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:57.450374 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:58.275767 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:58.269786 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:14:59.204911 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:14:59.204872 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:00.155047 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.155010 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:00.185926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.185888 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd"] Oct 30 20:15:00.186724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.186706 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" containerName="istio-proxy" Oct 30 20:15:00.186804 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.186725 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0153a5c2-c41c-4dd5-9bc2-035455025bd1" containerName="istio-proxy" Oct 30 20:15:00.186804 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.186749 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1e26c2b6-aafc-4eec-a7ab-eb24d6c456de" containerName="revoke-keys" Oct 30 20:15:00.186804 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.186755 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e26c2b6-aafc-4eec-a7ab-eb24d6c456de" containerName="revoke-keys" Oct 30 20:15:00.186804 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.186769 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" containerName="istio-proxy" Oct 30 20:15:00.186804 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.186775 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" containerName="istio-proxy" Oct 30 20:15:00.186980 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.186882 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9241c43-7eee-4c68-a8e6-6d89c81ec4e4" containerName="istio-proxy" Oct 30 20:15:00.186980 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.186902 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e26c2b6-aafc-4eec-a7ab-eb24d6c456de" containerName="revoke-keys" Oct 30 20:15:00.191744 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.191721 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" Oct 30 20:15:00.194213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.194190 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz"] Oct 30 20:15:00.198691 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.198666 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" Oct 30 20:15:00.199197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.199175 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc"] Oct 30 20:15:00.203837 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.203803 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d"] Oct 30 20:15:00.204720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.203954 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" Oct 30 20:15:00.211225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.211204 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" Oct 30 20:15:00.212230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.212207 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm"] Oct 30 20:15:00.217948 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.217898 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd"] Oct 30 20:15:00.218068 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.217973 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" Oct 30 20:15:00.223389 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.223365 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:15:00.223777 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.223725 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz"] Oct 30 20:15:00.234405 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.231020 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d"] Oct 30 20:15:00.266253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.266217 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc"] Oct 30 20:15:00.304851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.304818 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7"] Oct 30 20:15:00.313036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.313003 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v"] Oct 30 20:15:00.313601 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.313574 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" Oct 30 20:15:00.318095 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.318074 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" Oct 30 20:15:00.335987 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.327872 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6"] Oct 30 20:15:00.335987 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.334870 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" Oct 30 20:15:00.336840 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.336820 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct"] Oct 30 20:15:00.341834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.341810 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" Oct 30 20:15:00.362203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.350530 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm"] Oct 30 20:15:00.362203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.355031 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v"] Oct 30 20:15:00.362203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.359917 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27d65d4-ebb6-4254-89c9-febcacbdf9f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd\" (UID: \"d27d65d4-ebb6-4254-89c9-febcacbdf9f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" Oct 30 20:15:00.362203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.359950 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rttn\" (UniqueName: \"kubernetes.io/projected/d27d65d4-ebb6-4254-89c9-febcacbdf9f1-kube-api-access-2rttn\") pod \"maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd\" (UID: \"d27d65d4-ebb6-4254-89c9-febcacbdf9f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" Oct 30 20:15:00.362203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.359976 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh6zk\" (UniqueName: \"kubernetes.io/projected/78e70c72-9983-4b2e-bd0c-176f50822feb-kube-api-access-sh6zk\") pod \"maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d\" (UID: \"78e70c72-9983-4b2e-bd0c-176f50822feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" Oct 30 20:15:00.362203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.360024 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e70c72-9983-4b2e-bd0c-176f50822feb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d\" (UID: \"78e70c72-9983-4b2e-bd0c-176f50822feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" Oct 30 20:15:00.362203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.360043 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq6wr\" (UniqueName: \"kubernetes.io/projected/1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c-kube-api-access-pq6wr\") pod \"maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz\" (UID: \"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" Oct 30 20:15:00.362203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.360065 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f93e44d7-d7d5-4beb-9332-403ddc9bf336-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc\" (UID: \"f93e44d7-d7d5-4beb-9332-403ddc9bf336\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" Oct 30 20:15:00.362203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.360184 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz\" (UID: \"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" Oct 30 20:15:00.362203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.360242 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r9x2\" (UniqueName: \"kubernetes.io/projected/f93e44d7-d7d5-4beb-9332-403ddc9bf336-kube-api-access-2r9x2\") pod \"maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc\" (UID: \"f93e44d7-d7d5-4beb-9332-403ddc9bf336\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" Oct 30 20:15:00.362203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.360334 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e33682-f349-450b-ab65-1b60d304a6cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm\" (UID: \"87e33682-f349-450b-ab65-1b60d304a6cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" Oct 30 20:15:00.362203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.360400 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2x5r\" (UniqueName: \"kubernetes.io/projected/87e33682-f349-450b-ab65-1b60d304a6cf-kube-api-access-f2x5r\") pod \"maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm\" (UID: \"87e33682-f349-450b-ab65-1b60d304a6cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" Oct 30 20:15:00.362203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.360613 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28"] Oct 30 20:15:00.370178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.368805 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6"] Oct 30 20:15:00.370178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.368936 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" Oct 30 20:15:00.373788 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.373768 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7"] Oct 30 20:15:00.378422 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.378404 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl"] Oct 30 20:15:00.388327 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.388310 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx"] Oct 30 20:15:00.388632 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.388533 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" Oct 30 20:15:00.398445 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.398426 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" Oct 30 20:15:00.399175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.399130 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28"] Oct 30 20:15:00.405461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.405020 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct"] Oct 30 20:15:00.422135 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.422104 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx"] Oct 30 20:15:00.428904 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.428854 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb"] Oct 30 20:15:00.435660 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.435635 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.461546 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27d65d4-ebb6-4254-89c9-febcacbdf9f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd\" (UID: \"d27d65d4-ebb6-4254-89c9-febcacbdf9f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.461600 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rttn\" (UniqueName: \"kubernetes.io/projected/d27d65d4-ebb6-4254-89c9-febcacbdf9f1-kube-api-access-2rttn\") pod \"maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd\" (UID: \"d27d65d4-ebb6-4254-89c9-febcacbdf9f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.461644 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh6zk\" (UniqueName: \"kubernetes.io/projected/78e70c72-9983-4b2e-bd0c-176f50822feb-kube-api-access-sh6zk\") pod \"maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d\" (UID: \"78e70c72-9983-4b2e-bd0c-176f50822feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.461697 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee32ddbf-9640-4efd-b2a5-babcfb306088-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6\" (UID: \"ee32ddbf-9640-4efd-b2a5-babcfb306088\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.461758 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e70c72-9983-4b2e-bd0c-176f50822feb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d\" (UID: \"78e70c72-9983-4b2e-bd0c-176f50822feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.461787 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq6wr\" (UniqueName: \"kubernetes.io/projected/1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c-kube-api-access-pq6wr\") pod \"maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz\" (UID: \"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.462614 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f93e44d7-d7d5-4beb-9332-403ddc9bf336-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc\" (UID: \"f93e44d7-d7d5-4beb-9332-403ddc9bf336\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.461825 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f93e44d7-d7d5-4beb-9332-403ddc9bf336-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc\" (UID: \"f93e44d7-d7d5-4beb-9332-403ddc9bf336\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.462766 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/387559a4-9827-4a53-90ae-3f3f0852d25d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct\" (UID: \"387559a4-9827-4a53-90ae-3f3f0852d25d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.462787 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz\" (UID: \"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.462828 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r9x2\" (UniqueName: \"kubernetes.io/projected/f93e44d7-d7d5-4beb-9332-403ddc9bf336-kube-api-access-2r9x2\") pod \"maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc\" (UID: \"f93e44d7-d7d5-4beb-9332-403ddc9bf336\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.462847 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eafdae9-7fcd-4579-824b-60bf84bb9b7f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28\" (UID: \"4eafdae9-7fcd-4579-824b-60bf84bb9b7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.462879 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46t2w\" (UniqueName: \"kubernetes.io/projected/387559a4-9827-4a53-90ae-3f3f0852d25d-kube-api-access-46t2w\") pod \"maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct\" (UID: \"387559a4-9827-4a53-90ae-3f3f0852d25d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.462906 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67449cd5-2639-4625-b32e-ba3bec5a528b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v\" (UID: \"67449cd5-2639-4625-b32e-ba3bec5a528b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.462924 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4hgd\" (UniqueName: \"kubernetes.io/projected/67449cd5-2639-4625-b32e-ba3bec5a528b-kube-api-access-b4hgd\") pod \"maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v\" (UID: \"67449cd5-2639-4625-b32e-ba3bec5a528b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.462958 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e33682-f349-450b-ab65-1b60d304a6cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm\" (UID: \"87e33682-f349-450b-ab65-1b60d304a6cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.462982 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8698\" (UniqueName: \"kubernetes.io/projected/4eafdae9-7fcd-4579-824b-60bf84bb9b7f-kube-api-access-f8698\") pod \"maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28\" (UID: \"4eafdae9-7fcd-4579-824b-60bf84bb9b7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.463002 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d37c05b-93b8-4c8c-84d1-9e186d7ec265-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7\" (UID: \"3d37c05b-93b8-4c8c-84d1-9e186d7ec265\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.463035 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2x5r\" (UniqueName: \"kubernetes.io/projected/87e33682-f349-450b-ab65-1b60d304a6cf-kube-api-access-f2x5r\") pod \"maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm\" (UID: \"87e33682-f349-450b-ab65-1b60d304a6cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.463053 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d62b\" (UniqueName: \"kubernetes.io/projected/3d37c05b-93b8-4c8c-84d1-9e186d7ec265-kube-api-access-2d62b\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7\" (UID: \"3d37c05b-93b8-4c8c-84d1-9e186d7ec265\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.463075 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szcpv\" (UniqueName: \"kubernetes.io/projected/ee32ddbf-9640-4efd-b2a5-babcfb306088-kube-api-access-szcpv\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6\" (UID: \"ee32ddbf-9640-4efd-b2a5-babcfb306088\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" Oct 30 20:15:00.464205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.464081 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz\" (UID: \"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" Oct 30 20:15:00.465597 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.464509 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27d65d4-ebb6-4254-89c9-febcacbdf9f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd\" (UID: \"d27d65d4-ebb6-4254-89c9-febcacbdf9f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" Oct 30 20:15:00.465597 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.465129 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e33682-f349-450b-ab65-1b60d304a6cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm\" (UID: \"87e33682-f349-450b-ab65-1b60d304a6cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" Oct 30 20:15:00.470137 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.470115 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl"] Oct 30 20:15:00.470275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.470253 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb"] Oct 30 20:15:00.470344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.470287 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p"] Oct 30 20:15:00.472436 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.472410 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e70c72-9983-4b2e-bd0c-176f50822feb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d\" (UID: \"78e70c72-9983-4b2e-bd0c-176f50822feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" Oct 30 20:15:00.477399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.477378 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" Oct 30 20:15:00.483331 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.483312 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw"] Oct 30 20:15:00.492612 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.492585 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" Oct 30 20:15:00.492881 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.492501 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p"] Oct 30 20:15:00.499599 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.498719 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh6zk\" (UniqueName: \"kubernetes.io/projected/78e70c72-9983-4b2e-bd0c-176f50822feb-kube-api-access-sh6zk\") pod \"maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d\" (UID: \"78e70c72-9983-4b2e-bd0c-176f50822feb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" Oct 30 20:15:00.499599 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.499192 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm"] Oct 30 20:15:00.502081 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.502021 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2x5r\" (UniqueName: \"kubernetes.io/projected/87e33682-f349-450b-ab65-1b60d304a6cf-kube-api-access-f2x5r\") pod \"maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm\" (UID: \"87e33682-f349-450b-ab65-1b60d304a6cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" Oct 30 20:15:00.505379 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.505357 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw"] Oct 30 20:15:00.505471 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.505457 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" Oct 30 20:15:00.505575 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.505559 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rttn\" (UniqueName: \"kubernetes.io/projected/d27d65d4-ebb6-4254-89c9-febcacbdf9f1-kube-api-access-2rttn\") pod \"maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd\" (UID: \"d27d65d4-ebb6-4254-89c9-febcacbdf9f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" Oct 30 20:15:00.505662 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.505567 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r9x2\" (UniqueName: \"kubernetes.io/projected/f93e44d7-d7d5-4beb-9332-403ddc9bf336-kube-api-access-2r9x2\") pod \"maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc\" (UID: \"f93e44d7-d7d5-4beb-9332-403ddc9bf336\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" Oct 30 20:15:00.508038 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.508015 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq6wr\" (UniqueName: \"kubernetes.io/projected/1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c-kube-api-access-pq6wr\") pod \"maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz\" (UID: \"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" Oct 30 20:15:00.510777 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.510755 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" Oct 30 20:15:00.518290 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.518231 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm"] Oct 30 20:15:00.522907 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.522745 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" Oct 30 20:15:00.532661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.531416 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx"] Oct 30 20:15:00.538697 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.538674 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" Oct 30 20:15:00.553273 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.553245 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" Oct 30 20:15:00.583569 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.583536 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8336ff5-7043-4f2a-8ab1-60c4713596e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p\" (UID: \"b8336ff5-7043-4f2a-8ab1-60c4713596e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" Oct 30 20:15:00.584092 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.584063 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee32ddbf-9640-4efd-b2a5-babcfb306088-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6\" (UID: \"ee32ddbf-9640-4efd-b2a5-babcfb306088\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" Oct 30 20:15:00.586867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.585312 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee32ddbf-9640-4efd-b2a5-babcfb306088-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6\" (UID: \"ee32ddbf-9640-4efd-b2a5-babcfb306088\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" Oct 30 20:15:00.586867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.586793 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d9f0fd-4bb0-4773-8333-92f168e683c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb\" (UID: \"b7d9f0fd-4bb0-4773-8333-92f168e683c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" Oct 30 20:15:00.587061 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.586897 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wjjb\" (UniqueName: \"kubernetes.io/projected/b7d9f0fd-4bb0-4773-8333-92f168e683c3-kube-api-access-7wjjb\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb\" (UID: \"b7d9f0fd-4bb0-4773-8333-92f168e683c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" Oct 30 20:15:00.587061 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.586969 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/755783df-2b63-4013-9133-d39c004e577f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl\" (UID: \"755783df-2b63-4013-9133-d39c004e577f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" Oct 30 20:15:00.587061 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.587050 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c86zj\" (UniqueName: \"kubernetes.io/projected/755783df-2b63-4013-9133-d39c004e577f-kube-api-access-c86zj\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl\" (UID: \"755783df-2b63-4013-9133-d39c004e577f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" Oct 30 20:15:00.587254 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.587096 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/387559a4-9827-4a53-90ae-3f3f0852d25d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct\" (UID: \"387559a4-9827-4a53-90ae-3f3f0852d25d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" Oct 30 20:15:00.587254 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.587184 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eafdae9-7fcd-4579-824b-60bf84bb9b7f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28\" (UID: \"4eafdae9-7fcd-4579-824b-60bf84bb9b7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" Oct 30 20:15:00.587254 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.587239 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46t2w\" (UniqueName: \"kubernetes.io/projected/387559a4-9827-4a53-90ae-3f3f0852d25d-kube-api-access-46t2w\") pod \"maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct\" (UID: \"387559a4-9827-4a53-90ae-3f3f0852d25d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" Oct 30 20:15:00.587417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.587291 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67449cd5-2639-4625-b32e-ba3bec5a528b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v\" (UID: \"67449cd5-2639-4625-b32e-ba3bec5a528b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" Oct 30 20:15:00.587417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.587326 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4hgd\" (UniqueName: \"kubernetes.io/projected/67449cd5-2639-4625-b32e-ba3bec5a528b-kube-api-access-b4hgd\") pod \"maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v\" (UID: \"67449cd5-2639-4625-b32e-ba3bec5a528b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" Oct 30 20:15:00.587584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.587418 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8698\" (UniqueName: \"kubernetes.io/projected/4eafdae9-7fcd-4579-824b-60bf84bb9b7f-kube-api-access-f8698\") pod \"maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28\" (UID: \"4eafdae9-7fcd-4579-824b-60bf84bb9b7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" Oct 30 20:15:00.587584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.587459 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d37c05b-93b8-4c8c-84d1-9e186d7ec265-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7\" (UID: \"3d37c05b-93b8-4c8c-84d1-9e186d7ec265\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" Oct 30 20:15:00.587584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.587509 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p575b\" (UniqueName: \"kubernetes.io/projected/b8336ff5-7043-4f2a-8ab1-60c4713596e9-kube-api-access-p575b\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p\" (UID: \"b8336ff5-7043-4f2a-8ab1-60c4713596e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" Oct 30 20:15:00.587584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.587549 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d62b\" (UniqueName: \"kubernetes.io/projected/3d37c05b-93b8-4c8c-84d1-9e186d7ec265-kube-api-access-2d62b\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7\" (UID: \"3d37c05b-93b8-4c8c-84d1-9e186d7ec265\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" Oct 30 20:15:00.587813 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.587589 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szcpv\" (UniqueName: \"kubernetes.io/projected/ee32ddbf-9640-4efd-b2a5-babcfb306088-kube-api-access-szcpv\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6\" (UID: \"ee32ddbf-9640-4efd-b2a5-babcfb306088\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" Oct 30 20:15:00.593112 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.589474 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/387559a4-9827-4a53-90ae-3f3f0852d25d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct\" (UID: \"387559a4-9827-4a53-90ae-3f3f0852d25d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" Oct 30 20:15:00.593112 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.590200 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eafdae9-7fcd-4579-824b-60bf84bb9b7f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28\" (UID: \"4eafdae9-7fcd-4579-824b-60bf84bb9b7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" Oct 30 20:15:00.593112 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.590942 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d37c05b-93b8-4c8c-84d1-9e186d7ec265-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7\" (UID: \"3d37c05b-93b8-4c8c-84d1-9e186d7ec265\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" Oct 30 20:15:00.601999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.599011 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67449cd5-2639-4625-b32e-ba3bec5a528b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v\" (UID: \"67449cd5-2639-4625-b32e-ba3bec5a528b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" Oct 30 20:15:00.601999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.601214 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b69782b-78e9-4baf-8bdf-bcbf9b78767d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx\" (UID: \"8b69782b-78e9-4baf-8bdf-bcbf9b78767d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" Oct 30 20:15:00.601999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.601284 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmwgg\" (UniqueName: \"kubernetes.io/projected/8b69782b-78e9-4baf-8bdf-bcbf9b78767d-kube-api-access-kmwgg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx\" (UID: \"8b69782b-78e9-4baf-8bdf-bcbf9b78767d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" Oct 30 20:15:00.602912 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.602343 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" Oct 30 20:15:00.611191 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.610096 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx"] Oct 30 20:15:00.615268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.615247 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8698\" (UniqueName: \"kubernetes.io/projected/4eafdae9-7fcd-4579-824b-60bf84bb9b7f-kube-api-access-f8698\") pod \"maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28\" (UID: \"4eafdae9-7fcd-4579-824b-60bf84bb9b7f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" Oct 30 20:15:00.620996 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.620967 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szcpv\" (UniqueName: \"kubernetes.io/projected/ee32ddbf-9640-4efd-b2a5-babcfb306088-kube-api-access-szcpv\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6\" (UID: \"ee32ddbf-9640-4efd-b2a5-babcfb306088\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" Oct 30 20:15:00.621374 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.621349 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" Oct 30 20:15:00.625051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.624764 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4hgd\" (UniqueName: \"kubernetes.io/projected/67449cd5-2639-4625-b32e-ba3bec5a528b-kube-api-access-b4hgd\") pod \"maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v\" (UID: \"67449cd5-2639-4625-b32e-ba3bec5a528b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" Oct 30 20:15:00.627586 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.625790 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46t2w\" (UniqueName: \"kubernetes.io/projected/387559a4-9827-4a53-90ae-3f3f0852d25d-kube-api-access-46t2w\") pod \"maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct\" (UID: \"387559a4-9827-4a53-90ae-3f3f0852d25d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" Oct 30 20:15:00.627586 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.626448 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d62b\" (UniqueName: \"kubernetes.io/projected/3d37c05b-93b8-4c8c-84d1-9e186d7ec265-kube-api-access-2d62b\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7\" (UID: \"3d37c05b-93b8-4c8c-84d1-9e186d7ec265\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" Oct 30 20:15:00.671615 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.671588 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" Oct 30 20:15:00.703004 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.702703 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d9f0fd-4bb0-4773-8333-92f168e683c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb\" (UID: \"b7d9f0fd-4bb0-4773-8333-92f168e683c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" Oct 30 20:15:00.718449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.711134 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jt9l\" (UniqueName: \"kubernetes.io/projected/2ea189f9-f225-4fdb-8948-14cb53111cd5-kube-api-access-4jt9l\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw\" (UID: \"2ea189f9-f225-4fdb-8948-14cb53111cd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" Oct 30 20:15:00.718449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.703649 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d9f0fd-4bb0-4773-8333-92f168e683c3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb\" (UID: \"b7d9f0fd-4bb0-4773-8333-92f168e683c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" Oct 30 20:15:00.718449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.711209 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wjjb\" (UniqueName: \"kubernetes.io/projected/b7d9f0fd-4bb0-4773-8333-92f168e683c3-kube-api-access-7wjjb\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb\" (UID: \"b7d9f0fd-4bb0-4773-8333-92f168e683c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" Oct 30 20:15:00.718449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.711807 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/755783df-2b63-4013-9133-d39c004e577f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl\" (UID: \"755783df-2b63-4013-9133-d39c004e577f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" Oct 30 20:15:00.718449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.712078 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c86zj\" (UniqueName: \"kubernetes.io/projected/755783df-2b63-4013-9133-d39c004e577f-kube-api-access-c86zj\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl\" (UID: \"755783df-2b63-4013-9133-d39c004e577f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" Oct 30 20:15:00.718449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.712128 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b957\" (UniqueName: \"kubernetes.io/projected/8fb65533-38aa-48c4-a374-fcf2f5e96f99-kube-api-access-5b957\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm\" (UID: \"8fb65533-38aa-48c4-a374-fcf2f5e96f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" Oct 30 20:15:00.718449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.712523 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p575b\" (UniqueName: \"kubernetes.io/projected/b8336ff5-7043-4f2a-8ab1-60c4713596e9-kube-api-access-p575b\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p\" (UID: \"b8336ff5-7043-4f2a-8ab1-60c4713596e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" Oct 30 20:15:00.718449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.712611 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea189f9-f225-4fdb-8948-14cb53111cd5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw\" (UID: \"2ea189f9-f225-4fdb-8948-14cb53111cd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" Oct 30 20:15:00.718449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.712759 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" Oct 30 20:15:00.718449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.713480 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b69782b-78e9-4baf-8bdf-bcbf9b78767d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx\" (UID: \"8b69782b-78e9-4baf-8bdf-bcbf9b78767d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" Oct 30 20:15:00.718449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.713545 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmwgg\" (UniqueName: \"kubernetes.io/projected/8b69782b-78e9-4baf-8bdf-bcbf9b78767d-kube-api-access-kmwgg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx\" (UID: \"8b69782b-78e9-4baf-8bdf-bcbf9b78767d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" Oct 30 20:15:00.718449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.713575 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8336ff5-7043-4f2a-8ab1-60c4713596e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p\" (UID: \"b8336ff5-7043-4f2a-8ab1-60c4713596e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" Oct 30 20:15:00.718449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.713607 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb65533-38aa-48c4-a374-fcf2f5e96f99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm\" (UID: \"8fb65533-38aa-48c4-a374-fcf2f5e96f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" Oct 30 20:15:00.718449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.715383 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8336ff5-7043-4f2a-8ab1-60c4713596e9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p\" (UID: \"b8336ff5-7043-4f2a-8ab1-60c4713596e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" Oct 30 20:15:00.732245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.720026 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b69782b-78e9-4baf-8bdf-bcbf9b78767d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx\" (UID: \"8b69782b-78e9-4baf-8bdf-bcbf9b78767d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" Oct 30 20:15:00.732245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.723487 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" Oct 30 20:15:00.732245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.726249 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/755783df-2b63-4013-9133-d39c004e577f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl\" (UID: \"755783df-2b63-4013-9133-d39c004e577f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" Oct 30 20:15:00.742455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.737094 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c86zj\" (UniqueName: \"kubernetes.io/projected/755783df-2b63-4013-9133-d39c004e577f-kube-api-access-c86zj\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl\" (UID: \"755783df-2b63-4013-9133-d39c004e577f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" Oct 30 20:15:00.742455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.738600 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p575b\" (UniqueName: \"kubernetes.io/projected/b8336ff5-7043-4f2a-8ab1-60c4713596e9-kube-api-access-p575b\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p\" (UID: \"b8336ff5-7043-4f2a-8ab1-60c4713596e9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" Oct 30 20:15:00.742455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.740759 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wjjb\" (UniqueName: \"kubernetes.io/projected/b7d9f0fd-4bb0-4773-8333-92f168e683c3-kube-api-access-7wjjb\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb\" (UID: \"b7d9f0fd-4bb0-4773-8333-92f168e683c3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" Oct 30 20:15:00.742455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.741003 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" Oct 30 20:15:00.742455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.742312 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmwgg\" (UniqueName: \"kubernetes.io/projected/8b69782b-78e9-4baf-8bdf-bcbf9b78767d-kube-api-access-kmwgg\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx\" (UID: \"8b69782b-78e9-4baf-8bdf-bcbf9b78767d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" Oct 30 20:15:00.764104 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.755717 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" Oct 30 20:15:00.794569 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.794349 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" Oct 30 20:15:00.819412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.816495 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jt9l\" (UniqueName: \"kubernetes.io/projected/2ea189f9-f225-4fdb-8948-14cb53111cd5-kube-api-access-4jt9l\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw\" (UID: \"2ea189f9-f225-4fdb-8948-14cb53111cd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" Oct 30 20:15:00.819412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.816573 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5e04ce-a8ad-4b44-bd00-624a49388708-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx\" (UID: \"2a5e04ce-a8ad-4b44-bd00-624a49388708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" Oct 30 20:15:00.819412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.816699 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b957\" (UniqueName: \"kubernetes.io/projected/8fb65533-38aa-48c4-a374-fcf2f5e96f99-kube-api-access-5b957\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm\" (UID: \"8fb65533-38aa-48c4-a374-fcf2f5e96f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" Oct 30 20:15:00.819412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.816880 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh6zz\" (UniqueName: \"kubernetes.io/projected/2a5e04ce-a8ad-4b44-bd00-624a49388708-kube-api-access-vh6zz\") pod \"maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx\" (UID: \"2a5e04ce-a8ad-4b44-bd00-624a49388708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" Oct 30 20:15:00.819412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.817062 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea189f9-f225-4fdb-8948-14cb53111cd5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw\" (UID: \"2ea189f9-f225-4fdb-8948-14cb53111cd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" Oct 30 20:15:00.819412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.817173 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb65533-38aa-48c4-a374-fcf2f5e96f99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm\" (UID: \"8fb65533-38aa-48c4-a374-fcf2f5e96f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" Oct 30 20:15:00.819412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.818218 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb65533-38aa-48c4-a374-fcf2f5e96f99-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm\" (UID: \"8fb65533-38aa-48c4-a374-fcf2f5e96f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" Oct 30 20:15:00.820178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.819712 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea189f9-f225-4fdb-8948-14cb53111cd5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw\" (UID: \"2ea189f9-f225-4fdb-8948-14cb53111cd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" Oct 30 20:15:00.852627 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.852555 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b957\" (UniqueName: \"kubernetes.io/projected/8fb65533-38aa-48c4-a374-fcf2f5e96f99-kube-api-access-5b957\") pod \"maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm\" (UID: \"8fb65533-38aa-48c4-a374-fcf2f5e96f99\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" Oct 30 20:15:00.882294 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.882061 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" Oct 30 20:15:00.896215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.895690 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jt9l\" (UniqueName: \"kubernetes.io/projected/2ea189f9-f225-4fdb-8948-14cb53111cd5-kube-api-access-4jt9l\") pod \"maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw\" (UID: \"2ea189f9-f225-4fdb-8948-14cb53111cd5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" Oct 30 20:15:00.920453 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.919041 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5e04ce-a8ad-4b44-bd00-624a49388708-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx\" (UID: \"2a5e04ce-a8ad-4b44-bd00-624a49388708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" Oct 30 20:15:00.920453 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.919250 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh6zz\" (UniqueName: \"kubernetes.io/projected/2a5e04ce-a8ad-4b44-bd00-624a49388708-kube-api-access-vh6zz\") pod \"maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx\" (UID: \"2a5e04ce-a8ad-4b44-bd00-624a49388708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" Oct 30 20:15:00.920453 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.920395 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd"] Oct 30 20:15:00.921263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.921199 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5e04ce-a8ad-4b44-bd00-624a49388708-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx\" (UID: \"2a5e04ce-a8ad-4b44-bd00-624a49388708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" Oct 30 20:15:00.978412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.957521 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh6zz\" (UniqueName: \"kubernetes.io/projected/2a5e04ce-a8ad-4b44-bd00-624a49388708-kube-api-access-vh6zz\") pod \"maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx\" (UID: \"2a5e04ce-a8ad-4b44-bd00-624a49388708\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" Oct 30 20:15:00.985978 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:00.985716 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" Oct 30 20:15:01.015195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.013973 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" Oct 30 20:15:01.077310 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.077079 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" Oct 30 20:15:01.123882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.123636 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" Oct 30 20:15:01.155241 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.155204 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" Oct 30 20:15:01.231480 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.223781 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d"] Oct 30 20:15:01.250693 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:01.250638 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78e70c72_9983_4b2e_bd0c_176f50822feb.slice/crio-b50eb55c1a357c9682a14e2263fdf36d10c771f8a4a1bee8bc939e8e862a30e3 WatchSource:0}: Error finding container b50eb55c1a357c9682a14e2263fdf36d10c771f8a4a1bee8bc939e8e862a30e3: Status 404 returned error can't find the container with id b50eb55c1a357c9682a14e2263fdf36d10c771f8a4a1bee8bc939e8e862a30e3 Oct 30 20:15:01.530726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.530619 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:01.542063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.540804 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" event={"ID":"d27d65d4-ebb6-4254-89c9-febcacbdf9f1","Type":"ContainerStarted","Data":"c40c6b2b7c7f5df24b3445584f3d7a6f90a79018775654a287a8430e0376bb4f"} Oct 30 20:15:01.543167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.542568 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" event={"ID":"78e70c72-9983-4b2e-bd0c-176f50822feb","Type":"ContainerStarted","Data":"b50eb55c1a357c9682a14e2263fdf36d10c771f8a4a1bee8bc939e8e862a30e3"} Oct 30 20:15:01.641762 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:01.641636 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod755783df_2b63_4013_9133_d39c004e577f.slice/crio-edada6ee58e3fd9db1c59d1b4446e72994b732f41e24b63a5ea70cbe7c671e47 WatchSource:0}: Error finding container edada6ee58e3fd9db1c59d1b4446e72994b732f41e24b63a5ea70cbe7c671e47: Status 404 returned error can't find the container with id edada6ee58e3fd9db1c59d1b4446e72994b732f41e24b63a5ea70cbe7c671e47 Oct 30 20:15:01.644741 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:01.644712 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eafdae9_7fcd_4579_824b_60bf84bb9b7f.slice/crio-b45bf52a26b95fb23b7295ed0e00d7ad3ca78f459d4e9088f9e6e36c85ed883b WatchSource:0}: Error finding container b45bf52a26b95fb23b7295ed0e00d7ad3ca78f459d4e9088f9e6e36c85ed883b: Status 404 returned error can't find the container with id b45bf52a26b95fb23b7295ed0e00d7ad3ca78f459d4e9088f9e6e36c85ed883b Oct 30 20:15:01.697847 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.697765 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl"] Oct 30 20:15:01.703649 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:01.703612 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387559a4_9827_4a53_90ae_3f3f0852d25d.slice/crio-48c4b43dfded5e400e900d71a3dbaeaedb113521dc35ac4ca97a921f570c813b WatchSource:0}: Error finding container 48c4b43dfded5e400e900d71a3dbaeaedb113521dc35ac4ca97a921f570c813b: Status 404 returned error can't find the container with id 48c4b43dfded5e400e900d71a3dbaeaedb113521dc35ac4ca97a921f570c813b Oct 30 20:15:01.706767 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.706735 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28"] Oct 30 20:15:01.709396 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:01.708257 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8a41b6_c1fd_4d9b_8f3d_c0a93eb8434c.slice/crio-5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397 WatchSource:0}: Error finding container 5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397: Status 404 returned error can't find the container with id 5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397 Oct 30 20:15:01.710852 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:01.710808 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e33682_f349_450b_ab65_1b60d304a6cf.slice/crio-16d8dea45947d1d07cfb2825c499401ece1cdd6a200afdade6bbea2ac0e4e26a WatchSource:0}: Error finding container 16d8dea45947d1d07cfb2825c499401ece1cdd6a200afdade6bbea2ac0e4e26a: Status 404 returned error can't find the container with id 16d8dea45947d1d07cfb2825c499401ece1cdd6a200afdade6bbea2ac0e4e26a Oct 30 20:15:01.714870 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.714780 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7"] Oct 30 20:15:01.721959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.721906 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct"] Oct 30 20:15:01.737995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.734739 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm"] Oct 30 20:15:01.737995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.736509 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz"] Oct 30 20:15:01.744793 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.744678 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc"] Oct 30 20:15:01.773045 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.773016 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v"] Oct 30 20:15:01.869248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.869210 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p"] Oct 30 20:15:01.878457 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.878428 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6"] Oct 30 20:15:01.917626 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.917601 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw"] Oct 30 20:15:01.963739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.961232 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb"] Oct 30 20:15:01.964667 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:01.964610 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8336ff5_7043_4f2a_8ab1_60c4713596e9.slice/crio-235a73b79e3091f0abcdf604c8709fcd7345c221104f8f722f1d5f18b86c7335 WatchSource:0}: Error finding container 235a73b79e3091f0abcdf604c8709fcd7345c221104f8f722f1d5f18b86c7335: Status 404 returned error can't find the container with id 235a73b79e3091f0abcdf604c8709fcd7345c221104f8f722f1d5f18b86c7335 Oct 30 20:15:01.965785 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.965633 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm"] Oct 30 20:15:01.969344 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:01.969289 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee32ddbf_9640_4efd_b2a5_babcfb306088.slice/crio-43dc6447859b302fd7097330b4ebb4268afbcf252840558c7abf2c3739e4e373 WatchSource:0}: Error finding container 43dc6447859b302fd7097330b4ebb4268afbcf252840558c7abf2c3739e4e373: Status 404 returned error can't find the container with id 43dc6447859b302fd7097330b4ebb4268afbcf252840558c7abf2c3739e4e373 Oct 30 20:15:01.971029 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:01.970984 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ea189f9_f225_4fdb_8948_14cb53111cd5.slice/crio-3477becef01b5e48af6cb6cf5d5e453d19732979dd723e528eaf07bb8c9d9c91 WatchSource:0}: Error finding container 3477becef01b5e48af6cb6cf5d5e453d19732979dd723e528eaf07bb8c9d9c91: Status 404 returned error can't find the container with id 3477becef01b5e48af6cb6cf5d5e453d19732979dd723e528eaf07bb8c9d9c91 Oct 30 20:15:01.993021 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:01.990292 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx"] Oct 30 20:15:02.009622 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.009591 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx"] Oct 30 20:15:02.374074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.373873 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" podUID="4efa6cde-c023-40c7-a04b-a73b1605e9c9" containerName="payload-processing" containerID="cri-o://dbd3fa913e10304756a70c3ef7fc22861b674090864e442ff410993f69e8d049" gracePeriod=25 Oct 30 20:15:02.567923 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.567686 5091 generic.go:356] "Generic (PLEG): container finished" podID="d27d65d4-ebb6-4254-89c9-febcacbdf9f1" containerID="83d4558fe4ed31850e31b3a0edcb12ac071e197ab71e93d2de3aa045946186c7" exitCode=60 Oct 30 20:15:02.567923 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.567783 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" event={"ID":"d27d65d4-ebb6-4254-89c9-febcacbdf9f1","Type":"ContainerDied","Data":"83d4558fe4ed31850e31b3a0edcb12ac071e197ab71e93d2de3aa045946186c7"} Oct 30 20:15:02.568582 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.568140 5091 scope.go:122] "RemoveContainer" containerID="83d4558fe4ed31850e31b3a0edcb12ac071e197ab71e93d2de3aa045946186c7" Oct 30 20:15:02.571100 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.570908 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" event={"ID":"ee32ddbf-9640-4efd-b2a5-babcfb306088","Type":"ContainerStarted","Data":"43dc6447859b302fd7097330b4ebb4268afbcf252840558c7abf2c3739e4e373"} Oct 30 20:15:02.573343 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.573316 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" event={"ID":"87e33682-f349-450b-ab65-1b60d304a6cf","Type":"ContainerStarted","Data":"b0d1ecbd65b35d1a19f8105c6105db3e7dd0eb48a0de8a8ec8bb38c0d51a28f6"} Oct 30 20:15:02.574189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.573367 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" event={"ID":"87e33682-f349-450b-ab65-1b60d304a6cf","Type":"ContainerStarted","Data":"16d8dea45947d1d07cfb2825c499401ece1cdd6a200afdade6bbea2ac0e4e26a"} Oct 30 20:15:02.580851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.578661 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" event={"ID":"78e70c72-9983-4b2e-bd0c-176f50822feb","Type":"ContainerStarted","Data":"0e304a1e368555bad9562efa9aada8f13c1f0803b4b9c444671dd8df9c1b508a"} Oct 30 20:15:02.612935 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.612878 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" event={"ID":"b8336ff5-7043-4f2a-8ab1-60c4713596e9","Type":"ContainerStarted","Data":"235a73b79e3091f0abcdf604c8709fcd7345c221104f8f722f1d5f18b86c7335"} Oct 30 20:15:02.638407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.630733 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:02.666439 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.642440 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" event={"ID":"8b69782b-78e9-4baf-8bdf-bcbf9b78767d","Type":"ContainerStarted","Data":"bb37d9963649d52075801471dacd988eb63052af2263c1e24756dbc0a599d976"} Oct 30 20:15:02.723114 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.714661 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" podStartSLOduration=2.714630842 podStartE2EDuration="2.714630842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.713950915 +0000 UTC m=+2622.409158875" watchObservedRunningTime="2026-10-30 20:15:02.714630842 +0000 UTC m=+2622.409838801" Oct 30 20:15:02.725913 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.725091 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" event={"ID":"67449cd5-2639-4625-b32e-ba3bec5a528b","Type":"ContainerStarted","Data":"b09ae333e8f53d095f5360e3616e7186fcc62ee98369e1eba69478010afe952a"} Oct 30 20:15:02.725913 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.725188 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" event={"ID":"67449cd5-2639-4625-b32e-ba3bec5a528b","Type":"ContainerStarted","Data":"b6af598d1312f72f70f2d3451153ac3858055b1fdb7b985bb52c384f088e1fe6"} Oct 30 20:15:02.727001 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.726950 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" podStartSLOduration=2.726933447 podStartE2EDuration="2.726933447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.678744965 +0000 UTC m=+2622.373952925" watchObservedRunningTime="2026-10-30 20:15:02.726933447 +0000 UTC m=+2622.422141406" Oct 30 20:15:02.731965 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.731902 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" event={"ID":"2ea189f9-f225-4fdb-8948-14cb53111cd5","Type":"ContainerStarted","Data":"3477becef01b5e48af6cb6cf5d5e453d19732979dd723e528eaf07bb8c9d9c91"} Oct 30 20:15:02.736480 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.735296 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" event={"ID":"755783df-2b63-4013-9133-d39c004e577f","Type":"ContainerStarted","Data":"77257cc958eb62b14128c68472477e03979c68936c15bd84a43e91ec9cda768d"} Oct 30 20:15:02.736480 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.735347 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" event={"ID":"755783df-2b63-4013-9133-d39c004e577f","Type":"ContainerStarted","Data":"edada6ee58e3fd9db1c59d1b4446e72994b732f41e24b63a5ea70cbe7c671e47"} Oct 30 20:15:02.740347 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.739783 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" event={"ID":"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c","Type":"ContainerStarted","Data":"5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397"} Oct 30 20:15:02.742362 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.742280 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" event={"ID":"2a5e04ce-a8ad-4b44-bd00-624a49388708","Type":"ContainerStarted","Data":"afb8f2284444e05a734dc8a16c3cd76ee7d8dff69cf1ec33228679c8d68fdb43"} Oct 30 20:15:02.755216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.753478 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" event={"ID":"3d37c05b-93b8-4c8c-84d1-9e186d7ec265","Type":"ContainerStarted","Data":"9cbef9b0df16f0636e3685caece18351100d9ac67b37e47f1af539eb9eab51a9"} Oct 30 20:15:02.759990 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.759962 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" event={"ID":"f93e44d7-d7d5-4beb-9332-403ddc9bf336","Type":"ContainerStarted","Data":"2082571e46ccdf5242037fc6e0592ec56e244065c0631c319fcb6c0ac622f2ae"} Oct 30 20:15:02.760910 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.760888 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" event={"ID":"f93e44d7-d7d5-4beb-9332-403ddc9bf336","Type":"ContainerStarted","Data":"ba39929e3b274036365e5726479a5d018be6991c5330f1dade6c632ac73275cf"} Oct 30 20:15:02.763019 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.762973 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" podStartSLOduration=2.762962641 podStartE2EDuration="2.762962641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.754569469 +0000 UTC m=+2622.449777427" watchObservedRunningTime="2026-10-30 20:15:02.762962641 +0000 UTC m=+2622.458170577" Oct 30 20:15:02.763194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.763165 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" event={"ID":"387559a4-9827-4a53-90ae-3f3f0852d25d","Type":"ContainerStarted","Data":"48c4b43dfded5e400e900d71a3dbaeaedb113521dc35ac4ca97a921f570c813b"} Oct 30 20:15:02.768808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.768769 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" event={"ID":"8fb65533-38aa-48c4-a374-fcf2f5e96f99","Type":"ContainerStarted","Data":"913a8e9a93d2a42871908c7629bcaf16068ca3eff9e6684236e911ab0cf99844"} Oct 30 20:15:02.769050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.769031 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" event={"ID":"4eafdae9-7fcd-4579-824b-60bf84bb9b7f","Type":"ContainerStarted","Data":"b45bf52a26b95fb23b7295ed0e00d7ad3ca78f459d4e9088f9e6e36c85ed883b"} Oct 30 20:15:02.771079 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.770974 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" event={"ID":"b7d9f0fd-4bb0-4773-8333-92f168e683c3","Type":"ContainerStarted","Data":"1ae08a3ca619d3c72e4eecfe6172f0a90cbfe89251b05e4988849b12b55b1843"} Oct 30 20:15:02.783077 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.783020 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" podStartSLOduration=2.783003238 podStartE2EDuration="2.783003238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.778739191 +0000 UTC m=+2622.473947148" watchObservedRunningTime="2026-10-30 20:15:02.783003238 +0000 UTC m=+2622.478211196" Oct 30 20:15:02.835819 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.835753 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" podStartSLOduration=2.835740678 podStartE2EDuration="2.835740678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.810372024 +0000 UTC m=+2622.505579984" watchObservedRunningTime="2026-10-30 20:15:02.835740678 +0000 UTC m=+2622.530948654" Oct 30 20:15:02.840969 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.840913 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" podStartSLOduration=2.840901413 podStartE2EDuration="2.840901413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.83524118 +0000 UTC m=+2622.530449139" watchObservedRunningTime="2026-10-30 20:15:02.840901413 +0000 UTC m=+2622.536109371" Oct 30 20:15:02.913681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.913619 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" podStartSLOduration=2.913605009 podStartE2EDuration="2.913605009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.881312656 +0000 UTC m=+2622.576520615" watchObservedRunningTime="2026-10-30 20:15:02.913605009 +0000 UTC m=+2622.608812967" Oct 30 20:15:02.916636 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.916592 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" podStartSLOduration=2.916580465 podStartE2EDuration="2.916580465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.912829614 +0000 UTC m=+2622.608037572" watchObservedRunningTime="2026-10-30 20:15:02.916580465 +0000 UTC m=+2622.611788423" Oct 30 20:15:02.937323 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:02.937278 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" podStartSLOduration=2.937265984 podStartE2EDuration="2.937265984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:02.934941089 +0000 UTC m=+2622.630149047" watchObservedRunningTime="2026-10-30 20:15:02.937265984 +0000 UTC m=+2622.632473942" Oct 30 20:15:03.109818 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.109791 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:15:03.199267 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.195210 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efa6cde-c023-40c7-a04b-a73b1605e9c9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4efa6cde-c023-40c7-a04b-a73b1605e9c9-provider-credentials\") pod \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " Oct 30 20:15:03.199267 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.195390 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4efa6cde-c023-40c7-a04b-a73b1605e9c9-kube-api-access-999vt\" (UniqueName: \"kubernetes.io/projected/4efa6cde-c023-40c7-a04b-a73b1605e9c9-kube-api-access-999vt\") pod \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " Oct 30 20:15:03.199267 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.195494 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4efa6cde-c023-40c7-a04b-a73b1605e9c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efa6cde-c023-40c7-a04b-a73b1605e9c9-plugins-config-volume\") pod \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " Oct 30 20:15:03.199267 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.195665 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4efa6cde-c023-40c7-a04b-a73b1605e9c9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4efa6cde-c023-40c7-a04b-a73b1605e9c9-routing-overlay-volume\") pod \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\" (UID: \"4efa6cde-c023-40c7-a04b-a73b1605e9c9\") " Oct 30 20:15:03.199267 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.196283 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efa6cde-c023-40c7-a04b-a73b1605e9c9-plugins-config-volume" pod "4efa6cde-c023-40c7-a04b-a73b1605e9c9" (UID: "4efa6cde-c023-40c7-a04b-a73b1605e9c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:03.199267 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.196495 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4efa6cde-c023-40c7-a04b-a73b1605e9c9-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:03.199267 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.198249 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efa6cde-c023-40c7-a04b-a73b1605e9c9-provider-credentials" pod "4efa6cde-c023-40c7-a04b-a73b1605e9c9" (UID: "4efa6cde-c023-40c7-a04b-a73b1605e9c9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:03.199830 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.199810 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4efa6cde-c023-40c7-a04b-a73b1605e9c9-kube-api-access-999vt" pod "4efa6cde-c023-40c7-a04b-a73b1605e9c9" (UID: "4efa6cde-c023-40c7-a04b-a73b1605e9c9"). InnerVolumeSpecName "kube-api-access-999vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:03.258964 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.258912 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4efa6cde-c023-40c7-a04b-a73b1605e9c9-routing-overlay-volume" pod "4efa6cde-c023-40c7-a04b-a73b1605e9c9" (UID: "4efa6cde-c023-40c7-a04b-a73b1605e9c9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:03.297602 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.297567 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4efa6cde-c023-40c7-a04b-a73b1605e9c9-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:03.297602 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.297603 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4efa6cde-c023-40c7-a04b-a73b1605e9c9-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:03.297751 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.297613 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-999vt\" (UniqueName: \"kubernetes.io/projected/4efa6cde-c023-40c7-a04b-a73b1605e9c9-kube-api-access-999vt\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:03.726014 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.725886 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:03.785794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.785759 5091 generic.go:356] "Generic (PLEG): container finished" podID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" containerID="18365f83bfa0faf17b8fd049552b760882220df63095872e4ac180cc41cc15e5" exitCode=60 Oct 30 20:15:03.785977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.785873 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" event={"ID":"4eafdae9-7fcd-4579-824b-60bf84bb9b7f","Type":"ContainerDied","Data":"18365f83bfa0faf17b8fd049552b760882220df63095872e4ac180cc41cc15e5"} Oct 30 20:15:03.786158 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.786124 5091 scope.go:122] "RemoveContainer" containerID="18365f83bfa0faf17b8fd049552b760882220df63095872e4ac180cc41cc15e5" Oct 30 20:15:03.792623 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.792582 5091 generic.go:356] "Generic (PLEG): container finished" podID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" containerID="a58c8d9816ab1431997347ea673a58f89c87defdaef19ff0ecb0d3e03d657d2c" exitCode=60 Oct 30 20:15:03.792740 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.792713 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" event={"ID":"b7d9f0fd-4bb0-4773-8333-92f168e683c3","Type":"ContainerDied","Data":"a58c8d9816ab1431997347ea673a58f89c87defdaef19ff0ecb0d3e03d657d2c"} Oct 30 20:15:03.792967 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.792950 5091 scope.go:122] "RemoveContainer" containerID="a58c8d9816ab1431997347ea673a58f89c87defdaef19ff0ecb0d3e03d657d2c" Oct 30 20:15:03.795677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.795655 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" event={"ID":"d27d65d4-ebb6-4254-89c9-febcacbdf9f1","Type":"ContainerStarted","Data":"d2acfc31eda0f9cd25bd1af1d0f4b592dbf9006c52b6d04062a44cdcb3afa191"} Oct 30 20:15:03.799597 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.799575 5091 generic.go:356] "Generic (PLEG): container finished" podID="4efa6cde-c023-40c7-a04b-a73b1605e9c9" containerID="dbd3fa913e10304756a70c3ef7fc22861b674090864e442ff410993f69e8d049" exitCode=0 Oct 30 20:15:03.799709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.799673 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" event={"ID":"4efa6cde-c023-40c7-a04b-a73b1605e9c9","Type":"ContainerDied","Data":"dbd3fa913e10304756a70c3ef7fc22861b674090864e442ff410993f69e8d049"} Oct 30 20:15:03.799763 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.799714 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" event={"ID":"4efa6cde-c023-40c7-a04b-a73b1605e9c9","Type":"ContainerDied","Data":"ca5c5fb1fddac6c47787207b375c408bb145c3dccf97a5fc5977aafb5b7400c0"} Oct 30 20:15:03.799763 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.799677 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6976477d7d-88x4l" Oct 30 20:15:03.799763 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.799741 5091 scope.go:122] "RemoveContainer" containerID="dbd3fa913e10304756a70c3ef7fc22861b674090864e442ff410993f69e8d049" Oct 30 20:15:03.803444 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.803421 5091 generic.go:356] "Generic (PLEG): container finished" podID="ee32ddbf-9640-4efd-b2a5-babcfb306088" containerID="d40c8d8a897a6afffb0c68906d8650fb3167cfc98b56e6f2589fce4ebba729ac" exitCode=60 Oct 30 20:15:03.803687 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.803522 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" event={"ID":"ee32ddbf-9640-4efd-b2a5-babcfb306088","Type":"ContainerDied","Data":"d40c8d8a897a6afffb0c68906d8650fb3167cfc98b56e6f2589fce4ebba729ac"} Oct 30 20:15:03.803973 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.803786 5091 scope.go:122] "RemoveContainer" containerID="d40c8d8a897a6afffb0c68906d8650fb3167cfc98b56e6f2589fce4ebba729ac" Oct 30 20:15:03.811780 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.811756 5091 generic.go:356] "Generic (PLEG): container finished" podID="87e33682-f349-450b-ab65-1b60d304a6cf" containerID="b0d1ecbd65b35d1a19f8105c6105db3e7dd0eb48a0de8a8ec8bb38c0d51a28f6" exitCode=60 Oct 30 20:15:03.812423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.811921 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" event={"ID":"87e33682-f349-450b-ab65-1b60d304a6cf","Type":"ContainerDied","Data":"b0d1ecbd65b35d1a19f8105c6105db3e7dd0eb48a0de8a8ec8bb38c0d51a28f6"} Oct 30 20:15:03.812528 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.812460 5091 scope.go:122] "RemoveContainer" containerID="b0d1ecbd65b35d1a19f8105c6105db3e7dd0eb48a0de8a8ec8bb38c0d51a28f6" Oct 30 20:15:03.814582 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.814558 5091 generic.go:356] "Generic (PLEG): container finished" podID="78e70c72-9983-4b2e-bd0c-176f50822feb" containerID="0e304a1e368555bad9562efa9aada8f13c1f0803b4b9c444671dd8df9c1b508a" exitCode=60 Oct 30 20:15:03.814696 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.814637 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" event={"ID":"78e70c72-9983-4b2e-bd0c-176f50822feb","Type":"ContainerDied","Data":"0e304a1e368555bad9562efa9aada8f13c1f0803b4b9c444671dd8df9c1b508a"} Oct 30 20:15:03.814834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.814816 5091 scope.go:122] "RemoveContainer" containerID="0e304a1e368555bad9562efa9aada8f13c1f0803b4b9c444671dd8df9c1b508a" Oct 30 20:15:03.816559 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.816537 5091 generic.go:356] "Generic (PLEG): container finished" podID="b8336ff5-7043-4f2a-8ab1-60c4713596e9" containerID="3c1a45fc95a1afae10633d81e9b61cc00194472c5ca8a73b1d115f444ee23485" exitCode=60 Oct 30 20:15:03.816657 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.816622 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" event={"ID":"b8336ff5-7043-4f2a-8ab1-60c4713596e9","Type":"ContainerDied","Data":"3c1a45fc95a1afae10633d81e9b61cc00194472c5ca8a73b1d115f444ee23485"} Oct 30 20:15:03.816797 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.816778 5091 scope.go:122] "RemoveContainer" containerID="3c1a45fc95a1afae10633d81e9b61cc00194472c5ca8a73b1d115f444ee23485" Oct 30 20:15:03.823186 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.823094 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" event={"ID":"8b69782b-78e9-4baf-8bdf-bcbf9b78767d","Type":"ContainerStarted","Data":"6982059c208300e3100c40cadbbe5b3ef83236036bee49b0d077cad4aa200e6a"} Oct 30 20:15:03.825062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.825041 5091 generic.go:356] "Generic (PLEG): container finished" podID="67449cd5-2639-4625-b32e-ba3bec5a528b" containerID="b09ae333e8f53d095f5360e3616e7186fcc62ee98369e1eba69478010afe952a" exitCode=60 Oct 30 20:15:03.825172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.825121 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" event={"ID":"67449cd5-2639-4625-b32e-ba3bec5a528b","Type":"ContainerDied","Data":"b09ae333e8f53d095f5360e3616e7186fcc62ee98369e1eba69478010afe952a"} Oct 30 20:15:03.825362 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.825343 5091 scope.go:122] "RemoveContainer" containerID="b09ae333e8f53d095f5360e3616e7186fcc62ee98369e1eba69478010afe952a" Oct 30 20:15:03.826917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.826893 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" event={"ID":"2ea189f9-f225-4fdb-8948-14cb53111cd5","Type":"ContainerStarted","Data":"b1ebfdeb8724292516483c19ef7846c802777dfe97f5716c9e59468348670ece"} Oct 30 20:15:03.829652 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.828201 5091 scope.go:122] "RemoveContainer" containerID="dbd3fa913e10304756a70c3ef7fc22861b674090864e442ff410993f69e8d049" Oct 30 20:15:03.829652 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:03.828562 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd3fa913e10304756a70c3ef7fc22861b674090864e442ff410993f69e8d049\": container with ID starting with dbd3fa913e10304756a70c3ef7fc22861b674090864e442ff410993f69e8d049 not found: ID does not exist" containerID="dbd3fa913e10304756a70c3ef7fc22861b674090864e442ff410993f69e8d049" Oct 30 20:15:03.829652 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.828595 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd3fa913e10304756a70c3ef7fc22861b674090864e442ff410993f69e8d049"} err="failed to get container status \"dbd3fa913e10304756a70c3ef7fc22861b674090864e442ff410993f69e8d049\": rpc error: code = NotFound desc = could not find container \"dbd3fa913e10304756a70c3ef7fc22861b674090864e442ff410993f69e8d049\": container with ID starting with dbd3fa913e10304756a70c3ef7fc22861b674090864e442ff410993f69e8d049 not found: ID does not exist" Oct 30 20:15:03.829652 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.828824 5091 generic.go:356] "Generic (PLEG): container finished" podID="755783df-2b63-4013-9133-d39c004e577f" containerID="77257cc958eb62b14128c68472477e03979c68936c15bd84a43e91ec9cda768d" exitCode=60 Oct 30 20:15:03.829652 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.828882 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" event={"ID":"755783df-2b63-4013-9133-d39c004e577f","Type":"ContainerDied","Data":"77257cc958eb62b14128c68472477e03979c68936c15bd84a43e91ec9cda768d"} Oct 30 20:15:03.829652 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.829104 5091 scope.go:122] "RemoveContainer" containerID="77257cc958eb62b14128c68472477e03979c68936c15bd84a43e91ec9cda768d" Oct 30 20:15:03.839037 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.836764 5091 generic.go:356] "Generic (PLEG): container finished" podID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" containerID="f37f99519913ab4b604e0a8239612fa6ac44d8f2d14f603ab717b658d46c593c" exitCode=60 Oct 30 20:15:03.839037 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.836918 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" event={"ID":"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c","Type":"ContainerDied","Data":"f37f99519913ab4b604e0a8239612fa6ac44d8f2d14f603ab717b658d46c593c"} Oct 30 20:15:03.839037 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.838047 5091 scope.go:122] "RemoveContainer" containerID="f37f99519913ab4b604e0a8239612fa6ac44d8f2d14f603ab717b658d46c593c" Oct 30 20:15:03.842299 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.842118 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" event={"ID":"2a5e04ce-a8ad-4b44-bd00-624a49388708","Type":"ContainerStarted","Data":"daad74b112bde15bcf86632dedeb6f294f827b3a688a9d3ae2b634902f57dfcf"} Oct 30 20:15:03.844075 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.844056 5091 generic.go:356] "Generic (PLEG): container finished" podID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" containerID="73ba2056bf897ea06312ecd767cce3b4677471344853cf751c822c1fc1aa2e4c" exitCode=60 Oct 30 20:15:03.844354 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.844334 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" event={"ID":"3d37c05b-93b8-4c8c-84d1-9e186d7ec265","Type":"ContainerDied","Data":"73ba2056bf897ea06312ecd767cce3b4677471344853cf751c822c1fc1aa2e4c"} Oct 30 20:15:03.845234 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.845213 5091 scope.go:122] "RemoveContainer" containerID="73ba2056bf897ea06312ecd767cce3b4677471344853cf751c822c1fc1aa2e4c" Oct 30 20:15:03.850259 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.850232 5091 generic.go:356] "Generic (PLEG): container finished" podID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" containerID="2082571e46ccdf5242037fc6e0592ec56e244065c0631c319fcb6c0ac622f2ae" exitCode=60 Oct 30 20:15:03.850367 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.850343 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" event={"ID":"f93e44d7-d7d5-4beb-9332-403ddc9bf336","Type":"ContainerDied","Data":"2082571e46ccdf5242037fc6e0592ec56e244065c0631c319fcb6c0ac622f2ae"} Oct 30 20:15:03.850794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.850768 5091 scope.go:122] "RemoveContainer" containerID="2082571e46ccdf5242037fc6e0592ec56e244065c0631c319fcb6c0ac622f2ae" Oct 30 20:15:03.852681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.852655 5091 generic.go:356] "Generic (PLEG): container finished" podID="387559a4-9827-4a53-90ae-3f3f0852d25d" containerID="2d9798b1e40e0f88987d69326ed2e9945fcfe004e2fa7c0e7150b95ae7a3ddad" exitCode=60 Oct 30 20:15:03.852796 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.852780 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" event={"ID":"387559a4-9827-4a53-90ae-3f3f0852d25d","Type":"ContainerDied","Data":"2d9798b1e40e0f88987d69326ed2e9945fcfe004e2fa7c0e7150b95ae7a3ddad"} Oct 30 20:15:03.853934 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.853525 5091 scope.go:122] "RemoveContainer" containerID="2d9798b1e40e0f88987d69326ed2e9945fcfe004e2fa7c0e7150b95ae7a3ddad" Oct 30 20:15:03.855863 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.855840 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" event={"ID":"8fb65533-38aa-48c4-a374-fcf2f5e96f99","Type":"ContainerStarted","Data":"89f332143866d1718cf46a57c8fb7554759c3efe76bd70ef2a177ad661017a85"} Oct 30 20:15:03.881960 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.881835 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" podStartSLOduration=3.881818101 podStartE2EDuration="3.881818101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.880633884 +0000 UTC m=+2623.575841842" watchObservedRunningTime="2026-10-30 20:15:03.881818101 +0000 UTC m=+2623.577026060" Oct 30 20:15:03.916528 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:03.916480 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" podStartSLOduration=3.916466381 podStartE2EDuration="3.916466381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:03.912500993 +0000 UTC m=+2623.607708954" watchObservedRunningTime="2026-10-30 20:15:03.916466381 +0000 UTC m=+2623.611674339" Oct 30 20:15:04.012363 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.012313 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" podStartSLOduration=4.012296021 podStartE2EDuration="4.012296021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.008804345 +0000 UTC m=+2623.704012303" watchObservedRunningTime="2026-10-30 20:15:04.012296021 +0000 UTC m=+2623.707503978" Oct 30 20:15:04.029586 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.029556 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6976477d7d-88x4l"] Oct 30 20:15:04.037182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.037128 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6976477d7d-88x4l"] Oct 30 20:15:04.043094 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.043072 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29"] Oct 30 20:15:04.054594 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.053293 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce5f3aef-j2r29"] Oct 30 20:15:04.151448 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.151362 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" podStartSLOduration=4.151349998 podStartE2EDuration="4.151349998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.150811555 +0000 UTC m=+2623.846019514" watchObservedRunningTime="2026-10-30 20:15:04.151349998 +0000 UTC m=+2623.846557955" Oct 30 20:15:04.201109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.201056 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" podStartSLOduration=4.201041095 podStartE2EDuration="4.201041095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.199183362 +0000 UTC m=+2623.894391322" watchObservedRunningTime="2026-10-30 20:15:04.201041095 +0000 UTC m=+2623.896249092" Oct 30 20:15:04.444379 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.444307 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1af5285a-eea9-4f07-a464-7a3925cd78a7" path="/var/lib/kubelet/pods/1af5285a-eea9-4f07-a464-7a3925cd78a7/volumes" Oct 30 20:15:04.444787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.444772 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4efa6cde-c023-40c7-a04b-a73b1605e9c9" path="/var/lib/kubelet/pods/4efa6cde-c023-40c7-a04b-a73b1605e9c9/volumes" Oct 30 20:15:04.708946 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.703692 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:04.874382 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.874344 5091 generic.go:356] "Generic (PLEG): container finished" podID="2ea189f9-f225-4fdb-8948-14cb53111cd5" containerID="b1ebfdeb8724292516483c19ef7846c802777dfe97f5716c9e59468348670ece" exitCode=60 Oct 30 20:15:04.874904 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.874486 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" event={"ID":"2ea189f9-f225-4fdb-8948-14cb53111cd5","Type":"ContainerDied","Data":"b1ebfdeb8724292516483c19ef7846c802777dfe97f5716c9e59468348670ece"} Oct 30 20:15:04.877200 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.875170 5091 scope.go:122] "RemoveContainer" containerID="b1ebfdeb8724292516483c19ef7846c802777dfe97f5716c9e59468348670ece" Oct 30 20:15:04.882720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.882516 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" event={"ID":"755783df-2b63-4013-9133-d39c004e577f","Type":"ContainerStarted","Data":"d518b80f9487d20c1e27524014d8510b3151c85add30065fdc46eeeb6e569d83"} Oct 30 20:15:04.896964 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.895915 5091 generic.go:356] "Generic (PLEG): container finished" podID="2a5e04ce-a8ad-4b44-bd00-624a49388708" containerID="daad74b112bde15bcf86632dedeb6f294f827b3a688a9d3ae2b634902f57dfcf" exitCode=60 Oct 30 20:15:04.896964 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.896033 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" event={"ID":"2a5e04ce-a8ad-4b44-bd00-624a49388708","Type":"ContainerDied","Data":"daad74b112bde15bcf86632dedeb6f294f827b3a688a9d3ae2b634902f57dfcf"} Oct 30 20:15:04.897180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.897133 5091 scope.go:122] "RemoveContainer" containerID="daad74b112bde15bcf86632dedeb6f294f827b3a688a9d3ae2b634902f57dfcf" Oct 30 20:15:04.900026 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.899999 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" event={"ID":"387559a4-9827-4a53-90ae-3f3f0852d25d","Type":"ContainerStarted","Data":"f70ce32f693cb1edbfd8e1cc46058eb5803e02afbba4b84364824720004ef466"} Oct 30 20:15:04.903529 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.903504 5091 generic.go:356] "Generic (PLEG): container finished" podID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" containerID="89f332143866d1718cf46a57c8fb7554759c3efe76bd70ef2a177ad661017a85" exitCode=60 Oct 30 20:15:04.903654 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.903599 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" event={"ID":"8fb65533-38aa-48c4-a374-fcf2f5e96f99","Type":"ContainerDied","Data":"89f332143866d1718cf46a57c8fb7554759c3efe76bd70ef2a177ad661017a85"} Oct 30 20:15:04.904073 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.904046 5091 scope.go:122] "RemoveContainer" containerID="89f332143866d1718cf46a57c8fb7554759c3efe76bd70ef2a177ad661017a85" Oct 30 20:15:04.913417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.913389 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" event={"ID":"4eafdae9-7fcd-4579-824b-60bf84bb9b7f","Type":"ContainerStarted","Data":"250846c9f7cae177aa121adc9786a0544afb17d70a28204ec5e72f9caec9b3a9"} Oct 30 20:15:04.925898 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.925871 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" event={"ID":"b7d9f0fd-4bb0-4773-8333-92f168e683c3","Type":"ContainerStarted","Data":"101fe9c74699ac0e0a5b6c9ae289c320559cd9933c85443cdfc059f2ef180b01"} Oct 30 20:15:04.930490 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.930466 5091 generic.go:356] "Generic (PLEG): container finished" podID="d27d65d4-ebb6-4254-89c9-febcacbdf9f1" containerID="d2acfc31eda0f9cd25bd1af1d0f4b592dbf9006c52b6d04062a44cdcb3afa191" exitCode=60 Oct 30 20:15:04.930783 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.930763 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" event={"ID":"d27d65d4-ebb6-4254-89c9-febcacbdf9f1","Type":"ContainerDied","Data":"d2acfc31eda0f9cd25bd1af1d0f4b592dbf9006c52b6d04062a44cdcb3afa191"} Oct 30 20:15:04.930924 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.930821 5091 scope.go:122] "RemoveContainer" containerID="d2acfc31eda0f9cd25bd1af1d0f4b592dbf9006c52b6d04062a44cdcb3afa191" Oct 30 20:15:04.931222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.931200 5091 scope.go:122] "RemoveContainer" containerID="83d4558fe4ed31850e31b3a0edcb12ac071e197ab71e93d2de3aa045946186c7" Oct 30 20:15:04.931489 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:04.931452 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd_odh-ai-gateway-infra(d27d65d4-ebb6-4254-89c9-febcacbdf9f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" podUID="d27d65d4-ebb6-4254-89c9-febcacbdf9f1" Oct 30 20:15:04.936542 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.936515 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" event={"ID":"ee32ddbf-9640-4efd-b2a5-babcfb306088","Type":"ContainerStarted","Data":"5d0ca013a49dbf4aa2b7c2101ce60a5700645908e822083179cc5aba34f37e53"} Oct 30 20:15:04.939348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.939324 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" event={"ID":"87e33682-f349-450b-ab65-1b60d304a6cf","Type":"ContainerStarted","Data":"0d8d851af0c3a1086af686185eca57a61976cd4e46c92f1e78368f64360f6da0"} Oct 30 20:15:04.941491 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.941470 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" event={"ID":"78e70c72-9983-4b2e-bd0c-176f50822feb","Type":"ContainerStarted","Data":"6cc8aebd7337da21011cb107130eca75c7efe1683ced5f889ee2bffaf1154647"} Oct 30 20:15:04.944028 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.944006 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" event={"ID":"b8336ff5-7043-4f2a-8ab1-60c4713596e9","Type":"ContainerStarted","Data":"be87937e01bde7501599241c9bedbae8c14f3d13ec6dc0219ed302e5a34e3c51"} Oct 30 20:15:04.959536 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.959501 5091 generic.go:356] "Generic (PLEG): container finished" podID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" containerID="6982059c208300e3100c40cadbbe5b3ef83236036bee49b0d077cad4aa200e6a" exitCode=60 Oct 30 20:15:04.959651 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.959595 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" event={"ID":"8b69782b-78e9-4baf-8bdf-bcbf9b78767d","Type":"ContainerDied","Data":"6982059c208300e3100c40cadbbe5b3ef83236036bee49b0d077cad4aa200e6a"} Oct 30 20:15:04.959903 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.959885 5091 scope.go:122] "RemoveContainer" containerID="6982059c208300e3100c40cadbbe5b3ef83236036bee49b0d077cad4aa200e6a" Oct 30 20:15:04.969740 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.966920 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" event={"ID":"67449cd5-2639-4625-b32e-ba3bec5a528b","Type":"ContainerStarted","Data":"4359cdb70a9f6f96a6015f9a4602aafda71910976fac991ab52ea927faef2ad4"} Oct 30 20:15:04.998314 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:04.998267 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" podStartSLOduration=4.998247911 podStartE2EDuration="4.998247911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:04.99052247 +0000 UTC m=+2624.685730428" watchObservedRunningTime="2026-10-30 20:15:04.998247911 +0000 UTC m=+2624.693455871" Oct 30 20:15:05.021912 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.021852 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" podStartSLOduration=5.021837236 podStartE2EDuration="5.021837236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:05.018378937 +0000 UTC m=+2624.713586897" watchObservedRunningTime="2026-10-30 20:15:05.021837236 +0000 UTC m=+2624.717045194" Oct 30 20:15:05.187756 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.187706 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" podStartSLOduration=5.187693572 podStartE2EDuration="5.187693572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:05.186779231 +0000 UTC m=+2624.881987200" watchObservedRunningTime="2026-10-30 20:15:05.187693572 +0000 UTC m=+2624.882901530" Oct 30 20:15:05.824720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.824581 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:05.978133 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.978099 5091 generic.go:356] "Generic (PLEG): container finished" podID="78e70c72-9983-4b2e-bd0c-176f50822feb" containerID="6cc8aebd7337da21011cb107130eca75c7efe1683ced5f889ee2bffaf1154647" exitCode=60 Oct 30 20:15:05.978562 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.978178 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" event={"ID":"78e70c72-9983-4b2e-bd0c-176f50822feb","Type":"ContainerDied","Data":"6cc8aebd7337da21011cb107130eca75c7efe1683ced5f889ee2bffaf1154647"} Oct 30 20:15:05.978562 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.978257 5091 scope.go:122] "RemoveContainer" containerID="0e304a1e368555bad9562efa9aada8f13c1f0803b4b9c444671dd8df9c1b508a" Oct 30 20:15:05.978562 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.978512 5091 scope.go:122] "RemoveContainer" containerID="6cc8aebd7337da21011cb107130eca75c7efe1683ced5f889ee2bffaf1154647" Oct 30 20:15:05.978920 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:05.978890 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d_odh-ai-gateway-infra(78e70c72-9983-4b2e-bd0c-176f50822feb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" podUID="78e70c72-9983-4b2e-bd0c-176f50822feb" Oct 30 20:15:05.980922 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.980896 5091 generic.go:356] "Generic (PLEG): container finished" podID="b8336ff5-7043-4f2a-8ab1-60c4713596e9" containerID="be87937e01bde7501599241c9bedbae8c14f3d13ec6dc0219ed302e5a34e3c51" exitCode=60 Oct 30 20:15:05.981045 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.980980 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" event={"ID":"b8336ff5-7043-4f2a-8ab1-60c4713596e9","Type":"ContainerDied","Data":"be87937e01bde7501599241c9bedbae8c14f3d13ec6dc0219ed302e5a34e3c51"} Oct 30 20:15:05.981250 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.981234 5091 scope.go:122] "RemoveContainer" containerID="be87937e01bde7501599241c9bedbae8c14f3d13ec6dc0219ed302e5a34e3c51" Oct 30 20:15:05.981577 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:05.981551 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p_odh-ai-gateway-infra(b8336ff5-7043-4f2a-8ab1-60c4713596e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" podUID="b8336ff5-7043-4f2a-8ab1-60c4713596e9" Oct 30 20:15:05.983431 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.983406 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" event={"ID":"8b69782b-78e9-4baf-8bdf-bcbf9b78767d","Type":"ContainerStarted","Data":"b6b778b10292809aa1ffd6171d6c4a597bb04eb9e6eb69d3ec516e7c1bc1f83f"} Oct 30 20:15:05.986034 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.985880 5091 generic.go:356] "Generic (PLEG): container finished" podID="67449cd5-2639-4625-b32e-ba3bec5a528b" containerID="4359cdb70a9f6f96a6015f9a4602aafda71910976fac991ab52ea927faef2ad4" exitCode=60 Oct 30 20:15:05.986034 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.986003 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" event={"ID":"67449cd5-2639-4625-b32e-ba3bec5a528b","Type":"ContainerDied","Data":"4359cdb70a9f6f96a6015f9a4602aafda71910976fac991ab52ea927faef2ad4"} Oct 30 20:15:05.986220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.986196 5091 scope.go:122] "RemoveContainer" containerID="4359cdb70a9f6f96a6015f9a4602aafda71910976fac991ab52ea927faef2ad4" Oct 30 20:15:05.986611 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:05.986587 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v_odh-ai-gateway-infra(67449cd5-2639-4625-b32e-ba3bec5a528b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" podUID="67449cd5-2639-4625-b32e-ba3bec5a528b" Oct 30 20:15:05.988158 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.988122 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" event={"ID":"2ea189f9-f225-4fdb-8948-14cb53111cd5","Type":"ContainerStarted","Data":"4d3ca8b3f178e9621270db788279994ba03ec76906628e6c0bb3a6194db437bd"} Oct 30 20:15:05.990017 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.989992 5091 generic.go:356] "Generic (PLEG): container finished" podID="755783df-2b63-4013-9133-d39c004e577f" containerID="d518b80f9487d20c1e27524014d8510b3151c85add30065fdc46eeeb6e569d83" exitCode=60 Oct 30 20:15:05.990119 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.990084 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" event={"ID":"755783df-2b63-4013-9133-d39c004e577f","Type":"ContainerDied","Data":"d518b80f9487d20c1e27524014d8510b3151c85add30065fdc46eeeb6e569d83"} Oct 30 20:15:05.990356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.990340 5091 scope.go:122] "RemoveContainer" containerID="d518b80f9487d20c1e27524014d8510b3151c85add30065fdc46eeeb6e569d83" Oct 30 20:15:05.990610 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:05.990591 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl_odh-ai-gateway-infra(755783df-2b63-4013-9133-d39c004e577f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" podUID="755783df-2b63-4013-9133-d39c004e577f" Oct 30 20:15:05.992188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.992165 5091 generic.go:356] "Generic (PLEG): container finished" podID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" containerID="1767ef190a7b5110d5c2750c39245946fbc6f170526642fcb77b73cc0b616476" exitCode=60 Oct 30 20:15:05.992301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.992241 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" event={"ID":"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c","Type":"ContainerDied","Data":"1767ef190a7b5110d5c2750c39245946fbc6f170526642fcb77b73cc0b616476"} Oct 30 20:15:05.992477 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.992457 5091 scope.go:122] "RemoveContainer" containerID="1767ef190a7b5110d5c2750c39245946fbc6f170526642fcb77b73cc0b616476" Oct 30 20:15:05.992739 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:05.992715 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz_odh-ai-gateway-infra(1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" podUID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" Oct 30 20:15:05.994545 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.994525 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" event={"ID":"2a5e04ce-a8ad-4b44-bd00-624a49388708","Type":"ContainerStarted","Data":"9fd0a20b190e89a6cb45bdf7ee6d4597c4c45e9526b5b4f862b01f67833f4416"} Oct 30 20:15:05.997097 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.997058 5091 generic.go:356] "Generic (PLEG): container finished" podID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" containerID="228ce452fc244df668d0698165baecf6054ce4822ad94a73865a51e121000d09" exitCode=60 Oct 30 20:15:05.997200 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.997129 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" event={"ID":"3d37c05b-93b8-4c8c-84d1-9e186d7ec265","Type":"ContainerDied","Data":"228ce452fc244df668d0698165baecf6054ce4822ad94a73865a51e121000d09"} Oct 30 20:15:05.998597 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:05.997412 5091 scope.go:122] "RemoveContainer" containerID="228ce452fc244df668d0698165baecf6054ce4822ad94a73865a51e121000d09" Oct 30 20:15:05.998597 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:05.997658 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7_odh-ai-gateway-infra(3d37c05b-93b8-4c8c-84d1-9e186d7ec265)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" podUID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" Oct 30 20:15:06.003693 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.003669 5091 generic.go:356] "Generic (PLEG): container finished" podID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" containerID="5bcc7b29b6360ff50c0212de016b853d5c37197d5dbde8921e8f7fb4a00ef8fc" exitCode=60 Oct 30 20:15:06.003806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.003708 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" event={"ID":"f93e44d7-d7d5-4beb-9332-403ddc9bf336","Type":"ContainerDied","Data":"5bcc7b29b6360ff50c0212de016b853d5c37197d5dbde8921e8f7fb4a00ef8fc"} Oct 30 20:15:06.003991 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.003970 5091 scope.go:122] "RemoveContainer" containerID="5bcc7b29b6360ff50c0212de016b853d5c37197d5dbde8921e8f7fb4a00ef8fc" Oct 30 20:15:06.004311 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:06.004284 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc_odh-ai-gateway-infra(f93e44d7-d7d5-4beb-9332-403ddc9bf336)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" podUID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" Oct 30 20:15:06.005952 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.005931 5091 generic.go:356] "Generic (PLEG): container finished" podID="387559a4-9827-4a53-90ae-3f3f0852d25d" containerID="f70ce32f693cb1edbfd8e1cc46058eb5803e02afbba4b84364824720004ef466" exitCode=60 Oct 30 20:15:06.006053 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.006023 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" event={"ID":"387559a4-9827-4a53-90ae-3f3f0852d25d","Type":"ContainerDied","Data":"f70ce32f693cb1edbfd8e1cc46058eb5803e02afbba4b84364824720004ef466"} Oct 30 20:15:06.006279 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.006261 5091 scope.go:122] "RemoveContainer" containerID="f70ce32f693cb1edbfd8e1cc46058eb5803e02afbba4b84364824720004ef466" Oct 30 20:15:06.006518 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:06.006498 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct_odh-ai-gateway-infra(387559a4-9827-4a53-90ae-3f3f0852d25d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" podUID="387559a4-9827-4a53-90ae-3f3f0852d25d" Oct 30 20:15:06.008290 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.008267 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" event={"ID":"8fb65533-38aa-48c4-a374-fcf2f5e96f99","Type":"ContainerStarted","Data":"b3b874c4b57483016809f6f6bf50016c6a56399d516ff31b0615f3425d9dd39b"} Oct 30 20:15:06.010377 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.010354 5091 generic.go:356] "Generic (PLEG): container finished" podID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" containerID="250846c9f7cae177aa121adc9786a0544afb17d70a28204ec5e72f9caec9b3a9" exitCode=60 Oct 30 20:15:06.010479 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.010466 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" event={"ID":"4eafdae9-7fcd-4579-824b-60bf84bb9b7f","Type":"ContainerDied","Data":"250846c9f7cae177aa121adc9786a0544afb17d70a28204ec5e72f9caec9b3a9"} Oct 30 20:15:06.010681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.010650 5091 scope.go:122] "RemoveContainer" containerID="250846c9f7cae177aa121adc9786a0544afb17d70a28204ec5e72f9caec9b3a9" Oct 30 20:15:06.010930 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:06.010894 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28_odh-ai-gateway-infra(4eafdae9-7fcd-4579-824b-60bf84bb9b7f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" podUID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" Oct 30 20:15:06.012742 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.012715 5091 generic.go:356] "Generic (PLEG): container finished" podID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" containerID="101fe9c74699ac0e0a5b6c9ae289c320559cd9933c85443cdfc059f2ef180b01" exitCode=60 Oct 30 20:15:06.012855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.012796 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" event={"ID":"b7d9f0fd-4bb0-4773-8333-92f168e683c3","Type":"ContainerDied","Data":"101fe9c74699ac0e0a5b6c9ae289c320559cd9933c85443cdfc059f2ef180b01"} Oct 30 20:15:06.012984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.012963 5091 scope.go:122] "RemoveContainer" containerID="101fe9c74699ac0e0a5b6c9ae289c320559cd9933c85443cdfc059f2ef180b01" Oct 30 20:15:06.013255 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:06.013231 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb_odh-ai-gateway-infra(b7d9f0fd-4bb0-4773-8333-92f168e683c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" podUID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" Oct 30 20:15:06.014866 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.014844 5091 scope.go:122] "RemoveContainer" containerID="d2acfc31eda0f9cd25bd1af1d0f4b592dbf9006c52b6d04062a44cdcb3afa191" Oct 30 20:15:06.015103 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:06.015081 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd_odh-ai-gateway-infra(d27d65d4-ebb6-4254-89c9-febcacbdf9f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" podUID="d27d65d4-ebb6-4254-89c9-febcacbdf9f1" Oct 30 20:15:06.017240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.016768 5091 generic.go:356] "Generic (PLEG): container finished" podID="ee32ddbf-9640-4efd-b2a5-babcfb306088" containerID="5d0ca013a49dbf4aa2b7c2101ce60a5700645908e822083179cc5aba34f37e53" exitCode=60 Oct 30 20:15:06.017240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.016832 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" event={"ID":"ee32ddbf-9640-4efd-b2a5-babcfb306088","Type":"ContainerDied","Data":"5d0ca013a49dbf4aa2b7c2101ce60a5700645908e822083179cc5aba34f37e53"} Oct 30 20:15:06.017370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.017255 5091 scope.go:122] "RemoveContainer" containerID="5d0ca013a49dbf4aa2b7c2101ce60a5700645908e822083179cc5aba34f37e53" Oct 30 20:15:06.017535 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:06.017510 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6_odh-ai-gateway-infra(ee32ddbf-9640-4efd-b2a5-babcfb306088)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" podUID="ee32ddbf-9640-4efd-b2a5-babcfb306088" Oct 30 20:15:06.019062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.019029 5091 generic.go:356] "Generic (PLEG): container finished" podID="87e33682-f349-450b-ab65-1b60d304a6cf" containerID="0d8d851af0c3a1086af686185eca57a61976cd4e46c92f1e78368f64360f6da0" exitCode=60 Oct 30 20:15:06.019210 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.019100 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" event={"ID":"87e33682-f349-450b-ab65-1b60d304a6cf","Type":"ContainerDied","Data":"0d8d851af0c3a1086af686185eca57a61976cd4e46c92f1e78368f64360f6da0"} Oct 30 20:15:06.019414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.019393 5091 scope.go:122] "RemoveContainer" containerID="0d8d851af0c3a1086af686185eca57a61976cd4e46c92f1e78368f64360f6da0" Oct 30 20:15:06.019689 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:06.019655 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm_odh-ai-gateway-infra(87e33682-f349-450b-ab65-1b60d304a6cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" podUID="87e33682-f349-450b-ab65-1b60d304a6cf" Oct 30 20:15:06.024701 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.024679 5091 scope.go:122] "RemoveContainer" containerID="3c1a45fc95a1afae10633d81e9b61cc00194472c5ca8a73b1d115f444ee23485" Oct 30 20:15:06.044208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.044181 5091 scope.go:122] "RemoveContainer" containerID="b09ae333e8f53d095f5360e3616e7186fcc62ee98369e1eba69478010afe952a" Oct 30 20:15:06.162661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.162634 5091 scope.go:122] "RemoveContainer" containerID="77257cc958eb62b14128c68472477e03979c68936c15bd84a43e91ec9cda768d" Oct 30 20:15:06.408730 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.408703 5091 scope.go:122] "RemoveContainer" containerID="f37f99519913ab4b604e0a8239612fa6ac44d8f2d14f603ab717b658d46c593c" Oct 30 20:15:06.424639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.424614 5091 scope.go:122] "RemoveContainer" containerID="73ba2056bf897ea06312ecd767cce3b4677471344853cf751c822c1fc1aa2e4c" Oct 30 20:15:06.447261 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.447175 5091 scope.go:122] "RemoveContainer" containerID="2082571e46ccdf5242037fc6e0592ec56e244065c0631c319fcb6c0ac622f2ae" Oct 30 20:15:06.472415 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.472387 5091 scope.go:122] "RemoveContainer" containerID="2d9798b1e40e0f88987d69326ed2e9945fcfe004e2fa7c0e7150b95ae7a3ddad" Oct 30 20:15:06.488689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.488663 5091 scope.go:122] "RemoveContainer" containerID="18365f83bfa0faf17b8fd049552b760882220df63095872e4ac180cc41cc15e5" Oct 30 20:15:06.503333 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.503312 5091 scope.go:122] "RemoveContainer" containerID="a58c8d9816ab1431997347ea673a58f89c87defdaef19ff0ecb0d3e03d657d2c" Oct 30 20:15:06.516840 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.516818 5091 scope.go:122] "RemoveContainer" containerID="d40c8d8a897a6afffb0c68906d8650fb3167cfc98b56e6f2589fce4ebba729ac" Oct 30 20:15:06.533214 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.533181 5091 scope.go:122] "RemoveContainer" containerID="b0d1ecbd65b35d1a19f8105c6105db3e7dd0eb48a0de8a8ec8bb38c0d51a28f6" Oct 30 20:15:06.952595 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:06.952505 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:07.030743 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.030708 5091 generic.go:356] "Generic (PLEG): container finished" podID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" containerID="b3b874c4b57483016809f6f6bf50016c6a56399d516ff31b0615f3425d9dd39b" exitCode=60 Oct 30 20:15:07.031241 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.030816 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" event={"ID":"8fb65533-38aa-48c4-a374-fcf2f5e96f99","Type":"ContainerDied","Data":"b3b874c4b57483016809f6f6bf50016c6a56399d516ff31b0615f3425d9dd39b"} Oct 30 20:15:07.031241 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.030904 5091 scope.go:122] "RemoveContainer" containerID="89f332143866d1718cf46a57c8fb7554759c3efe76bd70ef2a177ad661017a85" Oct 30 20:15:07.031241 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.031122 5091 scope.go:122] "RemoveContainer" containerID="b3b874c4b57483016809f6f6bf50016c6a56399d516ff31b0615f3425d9dd39b" Oct 30 20:15:07.031518 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.031492 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm_odh-ai-gateway-infra(8fb65533-38aa-48c4-a374-fcf2f5e96f99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" podUID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" Oct 30 20:15:07.033514 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.033491 5091 scope.go:122] "RemoveContainer" containerID="250846c9f7cae177aa121adc9786a0544afb17d70a28204ec5e72f9caec9b3a9" Oct 30 20:15:07.033835 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.033816 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28_odh-ai-gateway-infra(4eafdae9-7fcd-4579-824b-60bf84bb9b7f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" podUID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" Oct 30 20:15:07.035112 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.035095 5091 scope.go:122] "RemoveContainer" containerID="101fe9c74699ac0e0a5b6c9ae289c320559cd9933c85443cdfc059f2ef180b01" Oct 30 20:15:07.035454 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.035434 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb_odh-ai-gateway-infra(b7d9f0fd-4bb0-4773-8333-92f168e683c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" podUID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" Oct 30 20:15:07.036792 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.036767 5091 scope.go:122] "RemoveContainer" containerID="5d0ca013a49dbf4aa2b7c2101ce60a5700645908e822083179cc5aba34f37e53" Oct 30 20:15:07.037087 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.037064 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6_odh-ai-gateway-infra(ee32ddbf-9640-4efd-b2a5-babcfb306088)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" podUID="ee32ddbf-9640-4efd-b2a5-babcfb306088" Oct 30 20:15:07.038554 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.038535 5091 scope.go:122] "RemoveContainer" containerID="0d8d851af0c3a1086af686185eca57a61976cd4e46c92f1e78368f64360f6da0" Oct 30 20:15:07.038770 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.038751 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm_odh-ai-gateway-infra(87e33682-f349-450b-ab65-1b60d304a6cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" podUID="87e33682-f349-450b-ab65-1b60d304a6cf" Oct 30 20:15:07.040332 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.040314 5091 scope.go:122] "RemoveContainer" containerID="6cc8aebd7337da21011cb107130eca75c7efe1683ced5f889ee2bffaf1154647" Oct 30 20:15:07.040571 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.040540 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d_odh-ai-gateway-infra(78e70c72-9983-4b2e-bd0c-176f50822feb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" podUID="78e70c72-9983-4b2e-bd0c-176f50822feb" Oct 30 20:15:07.042100 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.042081 5091 scope.go:122] "RemoveContainer" containerID="be87937e01bde7501599241c9bedbae8c14f3d13ec6dc0219ed302e5a34e3c51" Oct 30 20:15:07.042386 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.042363 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p_odh-ai-gateway-infra(b8336ff5-7043-4f2a-8ab1-60c4713596e9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" podUID="b8336ff5-7043-4f2a-8ab1-60c4713596e9" Oct 30 20:15:07.043774 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.043754 5091 generic.go:356] "Generic (PLEG): container finished" podID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" containerID="b6b778b10292809aa1ffd6171d6c4a597bb04eb9e6eb69d3ec516e7c1bc1f83f" exitCode=60 Oct 30 20:15:07.043879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.043816 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" event={"ID":"8b69782b-78e9-4baf-8bdf-bcbf9b78767d","Type":"ContainerDied","Data":"b6b778b10292809aa1ffd6171d6c4a597bb04eb9e6eb69d3ec516e7c1bc1f83f"} Oct 30 20:15:07.044058 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.044043 5091 scope.go:122] "RemoveContainer" containerID="b6b778b10292809aa1ffd6171d6c4a597bb04eb9e6eb69d3ec516e7c1bc1f83f" Oct 30 20:15:07.044342 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.044320 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx_odh-ai-gateway-infra(8b69782b-78e9-4baf-8bdf-bcbf9b78767d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" podUID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" Oct 30 20:15:07.045879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.045860 5091 scope.go:122] "RemoveContainer" containerID="4359cdb70a9f6f96a6015f9a4602aafda71910976fac991ab52ea927faef2ad4" Oct 30 20:15:07.046296 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.046271 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v_odh-ai-gateway-infra(67449cd5-2639-4625-b32e-ba3bec5a528b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" podUID="67449cd5-2639-4625-b32e-ba3bec5a528b" Oct 30 20:15:07.048142 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.048119 5091 generic.go:356] "Generic (PLEG): container finished" podID="2ea189f9-f225-4fdb-8948-14cb53111cd5" containerID="4d3ca8b3f178e9621270db788279994ba03ec76906628e6c0bb3a6194db437bd" exitCode=60 Oct 30 20:15:07.048285 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.048250 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" event={"ID":"2ea189f9-f225-4fdb-8948-14cb53111cd5","Type":"ContainerDied","Data":"4d3ca8b3f178e9621270db788279994ba03ec76906628e6c0bb3a6194db437bd"} Oct 30 20:15:07.048499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.048478 5091 scope.go:122] "RemoveContainer" containerID="4d3ca8b3f178e9621270db788279994ba03ec76906628e6c0bb3a6194db437bd" Oct 30 20:15:07.048796 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.048776 5091 scope.go:122] "RemoveContainer" containerID="6982059c208300e3100c40cadbbe5b3ef83236036bee49b0d077cad4aa200e6a" Oct 30 20:15:07.048977 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.048952 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw_odh-ai-gateway-infra(2ea189f9-f225-4fdb-8948-14cb53111cd5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" podUID="2ea189f9-f225-4fdb-8948-14cb53111cd5" Oct 30 20:15:07.050403 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.050385 5091 scope.go:122] "RemoveContainer" containerID="d518b80f9487d20c1e27524014d8510b3151c85add30065fdc46eeeb6e569d83" Oct 30 20:15:07.050619 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.050600 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl_odh-ai-gateway-infra(755783df-2b63-4013-9133-d39c004e577f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" podUID="755783df-2b63-4013-9133-d39c004e577f" Oct 30 20:15:07.052539 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.052519 5091 scope.go:122] "RemoveContainer" containerID="1767ef190a7b5110d5c2750c39245946fbc6f170526642fcb77b73cc0b616476" Oct 30 20:15:07.052775 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.052748 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz_odh-ai-gateway-infra(1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" podUID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" Oct 30 20:15:07.054794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.054771 5091 generic.go:356] "Generic (PLEG): container finished" podID="2a5e04ce-a8ad-4b44-bd00-624a49388708" containerID="9fd0a20b190e89a6cb45bdf7ee6d4597c4c45e9526b5b4f862b01f67833f4416" exitCode=60 Oct 30 20:15:07.055214 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.055194 5091 scope.go:122] "RemoveContainer" containerID="9fd0a20b190e89a6cb45bdf7ee6d4597c4c45e9526b5b4f862b01f67833f4416" Oct 30 20:15:07.055482 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.055457 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx_odh-ai-gateway-infra(2a5e04ce-a8ad-4b44-bd00-624a49388708)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" podUID="2a5e04ce-a8ad-4b44-bd00-624a49388708" Oct 30 20:15:07.055572 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.055501 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" event={"ID":"2a5e04ce-a8ad-4b44-bd00-624a49388708","Type":"ContainerDied","Data":"9fd0a20b190e89a6cb45bdf7ee6d4597c4c45e9526b5b4f862b01f67833f4416"} Oct 30 20:15:07.057900 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.057879 5091 scope.go:122] "RemoveContainer" containerID="228ce452fc244df668d0698165baecf6054ce4822ad94a73865a51e121000d09" Oct 30 20:15:07.058133 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.058111 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7_odh-ai-gateway-infra(3d37c05b-93b8-4c8c-84d1-9e186d7ec265)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" podUID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" Oct 30 20:15:07.060201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.060180 5091 scope.go:122] "RemoveContainer" containerID="5bcc7b29b6360ff50c0212de016b853d5c37197d5dbde8921e8f7fb4a00ef8fc" Oct 30 20:15:07.060441 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.060421 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc_odh-ai-gateway-infra(f93e44d7-d7d5-4beb-9332-403ddc9bf336)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" podUID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" Oct 30 20:15:07.062008 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.061992 5091 scope.go:122] "RemoveContainer" containerID="f70ce32f693cb1edbfd8e1cc46058eb5803e02afbba4b84364824720004ef466" Oct 30 20:15:07.062195 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:07.062178 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct_odh-ai-gateway-infra(387559a4-9827-4a53-90ae-3f3f0852d25d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" podUID="387559a4-9827-4a53-90ae-3f3f0852d25d" Oct 30 20:15:07.068602 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.068583 5091 scope.go:122] "RemoveContainer" containerID="b1ebfdeb8724292516483c19ef7846c802777dfe97f5716c9e59468348670ece" Oct 30 20:15:07.085932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.085906 5091 scope.go:122] "RemoveContainer" containerID="daad74b112bde15bcf86632dedeb6f294f827b3a688a9d3ae2b634902f57dfcf" Oct 30 20:15:07.941183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:07.935920 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:08.075616 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:08.075581 5091 scope.go:122] "RemoveContainer" containerID="b6b778b10292809aa1ffd6171d6c4a597bb04eb9e6eb69d3ec516e7c1bc1f83f" Oct 30 20:15:08.076087 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:08.076037 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx_odh-ai-gateway-infra(8b69782b-78e9-4baf-8bdf-bcbf9b78767d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" podUID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" Oct 30 20:15:08.077581 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:08.077556 5091 scope.go:122] "RemoveContainer" containerID="4d3ca8b3f178e9621270db788279994ba03ec76906628e6c0bb3a6194db437bd" Oct 30 20:15:08.077990 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:08.077960 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw_odh-ai-gateway-infra(2ea189f9-f225-4fdb-8948-14cb53111cd5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" podUID="2ea189f9-f225-4fdb-8948-14cb53111cd5" Oct 30 20:15:08.079395 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:08.079372 5091 scope.go:122] "RemoveContainer" containerID="9fd0a20b190e89a6cb45bdf7ee6d4597c4c45e9526b5b4f862b01f67833f4416" Oct 30 20:15:08.079741 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:08.079716 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx_odh-ai-gateway-infra(2a5e04ce-a8ad-4b44-bd00-624a49388708)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" podUID="2a5e04ce-a8ad-4b44-bd00-624a49388708" Oct 30 20:15:08.081432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:08.081410 5091 scope.go:122] "RemoveContainer" containerID="b3b874c4b57483016809f6f6bf50016c6a56399d516ff31b0615f3425d9dd39b" Oct 30 20:15:08.081776 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:08.081750 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm_odh-ai-gateway-infra(8fb65533-38aa-48c4-a374-fcf2f5e96f99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" podUID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" Oct 30 20:15:08.869630 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:08.869592 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:09.822170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:09.821702 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:10.874173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:10.873973 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:11.741579 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:11.741402 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:11.955273 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:11.929943 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:11.960413 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:11.955990 5091 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 20:15:11.986482 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:11.972485 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-zzkkt" podUID="c63bc1ab-ca04-4273-ae53-2d0923e4900b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:15:12.346177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:12.344665 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:15:12.664222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:12.647393 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:15:13.194197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:13.190898 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 20:15:13.231563 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:13.196888 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:15:13.232745 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:13.229753 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": context deadline exceeded" Oct 30 20:15:13.361248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:13.361187 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded" Oct 30 20:15:13.365362 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:13.365335 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 20:15:13.623175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:13.608016 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:13.870171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:13.869896 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-kkzgw" podUID="540a152c-2be5-4253-b272-586b34835043" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:15:13.920226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:13.918546 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt"] Oct 30 20:15:13.920226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:13.920032 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4efa6cde-c023-40c7-a04b-a73b1605e9c9" containerName="payload-processing" Oct 30 20:15:13.920226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:13.920054 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4efa6cde-c023-40c7-a04b-a73b1605e9c9" containerName="payload-processing" Oct 30 20:15:13.920534 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:13.920309 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4efa6cde-c023-40c7-a04b-a73b1605e9c9" containerName="payload-processing" Oct 30 20:15:13.936168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:13.932708 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/ready\": context deadline exceeded" Oct 30 20:15:14.111682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.111648 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 20:15:14.113171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.113135 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:14.136846 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.136809 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:14.194222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.162656 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:14.257027 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.256989 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:14.257350 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.257327 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:14.277588 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.277555 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kbmj\" (UniqueName: \"kubernetes.io/projected/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-kube-api-access-8kbmj\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:14.278134 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.195351 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"agc-cred-f4868d300b41\"" Oct 30 20:15:14.283787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.195255 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"agc-cred-aa686e60cc04\"" Oct 30 20:15:14.283787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.195457 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"agc-cred-52dc941f181c\"" Oct 30 20:15:14.283787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.195578 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"payload-processing-external-model-e2e--151804b4-dockercfg-xxgch\"" Oct 30 20:15:14.283787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.196307 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"kube-root-ca.crt\"" Oct 30 20:15:14.283787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.196369 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"routing-overlay\"" Oct 30 20:15:14.283787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.209646 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded" Oct 30 20:15:14.283787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.252460 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 20:15:14.284573 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.195180 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"openshift-service-ca.crt\"" Oct 30 20:15:14.288062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.195235 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt"] Oct 30 20:15:14.288062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.287780 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 20:15:14.288062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.287827 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 20:15:14.309165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.195409 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d97cc8\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d97cc8\"" Oct 30 20:15:14.331472 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.331432 5091 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" probeResult="failure" output="Get \"https://10.132.0.76:8000/health\": context deadline exceeded" Oct 30 20:15:14.358290 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.354970 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 20:15:14.358290 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.355450 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 20:15:14.394354 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.394308 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:14.394609 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.394590 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:14.394942 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.394921 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:14.395133 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.395109 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kbmj\" (UniqueName: \"kubernetes.io/projected/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-kube-api-access-8kbmj\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:14.396865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.396825 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:14.397934 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.397547 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:14.415827 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.415207 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:14.427186 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.427161 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kbmj\" (UniqueName: \"kubernetes.io/projected/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-kube-api-access-8kbmj\") pod \"payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:14.430521 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.430492 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:14.457251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.448487 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" podUID="542be5bf-ced0-491a-aade-a84c72a92e5c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 20:15:14.477186 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.477141 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 20:15:14.525881 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.525857 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7" Oct 30 20:15:14.756670 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.756636 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:14.942440 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:14.942403 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf6a7763_ca3a_48b9_8613_bc5e3e1bbd8c.slice/crio-0e3d46cf9f9c028e3c8e904c0ba6ab4013cf0c5e06695f8c54dc83b6e4364c79 WatchSource:0}: Error finding container 0e3d46cf9f9c028e3c8e904c0ba6ab4013cf0c5e06695f8c54dc83b6e4364c79: Status 404 returned error can't find the container with id 0e3d46cf9f9c028e3c8e904c0ba6ab4013cf0c5e06695f8c54dc83b6e4364c79 Oct 30 20:15:14.943443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.943417 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt"] Oct 30 20:15:14.998018 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:14.997983 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" event={"ID":"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c","Type":"ContainerStarted","Data":"0e3d46cf9f9c028e3c8e904c0ba6ab4013cf0c5e06695f8c54dc83b6e4364c79"} Oct 30 20:15:15.201172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:15.200889 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:15.981555 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:15.981511 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:15:16.010047 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:16.010010 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" event={"ID":"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c","Type":"ContainerStarted","Data":"787bb21fae10d114351909eaba5a774d7d5ffed55967fe65344373866492caa5"} Oct 30 20:15:16.010244 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:16.010078 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:16.039087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:16.039034 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" podStartSLOduration=3.039020106 podStartE2EDuration="3.039020106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:16.035252793 +0000 UTC m=+2635.730460750" watchObservedRunningTime="2026-10-30 20:15:16.039020106 +0000 UTC m=+2635.734228064" Oct 30 20:15:16.140654 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:16.137843 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:16.433669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:16.433644 5091 scope.go:122] "RemoveContainer" containerID="228ce452fc244df668d0698165baecf6054ce4822ad94a73865a51e121000d09" Oct 30 20:15:16.434116 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:16.433836 5091 scope.go:122] "RemoveContainer" containerID="9fd0a20b190e89a6cb45bdf7ee6d4597c4c45e9526b5b4f862b01f67833f4416" Oct 30 20:15:16.434116 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:16.434017 5091 scope.go:122] "RemoveContainer" containerID="1767ef190a7b5110d5c2750c39245946fbc6f170526642fcb77b73cc0b616476" Oct 30 20:15:16.434253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:16.434178 5091 scope.go:122] "RemoveContainer" containerID="101fe9c74699ac0e0a5b6c9ae289c320559cd9933c85443cdfc059f2ef180b01" Oct 30 20:15:17.025394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:17.025289 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" event={"ID":"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c","Type":"ContainerStarted","Data":"4d682d6a5a71759e7025769a04a160a5b488f0da3183e3303d8ceb948de78b91"} Oct 30 20:15:17.027419 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:17.027393 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" event={"ID":"2a5e04ce-a8ad-4b44-bd00-624a49388708","Type":"ContainerStarted","Data":"e88256460aea93f09c539137c45b9aa4fc8b02f33d5b92cdf742371e9f356f6e"} Oct 30 20:15:17.029381 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:17.029355 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" event={"ID":"3d37c05b-93b8-4c8c-84d1-9e186d7ec265","Type":"ContainerStarted","Data":"a5a14b39300756a8f84006d708fef317de7c5cce47c91ebab81674ef11df47d6"} Oct 30 20:15:17.031222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:17.031199 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" event={"ID":"b7d9f0fd-4bb0-4773-8333-92f168e683c3","Type":"ContainerStarted","Data":"b901bc6a805207cabdc58c6da33ab0f0e337b8de660cae6cf2bd1f5f381f5c06"} Oct 30 20:15:17.117016 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:17.116978 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:17.434421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:17.434332 5091 scope.go:122] "RemoveContainer" containerID="b6b778b10292809aa1ffd6171d6c4a597bb04eb9e6eb69d3ec516e7c1bc1f83f" Oct 30 20:15:17.434872 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:17.434628 5091 scope.go:122] "RemoveContainer" containerID="6cc8aebd7337da21011cb107130eca75c7efe1683ced5f889ee2bffaf1154647" Oct 30 20:15:18.043929 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.043896 5091 generic.go:356] "Generic (PLEG): container finished" podID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" containerID="b901bc6a805207cabdc58c6da33ab0f0e337b8de660cae6cf2bd1f5f381f5c06" exitCode=60 Oct 30 20:15:18.044103 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.043965 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" event={"ID":"b7d9f0fd-4bb0-4773-8333-92f168e683c3","Type":"ContainerDied","Data":"b901bc6a805207cabdc58c6da33ab0f0e337b8de660cae6cf2bd1f5f381f5c06"} Oct 30 20:15:18.044103 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.044030 5091 scope.go:122] "RemoveContainer" containerID="101fe9c74699ac0e0a5b6c9ae289c320559cd9933c85443cdfc059f2ef180b01" Oct 30 20:15:18.044494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.044474 5091 scope.go:122] "RemoveContainer" containerID="b901bc6a805207cabdc58c6da33ab0f0e337b8de660cae6cf2bd1f5f381f5c06" Oct 30 20:15:18.044948 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:18.044920 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb_odh-ai-gateway-infra(b7d9f0fd-4bb0-4773-8333-92f168e683c3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" podUID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" Oct 30 20:15:18.046784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.046755 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" event={"ID":"78e70c72-9983-4b2e-bd0c-176f50822feb","Type":"ContainerStarted","Data":"ace7e1be7c7e228f6a4f2c40c6b5f03aeb71ab9ee21887f7442848f767b4d2a6"} Oct 30 20:15:18.050209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.050184 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" event={"ID":"8b69782b-78e9-4baf-8bdf-bcbf9b78767d","Type":"ContainerStarted","Data":"c4a58bd90a5e408d5d35818ffec3302bad4aac3f0773b9692f011b8a516cae27"} Oct 30 20:15:18.053132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.053105 5091 generic.go:356] "Generic (PLEG): container finished" podID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" containerID="4d682d6a5a71759e7025769a04a160a5b488f0da3183e3303d8ceb948de78b91" exitCode=60 Oct 30 20:15:18.053266 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.053180 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" event={"ID":"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c","Type":"ContainerDied","Data":"4d682d6a5a71759e7025769a04a160a5b488f0da3183e3303d8ceb948de78b91"} Oct 30 20:15:18.053632 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.053609 5091 scope.go:122] "RemoveContainer" containerID="4d682d6a5a71759e7025769a04a160a5b488f0da3183e3303d8ceb948de78b91" Oct 30 20:15:18.054001 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:18.053971 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz_odh-ai-gateway-infra(1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" podUID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" Oct 30 20:15:18.056477 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.056454 5091 generic.go:356] "Generic (PLEG): container finished" podID="2a5e04ce-a8ad-4b44-bd00-624a49388708" containerID="e88256460aea93f09c539137c45b9aa4fc8b02f33d5b92cdf742371e9f356f6e" exitCode=60 Oct 30 20:15:18.056587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.056532 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" event={"ID":"2a5e04ce-a8ad-4b44-bd00-624a49388708","Type":"ContainerDied","Data":"e88256460aea93f09c539137c45b9aa4fc8b02f33d5b92cdf742371e9f356f6e"} Oct 30 20:15:18.056914 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.056890 5091 scope.go:122] "RemoveContainer" containerID="e88256460aea93f09c539137c45b9aa4fc8b02f33d5b92cdf742371e9f356f6e" Oct 30 20:15:18.057257 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:18.057231 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx_odh-ai-gateway-infra(2a5e04ce-a8ad-4b44-bd00-624a49388708)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" podUID="2a5e04ce-a8ad-4b44-bd00-624a49388708" Oct 30 20:15:18.059248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.059225 5091 generic.go:356] "Generic (PLEG): container finished" podID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" containerID="a5a14b39300756a8f84006d708fef317de7c5cce47c91ebab81674ef11df47d6" exitCode=60 Oct 30 20:15:18.059360 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.059274 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" event={"ID":"3d37c05b-93b8-4c8c-84d1-9e186d7ec265","Type":"ContainerDied","Data":"a5a14b39300756a8f84006d708fef317de7c5cce47c91ebab81674ef11df47d6"} Oct 30 20:15:18.059632 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.059614 5091 scope.go:122] "RemoveContainer" containerID="a5a14b39300756a8f84006d708fef317de7c5cce47c91ebab81674ef11df47d6" Oct 30 20:15:18.059948 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:18.059924 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7_odh-ai-gateway-infra(3d37c05b-93b8-4c8c-84d1-9e186d7ec265)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" podUID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" Oct 30 20:15:18.075617 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.075555 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz"] Oct 30 20:15:18.081331 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.081310 5091 scope.go:122] "RemoveContainer" containerID="1767ef190a7b5110d5c2750c39245946fbc6f170526642fcb77b73cc0b616476" Oct 30 20:15:18.097332 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.097314 5091 scope.go:122] "RemoveContainer" containerID="9fd0a20b190e89a6cb45bdf7ee6d4597c4c45e9526b5b4f862b01f67833f4416" Oct 30 20:15:18.112763 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.112740 5091 scope.go:122] "RemoveContainer" containerID="228ce452fc244df668d0698165baecf6054ce4822ad94a73865a51e121000d09" Oct 30 20:15:18.115467 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.115414 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb"] Oct 30 20:15:18.174864 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.174827 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx"] Oct 30 20:15:18.269244 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.266096 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:18.293518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.293486 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7"] Oct 30 20:15:18.434230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.434134 5091 scope.go:122] "RemoveContainer" containerID="be87937e01bde7501599241c9bedbae8c14f3d13ec6dc0219ed302e5a34e3c51" Oct 30 20:15:18.434414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.434310 5091 scope.go:122] "RemoveContainer" containerID="0d8d851af0c3a1086af686185eca57a61976cd4e46c92f1e78368f64360f6da0" Oct 30 20:15:18.434767 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:18.434728 5091 scope.go:122] "RemoveContainer" containerID="d2acfc31eda0f9cd25bd1af1d0f4b592dbf9006c52b6d04062a44cdcb3afa191" Oct 30 20:15:19.072534 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.072495 5091 generic.go:356] "Generic (PLEG): container finished" podID="78e70c72-9983-4b2e-bd0c-176f50822feb" containerID="ace7e1be7c7e228f6a4f2c40c6b5f03aeb71ab9ee21887f7442848f767b4d2a6" exitCode=60 Oct 30 20:15:19.072735 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.072573 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" event={"ID":"78e70c72-9983-4b2e-bd0c-176f50822feb","Type":"ContainerDied","Data":"ace7e1be7c7e228f6a4f2c40c6b5f03aeb71ab9ee21887f7442848f767b4d2a6"} Oct 30 20:15:19.072735 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.072637 5091 scope.go:122] "RemoveContainer" containerID="6cc8aebd7337da21011cb107130eca75c7efe1683ced5f889ee2bffaf1154647" Oct 30 20:15:19.073214 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.073191 5091 scope.go:122] "RemoveContainer" containerID="ace7e1be7c7e228f6a4f2c40c6b5f03aeb71ab9ee21887f7442848f767b4d2a6" Oct 30 20:15:19.073860 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:19.073661 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d_odh-ai-gateway-infra(78e70c72-9983-4b2e-bd0c-176f50822feb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" podUID="78e70c72-9983-4b2e-bd0c-176f50822feb" Oct 30 20:15:19.074962 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.074940 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" event={"ID":"b8336ff5-7043-4f2a-8ab1-60c4713596e9","Type":"ContainerStarted","Data":"7fd4f5c8ded3a7811e28436e6132fa9eb2cb385f60096febc8801bc1faf979eb"} Oct 30 20:15:19.078067 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.078041 5091 generic.go:356] "Generic (PLEG): container finished" podID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" containerID="c4a58bd90a5e408d5d35818ffec3302bad4aac3f0773b9692f011b8a516cae27" exitCode=60 Oct 30 20:15:19.078226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.078112 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" event={"ID":"8b69782b-78e9-4baf-8bdf-bcbf9b78767d","Type":"ContainerDied","Data":"c4a58bd90a5e408d5d35818ffec3302bad4aac3f0773b9692f011b8a516cae27"} Oct 30 20:15:19.078545 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.078515 5091 scope.go:122] "RemoveContainer" containerID="c4a58bd90a5e408d5d35818ffec3302bad4aac3f0773b9692f011b8a516cae27" Oct 30 20:15:19.078892 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:19.078864 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx_odh-ai-gateway-infra(8b69782b-78e9-4baf-8bdf-bcbf9b78767d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" podUID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" Oct 30 20:15:19.090801 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.090772 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" event={"ID":"d27d65d4-ebb6-4254-89c9-febcacbdf9f1","Type":"ContainerStarted","Data":"2539f17d62d021012e309681d2e6c9b1f05171e799271558c6c94c0e74579f86"} Oct 30 20:15:19.091803 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.091767 5091 scope.go:122] "RemoveContainer" containerID="b6b778b10292809aa1ffd6171d6c4a597bb04eb9e6eb69d3ec516e7c1bc1f83f" Oct 30 20:15:19.097317 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.097070 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" event={"ID":"87e33682-f349-450b-ab65-1b60d304a6cf","Type":"ContainerStarted","Data":"1d0067a0c832f8956c02d6afa30aaae3226c47f947edf95263c9afc8cf64cef6"} Oct 30 20:15:19.169192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.169137 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx"] Oct 30 20:15:19.219808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.219695 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d"] Oct 30 20:15:19.369263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.364495 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:19.392097 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.391880 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" Oct 30 20:15:19.412043 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.411999 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7d9f0fd-4bb0-4773-8333-92f168e683c3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d9f0fd-4bb0-4773-8333-92f168e683c3-maas-api-service-ca\") pod \"b7d9f0fd-4bb0-4773-8333-92f168e683c3\" (UID: \"b7d9f0fd-4bb0-4773-8333-92f168e683c3\") " Oct 30 20:15:19.412736 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.412714 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7d9f0fd-4bb0-4773-8333-92f168e683c3-kube-api-access-7wjjb\" (UniqueName: \"kubernetes.io/projected/b7d9f0fd-4bb0-4773-8333-92f168e683c3-kube-api-access-7wjjb\") pod \"b7d9f0fd-4bb0-4773-8333-92f168e683c3\" (UID: \"b7d9f0fd-4bb0-4773-8333-92f168e683c3\") " Oct 30 20:15:19.414209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.412645 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d9f0fd-4bb0-4773-8333-92f168e683c3-maas-api-service-ca" pod "b7d9f0fd-4bb0-4773-8333-92f168e683c3" (UID: "b7d9f0fd-4bb0-4773-8333-92f168e683c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:19.417484 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.417460 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d9f0fd-4bb0-4773-8333-92f168e683c3-kube-api-access-7wjjb" pod "b7d9f0fd-4bb0-4773-8333-92f168e683c3" (UID: "b7d9f0fd-4bb0-4773-8333-92f168e683c3"). InnerVolumeSpecName "kube-api-access-7wjjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:19.433760 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.433741 5091 scope.go:122] "RemoveContainer" containerID="4d3ca8b3f178e9621270db788279994ba03ec76906628e6c0bb3a6194db437bd" Oct 30 20:15:19.434804 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.434787 5091 scope.go:122] "RemoveContainer" containerID="5bcc7b29b6360ff50c0212de016b853d5c37197d5dbde8921e8f7fb4a00ef8fc" Oct 30 20:15:19.437615 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.437596 5091 scope.go:122] "RemoveContainer" containerID="4359cdb70a9f6f96a6015f9a4602aafda71910976fac991ab52ea927faef2ad4" Oct 30 20:15:19.438074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.438058 5091 scope.go:122] "RemoveContainer" containerID="f70ce32f693cb1edbfd8e1cc46058eb5803e02afbba4b84364824720004ef466" Oct 30 20:15:19.438520 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.438504 5091 scope.go:122] "RemoveContainer" containerID="b3b874c4b57483016809f6f6bf50016c6a56399d516ff31b0615f3425d9dd39b" Oct 30 20:15:19.438980 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.438964 5091 scope.go:122] "RemoveContainer" containerID="5d0ca013a49dbf4aa2b7c2101ce60a5700645908e822083179cc5aba34f37e53" Oct 30 20:15:19.439756 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.439171 5091 scope.go:122] "RemoveContainer" containerID="d518b80f9487d20c1e27524014d8510b3151c85add30065fdc46eeeb6e569d83" Oct 30 20:15:19.648530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.565313 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" Oct 30 20:15:19.648530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.641049 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7d9f0fd-4bb0-4773-8333-92f168e683c3-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:19.648530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.641083 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wjjb\" (UniqueName: \"kubernetes.io/projected/b7d9f0fd-4bb0-4773-8333-92f168e683c3-kube-api-access-7wjjb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:19.671376 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.669081 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:19.745211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.741919 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c-maas-api-service-ca\") pod \"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c\" (UID: \"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c\") " Oct 30 20:15:19.745211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.742734 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c-kube-api-access-pq6wr\" (UniqueName: \"kubernetes.io/projected/1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c-kube-api-access-pq6wr\") pod \"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c\" (UID: \"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c\") " Oct 30 20:15:19.745211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.744898 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c-maas-api-service-ca" pod "1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" (UID: "1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:19.749450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.749415 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c-kube-api-access-pq6wr" pod "1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" (UID: "1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c"). InnerVolumeSpecName "kube-api-access-pq6wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:19.839955 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:19.839897 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8336ff5_7043_4f2a_8ab1_60c4713596e9.slice/crio-conmon-7fd4f5c8ded3a7811e28436e6132fa9eb2cb385f60096febc8801bc1faf979eb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8336ff5_7043_4f2a_8ab1_60c4713596e9.slice/crio-7fd4f5c8ded3a7811e28436e6132fa9eb2cb385f60096febc8801bc1faf979eb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27d65d4_ebb6_4254_89c9_febcacbdf9f1.slice/crio-2539f17d62d021012e309681d2e6c9b1f05171e799271558c6c94c0e74579f86.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd27d65d4_ebb6_4254_89c9_febcacbdf9f1.slice/crio-conmon-2539f17d62d021012e309681d2e6c9b1f05171e799271558c6c94c0e74579f86.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e33682_f349_450b_ab65_1b60d304a6cf.slice/crio-conmon-1d0067a0c832f8956c02d6afa30aaae3226c47f947edf95263c9afc8cf64cef6.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e33682_f349_450b_ab65_1b60d304a6cf.slice/crio-1d0067a0c832f8956c02d6afa30aaae3226c47f947edf95263c9afc8cf64cef6.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:15:19.846725 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.846694 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:19.847925 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.847900 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pq6wr\" (UniqueName: \"kubernetes.io/projected/1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c-kube-api-access-pq6wr\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:19.914173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.914099 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" Oct 30 20:15:19.921651 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:19.921623 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" Oct 30 20:15:20.043546 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.030847 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:20.207526 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.207431 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a5e04ce-a8ad-4b44-bd00-624a49388708-kube-api-access-vh6zz\" (UniqueName: \"kubernetes.io/projected/2a5e04ce-a8ad-4b44-bd00-624a49388708-kube-api-access-vh6zz\") pod \"2a5e04ce-a8ad-4b44-bd00-624a49388708\" (UID: \"2a5e04ce-a8ad-4b44-bd00-624a49388708\") " Oct 30 20:15:20.207793 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.207759 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5e04ce-a8ad-4b44-bd00-624a49388708-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5e04ce-a8ad-4b44-bd00-624a49388708-maas-api-service-ca\") pod \"2a5e04ce-a8ad-4b44-bd00-624a49388708\" (UID: \"2a5e04ce-a8ad-4b44-bd00-624a49388708\") " Oct 30 20:15:20.208015 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.207995 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d37c05b-93b8-4c8c-84d1-9e186d7ec265-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d37c05b-93b8-4c8c-84d1-9e186d7ec265-maas-api-service-ca\") pod \"3d37c05b-93b8-4c8c-84d1-9e186d7ec265\" (UID: \"3d37c05b-93b8-4c8c-84d1-9e186d7ec265\") " Oct 30 20:15:20.208590 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.208564 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d37c05b-93b8-4c8c-84d1-9e186d7ec265-kube-api-access-2d62b\" (UniqueName: \"kubernetes.io/projected/3d37c05b-93b8-4c8c-84d1-9e186d7ec265-kube-api-access-2d62b\") pod \"3d37c05b-93b8-4c8c-84d1-9e186d7ec265\" (UID: \"3d37c05b-93b8-4c8c-84d1-9e186d7ec265\") " Oct 30 20:15:20.210428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.210403 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5e04ce-a8ad-4b44-bd00-624a49388708-maas-api-service-ca" pod "2a5e04ce-a8ad-4b44-bd00-624a49388708" (UID: "2a5e04ce-a8ad-4b44-bd00-624a49388708"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:20.214771 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.214712 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d37c05b-93b8-4c8c-84d1-9e186d7ec265-maas-api-service-ca" pod "3d37c05b-93b8-4c8c-84d1-9e186d7ec265" (UID: "3d37c05b-93b8-4c8c-84d1-9e186d7ec265"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:20.266384 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.266234 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p"] Oct 30 20:15:20.285542 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.284426 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5e04ce-a8ad-4b44-bd00-624a49388708-kube-api-access-vh6zz" pod "2a5e04ce-a8ad-4b44-bd00-624a49388708" (UID: "2a5e04ce-a8ad-4b44-bd00-624a49388708"). InnerVolumeSpecName "kube-api-access-vh6zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:20.363874 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.325330 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt"] Oct 30 20:15:20.363874 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.327973 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d37c05b-93b8-4c8c-84d1-9e186d7ec265-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:20.363874 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.328002 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh6zz\" (UniqueName: \"kubernetes.io/projected/2a5e04ce-a8ad-4b44-bd00-624a49388708-kube-api-access-vh6zz\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:20.363874 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.328021 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5e04ce-a8ad-4b44-bd00-624a49388708-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:20.381331 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.381291 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd"] Oct 30 20:15:20.418570 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.418534 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:20.469728 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.468595 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d37c05b-93b8-4c8c-84d1-9e186d7ec265-kube-api-access-2d62b" pod "3d37c05b-93b8-4c8c-84d1-9e186d7ec265" (UID: "3d37c05b-93b8-4c8c-84d1-9e186d7ec265"). InnerVolumeSpecName "kube-api-access-2d62b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:20.478077 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.478047 5091 scope.go:122] "RemoveContainer" containerID="250846c9f7cae177aa121adc9786a0544afb17d70a28204ec5e72f9caec9b3a9" Oct 30 20:15:20.649046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.648080 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" Oct 30 20:15:20.651272 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.651206 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d62b\" (UniqueName: \"kubernetes.io/projected/3d37c05b-93b8-4c8c-84d1-9e186d7ec265-kube-api-access-2d62b\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:20.661628 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.661595 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb" event={"ID":"b7d9f0fd-4bb0-4773-8333-92f168e683c3","Type":"ContainerDied","Data":"1ae08a3ca619d3c72e4eecfe6172f0a90cbfe89251b05e4988849b12b55b1843"} Oct 30 20:15:20.661744 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.661685 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm"] Oct 30 20:15:20.664873 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.663503 5091 scope.go:122] "RemoveContainer" containerID="b901bc6a805207cabdc58c6da33ab0f0e337b8de660cae6cf2bd1f5f381f5c06" Oct 30 20:15:20.665828 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.665368 5091 generic.go:356] "Generic (PLEG): container finished" podID="d27d65d4-ebb6-4254-89c9-febcacbdf9f1" containerID="2539f17d62d021012e309681d2e6c9b1f05171e799271558c6c94c0e74579f86" exitCode=60 Oct 30 20:15:20.665933 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.665834 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" event={"ID":"d27d65d4-ebb6-4254-89c9-febcacbdf9f1","Type":"ContainerDied","Data":"2539f17d62d021012e309681d2e6c9b1f05171e799271558c6c94c0e74579f86"} Oct 30 20:15:20.672879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.672728 5091 generic.go:356] "Generic (PLEG): container finished" podID="87e33682-f349-450b-ab65-1b60d304a6cf" containerID="1d0067a0c832f8956c02d6afa30aaae3226c47f947edf95263c9afc8cf64cef6" exitCode=60 Oct 30 20:15:20.673292 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.673235 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" event={"ID":"87e33682-f349-450b-ab65-1b60d304a6cf","Type":"ContainerDied","Data":"1d0067a0c832f8956c02d6afa30aaae3226c47f947edf95263c9afc8cf64cef6"} Oct 30 20:15:20.693211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.693060 5091 generic.go:356] "Generic (PLEG): container finished" podID="b8336ff5-7043-4f2a-8ab1-60c4713596e9" containerID="7fd4f5c8ded3a7811e28436e6132fa9eb2cb385f60096febc8801bc1faf979eb" exitCode=60 Oct 30 20:15:20.694356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.693135 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" event={"ID":"b8336ff5-7043-4f2a-8ab1-60c4713596e9","Type":"ContainerDied","Data":"7fd4f5c8ded3a7811e28436e6132fa9eb2cb385f60096febc8801bc1faf979eb"} Oct 30 20:15:20.719355 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.719140 5091 scope.go:122] "RemoveContainer" containerID="d2acfc31eda0f9cd25bd1af1d0f4b592dbf9006c52b6d04062a44cdcb3afa191" Oct 30 20:15:20.748203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.747873 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" event={"ID":"1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c","Type":"ContainerDied","Data":"5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397"} Oct 30 20:15:20.748203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.748002 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz" Oct 30 20:15:20.774370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.772398 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" event={"ID":"2a5e04ce-a8ad-4b44-bd00-624a49388708","Type":"ContainerDied","Data":"afb8f2284444e05a734dc8a16c3cd76ee7d8dff69cf1ec33228679c8d68fdb43"} Oct 30 20:15:20.774370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.772531 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx" Oct 30 20:15:20.796910 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.796129 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" event={"ID":"3d37c05b-93b8-4c8c-84d1-9e186d7ec265","Type":"ContainerDied","Data":"9cbef9b0df16f0636e3685caece18351100d9ac67b37e47f1af539eb9eab51a9"} Oct 30 20:15:20.796910 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.796275 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7" Oct 30 20:15:20.801935 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.801893 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb"] Oct 30 20:15:20.826340 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.826290 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-779fa3-29889855-65krb"] Oct 30 20:15:20.829373 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.829097 5091 scope.go:122] "RemoveContainer" containerID="0d8d851af0c3a1086af686185eca57a61976cd4e46c92f1e78368f64360f6da0" Oct 30 20:15:21.021215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:20.975214 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:21.021215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.020420 5091 scope.go:122] "RemoveContainer" containerID="be87937e01bde7501599241c9bedbae8c14f3d13ec6dc0219ed302e5a34e3c51" Oct 30 20:15:21.064256 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.060070 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx"] Oct 30 20:15:21.101535 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.101499 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d97cc8-29889855-729kx"] Oct 30 20:15:21.104344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.104321 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7"] Oct 30 20:15:21.105685 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.105664 5091 scope.go:122] "RemoveContainer" containerID="4d682d6a5a71759e7025769a04a160a5b488f0da3183e3303d8ceb948de78b91" Oct 30 20:15:21.109324 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.109299 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-4d2317-29889855-4p5v7"] Oct 30 20:15:21.114207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.114123 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz"] Oct 30 20:15:21.120103 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.120078 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-222480-29889855-tgxhz"] Oct 30 20:15:21.121843 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.121821 5091 scope.go:122] "RemoveContainer" containerID="e88256460aea93f09c539137c45b9aa4fc8b02f33d5b92cdf742371e9f356f6e" Oct 30 20:15:21.136443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.136429 5091 scope.go:122] "RemoveContainer" containerID="a5a14b39300756a8f84006d708fef317de7c5cce47c91ebab81674ef11df47d6" Oct 30 20:15:21.241906 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.241876 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" Oct 30 20:15:21.280040 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.279971 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" Oct 30 20:15:21.400451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.397403 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e33682-f349-450b-ab65-1b60d304a6cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e33682-f349-450b-ab65-1b60d304a6cf-maas-api-service-ca\") pod \"87e33682-f349-450b-ab65-1b60d304a6cf\" (UID: \"87e33682-f349-450b-ab65-1b60d304a6cf\") " Oct 30 20:15:21.400451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.397550 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78e70c72-9983-4b2e-bd0c-176f50822feb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e70c72-9983-4b2e-bd0c-176f50822feb-maas-api-service-ca\") pod \"78e70c72-9983-4b2e-bd0c-176f50822feb\" (UID: \"78e70c72-9983-4b2e-bd0c-176f50822feb\") " Oct 30 20:15:21.400451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.397646 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e33682-f349-450b-ab65-1b60d304a6cf-kube-api-access-f2x5r\" (UniqueName: \"kubernetes.io/projected/87e33682-f349-450b-ab65-1b60d304a6cf-kube-api-access-f2x5r\") pod \"87e33682-f349-450b-ab65-1b60d304a6cf\" (UID: \"87e33682-f349-450b-ab65-1b60d304a6cf\") " Oct 30 20:15:21.400451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.397671 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78e70c72-9983-4b2e-bd0c-176f50822feb-kube-api-access-sh6zk\" (UniqueName: \"kubernetes.io/projected/78e70c72-9983-4b2e-bd0c-176f50822feb-kube-api-access-sh6zk\") pod \"78e70c72-9983-4b2e-bd0c-176f50822feb\" (UID: \"78e70c72-9983-4b2e-bd0c-176f50822feb\") " Oct 30 20:15:21.400451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.399859 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78e70c72-9983-4b2e-bd0c-176f50822feb-maas-api-service-ca" pod "78e70c72-9983-4b2e-bd0c-176f50822feb" (UID: "78e70c72-9983-4b2e-bd0c-176f50822feb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:21.400451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.400410 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e33682-f349-450b-ab65-1b60d304a6cf-maas-api-service-ca" pod "87e33682-f349-450b-ab65-1b60d304a6cf" (UID: "87e33682-f349-450b-ab65-1b60d304a6cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:21.484617 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.484547 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e33682-f349-450b-ab65-1b60d304a6cf-kube-api-access-f2x5r" pod "87e33682-f349-450b-ab65-1b60d304a6cf" (UID: "87e33682-f349-450b-ab65-1b60d304a6cf"). InnerVolumeSpecName "kube-api-access-f2x5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:21.495103 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.495054 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78e70c72-9983-4b2e-bd0c-176f50822feb-kube-api-access-sh6zk" pod "78e70c72-9983-4b2e-bd0c-176f50822feb" (UID: "78e70c72-9983-4b2e-bd0c-176f50822feb"). InnerVolumeSpecName "kube-api-access-sh6zk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:21.502385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.502280 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87e33682-f349-450b-ab65-1b60d304a6cf-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.502385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.502325 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78e70c72-9983-4b2e-bd0c-176f50822feb-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.502385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.502343 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2x5r\" (UniqueName: \"kubernetes.io/projected/87e33682-f349-450b-ab65-1b60d304a6cf-kube-api-access-f2x5r\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.502385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.502358 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sh6zk\" (UniqueName: \"kubernetes.io/projected/78e70c72-9983-4b2e-bd0c-176f50822feb-kube-api-access-sh6zk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.615092 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.615066 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" Oct 30 20:15:21.624280 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.624254 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" Oct 30 20:15:21.629640 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.629617 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" Oct 30 20:15:21.705586 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.705553 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8336ff5-7043-4f2a-8ab1-60c4713596e9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8336ff5-7043-4f2a-8ab1-60c4713596e9-maas-api-service-ca\") pod \"b8336ff5-7043-4f2a-8ab1-60c4713596e9\" (UID: \"b8336ff5-7043-4f2a-8ab1-60c4713596e9\") " Oct 30 20:15:21.705586 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.705591 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8336ff5-7043-4f2a-8ab1-60c4713596e9-kube-api-access-p575b\" (UniqueName: \"kubernetes.io/projected/b8336ff5-7043-4f2a-8ab1-60c4713596e9-kube-api-access-p575b\") pod \"b8336ff5-7043-4f2a-8ab1-60c4713596e9\" (UID: \"b8336ff5-7043-4f2a-8ab1-60c4713596e9\") " Oct 30 20:15:21.705850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.705697 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b69782b-78e9-4baf-8bdf-bcbf9b78767d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b69782b-78e9-4baf-8bdf-bcbf9b78767d-maas-api-service-ca\") pod \"8b69782b-78e9-4baf-8bdf-bcbf9b78767d\" (UID: \"8b69782b-78e9-4baf-8bdf-bcbf9b78767d\") " Oct 30 20:15:21.705850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.705793 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b69782b-78e9-4baf-8bdf-bcbf9b78767d-kube-api-access-kmwgg\" (UniqueName: \"kubernetes.io/projected/8b69782b-78e9-4baf-8bdf-bcbf9b78767d-kube-api-access-kmwgg\") pod \"8b69782b-78e9-4baf-8bdf-bcbf9b78767d\" (UID: \"8b69782b-78e9-4baf-8bdf-bcbf9b78767d\") " Oct 30 20:15:21.705850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.705816 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d27d65d4-ebb6-4254-89c9-febcacbdf9f1-kube-api-access-2rttn\" (UniqueName: \"kubernetes.io/projected/d27d65d4-ebb6-4254-89c9-febcacbdf9f1-kube-api-access-2rttn\") pod \"d27d65d4-ebb6-4254-89c9-febcacbdf9f1\" (UID: \"d27d65d4-ebb6-4254-89c9-febcacbdf9f1\") " Oct 30 20:15:21.706020 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.705918 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d27d65d4-ebb6-4254-89c9-febcacbdf9f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27d65d4-ebb6-4254-89c9-febcacbdf9f1-maas-api-service-ca\") pod \"d27d65d4-ebb6-4254-89c9-febcacbdf9f1\" (UID: \"d27d65d4-ebb6-4254-89c9-febcacbdf9f1\") " Oct 30 20:15:21.706692 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.706587 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b69782b-78e9-4baf-8bdf-bcbf9b78767d-maas-api-service-ca" pod "8b69782b-78e9-4baf-8bdf-bcbf9b78767d" (UID: "8b69782b-78e9-4baf-8bdf-bcbf9b78767d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:21.706820 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.706764 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d27d65d4-ebb6-4254-89c9-febcacbdf9f1-maas-api-service-ca" pod "d27d65d4-ebb6-4254-89c9-febcacbdf9f1" (UID: "d27d65d4-ebb6-4254-89c9-febcacbdf9f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:21.707227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.707203 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8336ff5-7043-4f2a-8ab1-60c4713596e9-maas-api-service-ca" pod "b8336ff5-7043-4f2a-8ab1-60c4713596e9" (UID: "b8336ff5-7043-4f2a-8ab1-60c4713596e9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:21.711212 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.709557 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d27d65d4-ebb6-4254-89c9-febcacbdf9f1-kube-api-access-2rttn" pod "d27d65d4-ebb6-4254-89c9-febcacbdf9f1" (UID: "d27d65d4-ebb6-4254-89c9-febcacbdf9f1"). InnerVolumeSpecName "kube-api-access-2rttn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:21.711640 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.711611 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8336ff5-7043-4f2a-8ab1-60c4713596e9-kube-api-access-p575b" pod "b8336ff5-7043-4f2a-8ab1-60c4713596e9" (UID: "b8336ff5-7043-4f2a-8ab1-60c4713596e9"). InnerVolumeSpecName "kube-api-access-p575b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:21.713078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.713050 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b69782b-78e9-4baf-8bdf-bcbf9b78767d-kube-api-access-kmwgg" pod "8b69782b-78e9-4baf-8bdf-bcbf9b78767d" (UID: "8b69782b-78e9-4baf-8bdf-bcbf9b78767d"). InnerVolumeSpecName "kube-api-access-kmwgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:21.807160 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.807113 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" Oct 30 20:15:21.807160 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.807112 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d" event={"ID":"78e70c72-9983-4b2e-bd0c-176f50822feb","Type":"ContainerDied","Data":"b50eb55c1a357c9682a14e2263fdf36d10c771f8a4a1bee8bc939e8e862a30e3"} Oct 30 20:15:21.807385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.807222 5091 scope.go:122] "RemoveContainer" containerID="ace7e1be7c7e228f6a4f2c40c6b5f03aeb71ab9ee21887f7442848f767b4d2a6" Oct 30 20:15:21.808111 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.808083 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d27d65d4-ebb6-4254-89c9-febcacbdf9f1-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.808111 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.808111 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8336ff5-7043-4f2a-8ab1-60c4713596e9-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.808339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.808125 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p575b\" (UniqueName: \"kubernetes.io/projected/b8336ff5-7043-4f2a-8ab1-60c4713596e9-kube-api-access-p575b\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.808339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.808142 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b69782b-78e9-4baf-8bdf-bcbf9b78767d-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.808339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.808168 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmwgg\" (UniqueName: \"kubernetes.io/projected/8b69782b-78e9-4baf-8bdf-bcbf9b78767d-kube-api-access-kmwgg\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.808339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.808182 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rttn\" (UniqueName: \"kubernetes.io/projected/d27d65d4-ebb6-4254-89c9-febcacbdf9f1-kube-api-access-2rttn\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:21.808856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.808794 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" event={"ID":"b8336ff5-7043-4f2a-8ab1-60c4713596e9","Type":"ContainerDied","Data":"235a73b79e3091f0abcdf604c8709fcd7345c221104f8f722f1d5f18b86c7335"} Oct 30 20:15:21.808856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.808801 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p" Oct 30 20:15:21.810635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.810616 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" Oct 30 20:15:21.810763 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.810635 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx" event={"ID":"8b69782b-78e9-4baf-8bdf-bcbf9b78767d","Type":"ContainerDied","Data":"bb37d9963649d52075801471dacd988eb63052af2263c1e24756dbc0a599d976"} Oct 30 20:15:21.813119 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.813091 5091 generic.go:356] "Generic (PLEG): container finished" podID="67449cd5-2639-4625-b32e-ba3bec5a528b" containerID="f7faccef72da9ee56931a82f0c07b636dd48bd38810bf1e894f1c5826a748e69" exitCode=60 Oct 30 20:15:21.813406 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.813129 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" event={"ID":"67449cd5-2639-4625-b32e-ba3bec5a528b","Type":"ContainerDied","Data":"f7faccef72da9ee56931a82f0c07b636dd48bd38810bf1e894f1c5826a748e69"} Oct 30 20:15:21.815524 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.815505 5091 scope.go:122] "RemoveContainer" containerID="f7faccef72da9ee56931a82f0c07b636dd48bd38810bf1e894f1c5826a748e69" Oct 30 20:15:21.815913 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:21.815892 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v_odh-ai-gateway-infra(67449cd5-2639-4625-b32e-ba3bec5a528b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" podUID="67449cd5-2639-4625-b32e-ba3bec5a528b" Oct 30 20:15:21.816402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.816377 5091 generic.go:356] "Generic (PLEG): container finished" podID="2ea189f9-f225-4fdb-8948-14cb53111cd5" containerID="c19add837ad2406e84e69e8ce6dd2edc9d24ff5295b6b2dc4eb92ca80a0f9f45" exitCode=60 Oct 30 20:15:21.816508 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.816402 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" event={"ID":"2ea189f9-f225-4fdb-8948-14cb53111cd5","Type":"ContainerDied","Data":"c19add837ad2406e84e69e8ce6dd2edc9d24ff5295b6b2dc4eb92ca80a0f9f45"} Oct 30 20:15:21.816788 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.816771 5091 scope.go:122] "RemoveContainer" containerID="c19add837ad2406e84e69e8ce6dd2edc9d24ff5295b6b2dc4eb92ca80a0f9f45" Oct 30 20:15:21.817347 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:21.817244 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw_odh-ai-gateway-infra(2ea189f9-f225-4fdb-8948-14cb53111cd5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" podUID="2ea189f9-f225-4fdb-8948-14cb53111cd5" Oct 30 20:15:21.819037 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.819012 5091 generic.go:356] "Generic (PLEG): container finished" podID="755783df-2b63-4013-9133-d39c004e577f" containerID="b287a099a3b6b72cbb9c4365b63d079c22947391f63255a37b17a02ce3603ae6" exitCode=60 Oct 30 20:15:21.819192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.819175 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" event={"ID":"755783df-2b63-4013-9133-d39c004e577f","Type":"ContainerDied","Data":"b287a099a3b6b72cbb9c4365b63d079c22947391f63255a37b17a02ce3603ae6"} Oct 30 20:15:21.819582 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.819565 5091 scope.go:122] "RemoveContainer" containerID="b287a099a3b6b72cbb9c4365b63d079c22947391f63255a37b17a02ce3603ae6" Oct 30 20:15:21.819884 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:21.819831 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl_odh-ai-gateway-infra(755783df-2b63-4013-9133-d39c004e577f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" podUID="755783df-2b63-4013-9133-d39c004e577f" Oct 30 20:15:21.823881 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.823866 5091 scope.go:122] "RemoveContainer" containerID="7fd4f5c8ded3a7811e28436e6132fa9eb2cb385f60096febc8801bc1faf979eb" Oct 30 20:15:21.825292 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.825271 5091 generic.go:356] "Generic (PLEG): container finished" podID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" containerID="ba39530866b38d725dc8c5f63f06efc9a1805c491bb34c963bff264deb9d3f2e" exitCode=60 Oct 30 20:15:21.825365 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.825343 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" event={"ID":"f93e44d7-d7d5-4beb-9332-403ddc9bf336","Type":"ContainerDied","Data":"ba39530866b38d725dc8c5f63f06efc9a1805c491bb34c963bff264deb9d3f2e"} Oct 30 20:15:21.825699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.825678 5091 scope.go:122] "RemoveContainer" containerID="ba39530866b38d725dc8c5f63f06efc9a1805c491bb34c963bff264deb9d3f2e" Oct 30 20:15:21.825989 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:21.825964 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc_odh-ai-gateway-infra(f93e44d7-d7d5-4beb-9332-403ddc9bf336)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" podUID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" Oct 30 20:15:21.827385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.827360 5091 generic.go:356] "Generic (PLEG): container finished" podID="387559a4-9827-4a53-90ae-3f3f0852d25d" containerID="63b792b48d3ddf3b6f8e61e6a6e05401e1dc15b1aee4e08e8e0ce3808d1695b9" exitCode=60 Oct 30 20:15:21.827474 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.827428 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" event={"ID":"387559a4-9827-4a53-90ae-3f3f0852d25d","Type":"ContainerDied","Data":"63b792b48d3ddf3b6f8e61e6a6e05401e1dc15b1aee4e08e8e0ce3808d1695b9"} Oct 30 20:15:21.827734 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.827715 5091 scope.go:122] "RemoveContainer" containerID="63b792b48d3ddf3b6f8e61e6a6e05401e1dc15b1aee4e08e8e0ce3808d1695b9" Oct 30 20:15:21.827954 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:21.827931 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct_odh-ai-gateway-infra(387559a4-9827-4a53-90ae-3f3f0852d25d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" podUID="387559a4-9827-4a53-90ae-3f3f0852d25d" Oct 30 20:15:21.829525 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.829506 5091 generic.go:356] "Generic (PLEG): container finished" podID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" containerID="8fab7b830d73e0a77b396d584d23155b1325a17b868d62b4567c5071b219a3e5" exitCode=60 Oct 30 20:15:21.829616 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.829587 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" event={"ID":"8fb65533-38aa-48c4-a374-fcf2f5e96f99","Type":"ContainerDied","Data":"8fab7b830d73e0a77b396d584d23155b1325a17b868d62b4567c5071b219a3e5"} Oct 30 20:15:21.829965 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.829944 5091 scope.go:122] "RemoveContainer" containerID="8fab7b830d73e0a77b396d584d23155b1325a17b868d62b4567c5071b219a3e5" Oct 30 20:15:21.830248 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:21.830225 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm_odh-ai-gateway-infra(8fb65533-38aa-48c4-a374-fcf2f5e96f99)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" podUID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" Oct 30 20:15:21.831997 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.831974 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" event={"ID":"4eafdae9-7fcd-4579-824b-60bf84bb9b7f","Type":"ContainerStarted","Data":"5c396d0d0d50df725b20e69f96a81af10c88a90e390b271fb2095f1661918b23"} Oct 30 20:15:21.834427 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.834411 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" Oct 30 20:15:21.834534 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.834448 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd" event={"ID":"d27d65d4-ebb6-4254-89c9-febcacbdf9f1","Type":"ContainerDied","Data":"c40c6b2b7c7f5df24b3445584f3d7a6f90a79018775654a287a8430e0376bb4f"} Oct 30 20:15:21.836876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.836859 5091 scope.go:122] "RemoveContainer" containerID="c4a58bd90a5e408d5d35818ffec3302bad4aac3f0773b9692f011b8a516cae27" Oct 30 20:15:21.836940 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.836892 5091 generic.go:356] "Generic (PLEG): container finished" podID="ee32ddbf-9640-4efd-b2a5-babcfb306088" containerID="e0656ea9afd98958113d06b349b24abcc8e32479493e1d2d961b498621c1f425" exitCode=60 Oct 30 20:15:21.836999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.836952 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" event={"ID":"ee32ddbf-9640-4efd-b2a5-babcfb306088","Type":"ContainerDied","Data":"e0656ea9afd98958113d06b349b24abcc8e32479493e1d2d961b498621c1f425"} Oct 30 20:15:21.837376 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.837356 5091 scope.go:122] "RemoveContainer" containerID="e0656ea9afd98958113d06b349b24abcc8e32479493e1d2d961b498621c1f425" Oct 30 20:15:21.837590 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:21.837569 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6_odh-ai-gateway-infra(ee32ddbf-9640-4efd-b2a5-babcfb306088)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" podUID="ee32ddbf-9640-4efd-b2a5-babcfb306088" Oct 30 20:15:21.838541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.838492 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" event={"ID":"87e33682-f349-450b-ab65-1b60d304a6cf","Type":"ContainerDied","Data":"16d8dea45947d1d07cfb2825c499401ece1cdd6a200afdade6bbea2ac0e4e26a"} Oct 30 20:15:21.838541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.838525 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm" Oct 30 20:15:21.848654 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.848618 5091 scope.go:122] "RemoveContainer" containerID="4359cdb70a9f6f96a6015f9a4602aafda71910976fac991ab52ea927faef2ad4" Oct 30 20:15:21.952170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.943609 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:21.979476 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.979375 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx"] Oct 30 20:15:21.986645 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.986615 5091 scope.go:122] "RemoveContainer" containerID="4d3ca8b3f178e9621270db788279994ba03ec76906628e6c0bb3a6194db437bd" Oct 30 20:15:21.987941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:21.987916 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-981ee5-29889855-j4psx"] Oct 30 20:15:22.007623 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.007505 5091 scope.go:122] "RemoveContainer" containerID="d518b80f9487d20c1e27524014d8510b3151c85add30065fdc46eeeb6e569d83" Oct 30 20:15:22.012272 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.012221 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd"] Oct 30 20:15:22.017656 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.017624 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-5f9b3c-29889855-clpmd"] Oct 30 20:15:22.029601 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.029578 5091 scope.go:122] "RemoveContainer" containerID="5bcc7b29b6360ff50c0212de016b853d5c37197d5dbde8921e8f7fb4a00ef8fc" Oct 30 20:15:22.030953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.030929 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d"] Oct 30 20:15:22.035452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.035424 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c9bce8-29889855-mqw2d"] Oct 30 20:15:22.044536 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.044518 5091 scope.go:122] "RemoveContainer" containerID="f70ce32f693cb1edbfd8e1cc46058eb5803e02afbba4b84364824720004ef466" Oct 30 20:15:22.061649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.061626 5091 scope.go:122] "RemoveContainer" containerID="b3b874c4b57483016809f6f6bf50016c6a56399d516ff31b0615f3425d9dd39b" Oct 30 20:15:22.077544 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.077521 5091 scope.go:122] "RemoveContainer" containerID="2539f17d62d021012e309681d2e6c9b1f05171e799271558c6c94c0e74579f86" Oct 30 20:15:22.088429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.088409 5091 scope.go:122] "RemoveContainer" containerID="5d0ca013a49dbf4aa2b7c2101ce60a5700645908e822083179cc5aba34f37e53" Oct 30 20:15:22.102661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.102639 5091 scope.go:122] "RemoveContainer" containerID="1d0067a0c832f8956c02d6afa30aaae3226c47f947edf95263c9afc8cf64cef6" Oct 30 20:15:22.161204 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.161169 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm"] Oct 30 20:15:22.165502 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.165478 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-969878-29889855-d2dfm"] Oct 30 20:15:22.204004 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.203965 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p"] Oct 30 20:15:22.216529 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.216495 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-54d4c4-29889855-wwg2p"] Oct 30 20:15:22.445739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.445658 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" path="/var/lib/kubelet/pods/1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c/volumes" Oct 30 20:15:22.446195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.446138 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a5e04ce-a8ad-4b44-bd00-624a49388708" path="/var/lib/kubelet/pods/2a5e04ce-a8ad-4b44-bd00-624a49388708/volumes" Oct 30 20:15:22.446649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.446634 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" path="/var/lib/kubelet/pods/3d37c05b-93b8-4c8c-84d1-9e186d7ec265/volumes" Oct 30 20:15:22.447424 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.447404 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78e70c72-9983-4b2e-bd0c-176f50822feb" path="/var/lib/kubelet/pods/78e70c72-9983-4b2e-bd0c-176f50822feb/volumes" Oct 30 20:15:22.447836 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.447820 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e33682-f349-450b-ab65-1b60d304a6cf" path="/var/lib/kubelet/pods/87e33682-f349-450b-ab65-1b60d304a6cf/volumes" Oct 30 20:15:22.448256 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.448240 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" path="/var/lib/kubelet/pods/8b69782b-78e9-4baf-8bdf-bcbf9b78767d/volumes" Oct 30 20:15:22.448958 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.448940 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" path="/var/lib/kubelet/pods/b7d9f0fd-4bb0-4773-8333-92f168e683c3/volumes" Oct 30 20:15:22.449369 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.449354 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8336ff5-7043-4f2a-8ab1-60c4713596e9" path="/var/lib/kubelet/pods/b8336ff5-7043-4f2a-8ab1-60c4713596e9/volumes" Oct 30 20:15:22.449754 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.449740 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d27d65d4-ebb6-4254-89c9-febcacbdf9f1" path="/var/lib/kubelet/pods/d27d65d4-ebb6-4254-89c9-febcacbdf9f1/volumes" Oct 30 20:15:22.858334 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.858302 5091 generic.go:356] "Generic (PLEG): container finished" podID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" containerID="5c396d0d0d50df725b20e69f96a81af10c88a90e390b271fb2095f1661918b23" exitCode=60 Oct 30 20:15:22.858818 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.858368 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" event={"ID":"4eafdae9-7fcd-4579-824b-60bf84bb9b7f","Type":"ContainerDied","Data":"5c396d0d0d50df725b20e69f96a81af10c88a90e390b271fb2095f1661918b23"} Oct 30 20:15:22.858818 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.858423 5091 scope.go:122] "RemoveContainer" containerID="250846c9f7cae177aa121adc9786a0544afb17d70a28204ec5e72f9caec9b3a9" Oct 30 20:15:22.858818 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.858819 5091 scope.go:122] "RemoveContainer" containerID="5c396d0d0d50df725b20e69f96a81af10c88a90e390b271fb2095f1661918b23" Oct 30 20:15:22.859238 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:22.859210 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28_odh-ai-gateway-infra(4eafdae9-7fcd-4579-824b-60bf84bb9b7f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" podUID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" Oct 30 20:15:22.874919 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.874894 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v"] Oct 30 20:15:22.906727 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:22.906692 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28"] Oct 30 20:15:23.039527 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.039464 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl"] Oct 30 20:15:23.043445 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.043343 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:23.113492 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.113402 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw"] Oct 30 20:15:23.125606 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.125575 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct"] Oct 30 20:15:23.138952 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.138926 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc"] Oct 30 20:15:23.199591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.199461 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm"] Oct 30 20:15:23.242077 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.236792 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6"] Oct 30 20:15:23.291488 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.290997 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" Oct 30 20:15:23.460419 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.452505 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67449cd5-2639-4625-b32e-ba3bec5a528b-kube-api-access-b4hgd\" (UniqueName: \"kubernetes.io/projected/67449cd5-2639-4625-b32e-ba3bec5a528b-kube-api-access-b4hgd\") pod \"67449cd5-2639-4625-b32e-ba3bec5a528b\" (UID: \"67449cd5-2639-4625-b32e-ba3bec5a528b\") " Oct 30 20:15:23.460419 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.452820 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67449cd5-2639-4625-b32e-ba3bec5a528b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67449cd5-2639-4625-b32e-ba3bec5a528b-maas-api-service-ca\") pod \"67449cd5-2639-4625-b32e-ba3bec5a528b\" (UID: \"67449cd5-2639-4625-b32e-ba3bec5a528b\") " Oct 30 20:15:23.460419 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.453934 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67449cd5-2639-4625-b32e-ba3bec5a528b-maas-api-service-ca" pod "67449cd5-2639-4625-b32e-ba3bec5a528b" (UID: "67449cd5-2639-4625-b32e-ba3bec5a528b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:23.485348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.481388 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67449cd5-2639-4625-b32e-ba3bec5a528b-kube-api-access-b4hgd" pod "67449cd5-2639-4625-b32e-ba3bec5a528b" (UID: "67449cd5-2639-4625-b32e-ba3bec5a528b"). InnerVolumeSpecName "kube-api-access-b4hgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:23.556598 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.555374 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/67449cd5-2639-4625-b32e-ba3bec5a528b-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:23.556598 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.555410 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4hgd\" (UniqueName: \"kubernetes.io/projected/67449cd5-2639-4625-b32e-ba3bec5a528b-kube-api-access-b4hgd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:23.568850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.567610 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" Oct 30 20:15:23.571078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.570744 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" Oct 30 20:15:23.573097 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.573032 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" Oct 30 20:15:23.577869 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.577803 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" Oct 30 20:15:23.639747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.639416 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" Oct 30 20:15:23.643137 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.643114 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" Oct 30 20:15:23.656689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.656654 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea189f9-f225-4fdb-8948-14cb53111cd5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea189f9-f225-4fdb-8948-14cb53111cd5-maas-api-service-ca\") pod \"2ea189f9-f225-4fdb-8948-14cb53111cd5\" (UID: \"2ea189f9-f225-4fdb-8948-14cb53111cd5\") " Oct 30 20:15:23.656922 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.656900 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f93e44d7-d7d5-4beb-9332-403ddc9bf336-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f93e44d7-d7d5-4beb-9332-403ddc9bf336-maas-api-service-ca\") pod \"f93e44d7-d7d5-4beb-9332-403ddc9bf336\" (UID: \"f93e44d7-d7d5-4beb-9332-403ddc9bf336\") " Oct 30 20:15:23.657053 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.656960 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93e44d7-d7d5-4beb-9332-403ddc9bf336-kube-api-access-2r9x2\" (UniqueName: \"kubernetes.io/projected/f93e44d7-d7d5-4beb-9332-403ddc9bf336-kube-api-access-2r9x2\") pod \"f93e44d7-d7d5-4beb-9332-403ddc9bf336\" (UID: \"f93e44d7-d7d5-4beb-9332-403ddc9bf336\") " Oct 30 20:15:23.657120 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.657043 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea189f9-f225-4fdb-8948-14cb53111cd5-maas-api-service-ca" pod "2ea189f9-f225-4fdb-8948-14cb53111cd5" (UID: "2ea189f9-f225-4fdb-8948-14cb53111cd5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:23.657120 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.657087 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/755783df-2b63-4013-9133-d39c004e577f-kube-api-access-c86zj\" (UniqueName: \"kubernetes.io/projected/755783df-2b63-4013-9133-d39c004e577f-kube-api-access-c86zj\") pod \"755783df-2b63-4013-9133-d39c004e577f\" (UID: \"755783df-2b63-4013-9133-d39c004e577f\") " Oct 30 20:15:23.657255 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.657133 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/755783df-2b63-4013-9133-d39c004e577f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/755783df-2b63-4013-9133-d39c004e577f-maas-api-service-ca\") pod \"755783df-2b63-4013-9133-d39c004e577f\" (UID: \"755783df-2b63-4013-9133-d39c004e577f\") " Oct 30 20:15:23.657255 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.657213 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387559a4-9827-4a53-90ae-3f3f0852d25d-kube-api-access-46t2w\" (UniqueName: \"kubernetes.io/projected/387559a4-9827-4a53-90ae-3f3f0852d25d-kube-api-access-46t2w\") pod \"387559a4-9827-4a53-90ae-3f3f0852d25d\" (UID: \"387559a4-9827-4a53-90ae-3f3f0852d25d\") " Oct 30 20:15:23.657255 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.657239 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea189f9-f225-4fdb-8948-14cb53111cd5-kube-api-access-4jt9l\" (UniqueName: \"kubernetes.io/projected/2ea189f9-f225-4fdb-8948-14cb53111cd5-kube-api-access-4jt9l\") pod \"2ea189f9-f225-4fdb-8948-14cb53111cd5\" (UID: \"2ea189f9-f225-4fdb-8948-14cb53111cd5\") " Oct 30 20:15:23.657418 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.657272 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387559a4-9827-4a53-90ae-3f3f0852d25d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/387559a4-9827-4a53-90ae-3f3f0852d25d-maas-api-service-ca\") pod \"387559a4-9827-4a53-90ae-3f3f0852d25d\" (UID: \"387559a4-9827-4a53-90ae-3f3f0852d25d\") " Oct 30 20:15:23.659078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.657506 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93e44d7-d7d5-4beb-9332-403ddc9bf336-maas-api-service-ca" pod "f93e44d7-d7d5-4beb-9332-403ddc9bf336" (UID: "f93e44d7-d7d5-4beb-9332-403ddc9bf336"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:23.659078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.657543 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/755783df-2b63-4013-9133-d39c004e577f-maas-api-service-ca" pod "755783df-2b63-4013-9133-d39c004e577f" (UID: "755783df-2b63-4013-9133-d39c004e577f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:23.659078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.658190 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/755783df-2b63-4013-9133-d39c004e577f-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:23.659078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.658218 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ea189f9-f225-4fdb-8948-14cb53111cd5-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:23.659078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.658232 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f93e44d7-d7d5-4beb-9332-403ddc9bf336-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:23.659078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.658521 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387559a4-9827-4a53-90ae-3f3f0852d25d-maas-api-service-ca" pod "387559a4-9827-4a53-90ae-3f3f0852d25d" (UID: "387559a4-9827-4a53-90ae-3f3f0852d25d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:23.674208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.674113 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387559a4-9827-4a53-90ae-3f3f0852d25d-kube-api-access-46t2w" pod "387559a4-9827-4a53-90ae-3f3f0852d25d" (UID: "387559a4-9827-4a53-90ae-3f3f0852d25d"). InnerVolumeSpecName "kube-api-access-46t2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:23.674460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.674365 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93e44d7-d7d5-4beb-9332-403ddc9bf336-kube-api-access-2r9x2" pod "f93e44d7-d7d5-4beb-9332-403ddc9bf336" (UID: "f93e44d7-d7d5-4beb-9332-403ddc9bf336"). InnerVolumeSpecName "kube-api-access-2r9x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:23.674460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.674428 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/755783df-2b63-4013-9133-d39c004e577f-kube-api-access-c86zj" pod "755783df-2b63-4013-9133-d39c004e577f" (UID: "755783df-2b63-4013-9133-d39c004e577f"). InnerVolumeSpecName "kube-api-access-c86zj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:23.674685 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.674665 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea189f9-f225-4fdb-8948-14cb53111cd5-kube-api-access-4jt9l" pod "2ea189f9-f225-4fdb-8948-14cb53111cd5" (UID: "2ea189f9-f225-4fdb-8948-14cb53111cd5"). InnerVolumeSpecName "kube-api-access-4jt9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:23.759195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.759127 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee32ddbf-9640-4efd-b2a5-babcfb306088-kube-api-access-szcpv\" (UniqueName: \"kubernetes.io/projected/ee32ddbf-9640-4efd-b2a5-babcfb306088-kube-api-access-szcpv\") pod \"ee32ddbf-9640-4efd-b2a5-babcfb306088\" (UID: \"ee32ddbf-9640-4efd-b2a5-babcfb306088\") " Oct 30 20:15:23.759387 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.759356 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb65533-38aa-48c4-a374-fcf2f5e96f99-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb65533-38aa-48c4-a374-fcf2f5e96f99-maas-api-service-ca\") pod \"8fb65533-38aa-48c4-a374-fcf2f5e96f99\" (UID: \"8fb65533-38aa-48c4-a374-fcf2f5e96f99\") " Oct 30 20:15:23.759490 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.759472 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee32ddbf-9640-4efd-b2a5-babcfb306088-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee32ddbf-9640-4efd-b2a5-babcfb306088-maas-api-service-ca\") pod \"ee32ddbf-9640-4efd-b2a5-babcfb306088\" (UID: \"ee32ddbf-9640-4efd-b2a5-babcfb306088\") " Oct 30 20:15:23.759981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.759774 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb65533-38aa-48c4-a374-fcf2f5e96f99-kube-api-access-5b957\" (UniqueName: \"kubernetes.io/projected/8fb65533-38aa-48c4-a374-fcf2f5e96f99-kube-api-access-5b957\") pod \"8fb65533-38aa-48c4-a374-fcf2f5e96f99\" (UID: \"8fb65533-38aa-48c4-a374-fcf2f5e96f99\") " Oct 30 20:15:23.759981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.759943 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee32ddbf-9640-4efd-b2a5-babcfb306088-maas-api-service-ca" pod "ee32ddbf-9640-4efd-b2a5-babcfb306088" (UID: "ee32ddbf-9640-4efd-b2a5-babcfb306088"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:23.760513 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.760491 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb65533-38aa-48c4-a374-fcf2f5e96f99-maas-api-service-ca" pod "8fb65533-38aa-48c4-a374-fcf2f5e96f99" (UID: "8fb65533-38aa-48c4-a374-fcf2f5e96f99"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:23.760666 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.760589 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r9x2\" (UniqueName: \"kubernetes.io/projected/f93e44d7-d7d5-4beb-9332-403ddc9bf336-kube-api-access-2r9x2\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:23.760776 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.760761 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee32ddbf-9640-4efd-b2a5-babcfb306088-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:23.760906 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.760892 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c86zj\" (UniqueName: \"kubernetes.io/projected/755783df-2b63-4013-9133-d39c004e577f-kube-api-access-c86zj\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:23.760999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.760987 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46t2w\" (UniqueName: \"kubernetes.io/projected/387559a4-9827-4a53-90ae-3f3f0852d25d-kube-api-access-46t2w\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:23.761086 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.761073 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jt9l\" (UniqueName: \"kubernetes.io/projected/2ea189f9-f225-4fdb-8948-14cb53111cd5-kube-api-access-4jt9l\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:23.761208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.761192 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/387559a4-9827-4a53-90ae-3f3f0852d25d-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:23.762044 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.762017 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee32ddbf-9640-4efd-b2a5-babcfb306088-kube-api-access-szcpv" pod "ee32ddbf-9640-4efd-b2a5-babcfb306088" (UID: "ee32ddbf-9640-4efd-b2a5-babcfb306088"). InnerVolumeSpecName "kube-api-access-szcpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:23.762651 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.762624 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb65533-38aa-48c4-a374-fcf2f5e96f99-kube-api-access-5b957" pod "8fb65533-38aa-48c4-a374-fcf2f5e96f99" (UID: "8fb65533-38aa-48c4-a374-fcf2f5e96f99"). InnerVolumeSpecName "kube-api-access-5b957". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:23.862961 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.862927 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b957\" (UniqueName: \"kubernetes.io/projected/8fb65533-38aa-48c4-a374-fcf2f5e96f99-kube-api-access-5b957\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:23.862961 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.862960 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szcpv\" (UniqueName: \"kubernetes.io/projected/ee32ddbf-9640-4efd-b2a5-babcfb306088-kube-api-access-szcpv\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:23.862961 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.862971 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8fb65533-38aa-48c4-a374-fcf2f5e96f99-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:23.897838 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.897804 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" Oct 30 20:15:23.900577 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.897813 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v" event={"ID":"67449cd5-2639-4625-b32e-ba3bec5a528b","Type":"ContainerDied","Data":"b6af598d1312f72f70f2d3451153ac3858055b1fdb7b985bb52c384f088e1fe6"} Oct 30 20:15:23.900753 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.900729 5091 scope.go:122] "RemoveContainer" containerID="f7faccef72da9ee56931a82f0c07b636dd48bd38810bf1e894f1c5826a748e69" Oct 30 20:15:23.901999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.901964 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" event={"ID":"2ea189f9-f225-4fdb-8948-14cb53111cd5","Type":"ContainerDied","Data":"3477becef01b5e48af6cb6cf5d5e453d19732979dd723e528eaf07bb8c9d9c91"} Oct 30 20:15:23.902117 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.902050 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw" Oct 30 20:15:23.904476 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.904441 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" event={"ID":"755783df-2b63-4013-9133-d39c004e577f","Type":"ContainerDied","Data":"edada6ee58e3fd9db1c59d1b4446e72994b732f41e24b63a5ea70cbe7c671e47"} Oct 30 20:15:23.904567 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.904556 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl" Oct 30 20:15:23.911631 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.911611 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" Oct 30 20:15:23.911733 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.911626 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc" event={"ID":"f93e44d7-d7d5-4beb-9332-403ddc9bf336","Type":"ContainerDied","Data":"ba39929e3b274036365e5726479a5d018be6991c5330f1dade6c632ac73275cf"} Oct 30 20:15:23.913956 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.913926 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" event={"ID":"387559a4-9827-4a53-90ae-3f3f0852d25d","Type":"ContainerDied","Data":"48c4b43dfded5e400e900d71a3dbaeaedb113521dc35ac4ca97a921f570c813b"} Oct 30 20:15:23.914054 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.914016 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct" Oct 30 20:15:23.916210 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.916188 5091 scope.go:122] "RemoveContainer" containerID="c19add837ad2406e84e69e8ce6dd2edc9d24ff5295b6b2dc4eb92ca80a0f9f45" Oct 30 20:15:23.917396 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.917316 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" event={"ID":"8fb65533-38aa-48c4-a374-fcf2f5e96f99","Type":"ContainerDied","Data":"913a8e9a93d2a42871908c7629bcaf16068ca3eff9e6684236e911ab0cf99844"} Oct 30 20:15:23.917396 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.917321 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm" Oct 30 20:15:23.921625 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.921579 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" event={"ID":"ee32ddbf-9640-4efd-b2a5-babcfb306088","Type":"ContainerDied","Data":"43dc6447859b302fd7097330b4ebb4268afbcf252840558c7abf2c3739e4e373"} Oct 30 20:15:23.921700 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.921670 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6" Oct 30 20:15:23.930057 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.930037 5091 scope.go:122] "RemoveContainer" containerID="b287a099a3b6b72cbb9c4365b63d079c22947391f63255a37b17a02ce3603ae6" Oct 30 20:15:23.942945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.942910 5091 scope.go:122] "RemoveContainer" containerID="ba39530866b38d725dc8c5f63f06efc9a1805c491bb34c963bff264deb9d3f2e" Oct 30 20:15:23.967123 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.967093 5091 scope.go:122] "RemoveContainer" containerID="63b792b48d3ddf3b6f8e61e6a6e05401e1dc15b1aee4e08e8e0ce3808d1695b9" Oct 30 20:15:23.984916 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.984888 5091 scope.go:122] "RemoveContainer" containerID="8fab7b830d73e0a77b396d584d23155b1325a17b868d62b4567c5071b219a3e5" Oct 30 20:15:23.991209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:23.991181 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl"] Oct 30 20:15:24.004213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.004188 5091 scope.go:122] "RemoveContainer" containerID="e0656ea9afd98958113d06b349b24abcc8e32479493e1d2d961b498621c1f425" Oct 30 20:15:24.005759 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.005700 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-b7431c-29889855-q52hl"] Oct 30 20:15:24.023997 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.023966 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw"] Oct 30 20:15:24.029641 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.029617 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-7a745c-29889855-4zthw"] Oct 30 20:15:24.045683 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.045650 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v"] Oct 30 20:15:24.133203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.115798 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:24.161726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.161693 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-2a2286-29889855-c2n5v"] Oct 30 20:15:24.168796 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.168746 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct"] Oct 30 20:15:24.182000 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.181823 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-517027-29889855-mb9ct"] Oct 30 20:15:24.185844 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.185815 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc"] Oct 30 20:15:24.186163 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.186123 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" Oct 30 20:15:24.194095 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.194061 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-4d5758-29889855-57sbc"] Oct 30 20:15:24.214920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.214890 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm"] Oct 30 20:15:24.218902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.218881 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-c1fdd6-29889855-zh8hm"] Oct 30 20:15:24.224840 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.224815 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eafdae9-7fcd-4579-824b-60bf84bb9b7f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eafdae9-7fcd-4579-824b-60bf84bb9b7f-maas-api-service-ca\") pod \"4eafdae9-7fcd-4579-824b-60bf84bb9b7f\" (UID: \"4eafdae9-7fcd-4579-824b-60bf84bb9b7f\") " Oct 30 20:15:24.225034 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.225018 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eafdae9-7fcd-4579-824b-60bf84bb9b7f-kube-api-access-f8698\" (UniqueName: \"kubernetes.io/projected/4eafdae9-7fcd-4579-824b-60bf84bb9b7f-kube-api-access-f8698\") pod \"4eafdae9-7fcd-4579-824b-60bf84bb9b7f\" (UID: \"4eafdae9-7fcd-4579-824b-60bf84bb9b7f\") " Oct 30 20:15:24.225206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.225142 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eafdae9-7fcd-4579-824b-60bf84bb9b7f-maas-api-service-ca" pod "4eafdae9-7fcd-4579-824b-60bf84bb9b7f" (UID: "4eafdae9-7fcd-4579-824b-60bf84bb9b7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:24.225768 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.225747 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4eafdae9-7fcd-4579-824b-60bf84bb9b7f-maas-api-service-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:24.227289 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.227266 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eafdae9-7fcd-4579-824b-60bf84bb9b7f-kube-api-access-f8698" pod "4eafdae9-7fcd-4579-824b-60bf84bb9b7f" (UID: "4eafdae9-7fcd-4579-824b-60bf84bb9b7f"). InnerVolumeSpecName "kube-api-access-f8698". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:24.234181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.234159 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6"] Oct 30 20:15:24.239536 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.239515 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-75b361-29889855-fpgz6"] Oct 30 20:15:24.327580 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.327545 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8698\" (UniqueName: \"kubernetes.io/projected/4eafdae9-7fcd-4579-824b-60bf84bb9b7f-kube-api-access-f8698\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:24.443751 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.443666 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea189f9-f225-4fdb-8948-14cb53111cd5" path="/var/lib/kubelet/pods/2ea189f9-f225-4fdb-8948-14cb53111cd5/volumes" Oct 30 20:15:24.444140 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.444124 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="387559a4-9827-4a53-90ae-3f3f0852d25d" path="/var/lib/kubelet/pods/387559a4-9827-4a53-90ae-3f3f0852d25d/volumes" Oct 30 20:15:24.444621 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.444598 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67449cd5-2639-4625-b32e-ba3bec5a528b" path="/var/lib/kubelet/pods/67449cd5-2639-4625-b32e-ba3bec5a528b/volumes" Oct 30 20:15:24.445448 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.445413 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="755783df-2b63-4013-9133-d39c004e577f" path="/var/lib/kubelet/pods/755783df-2b63-4013-9133-d39c004e577f/volumes" Oct 30 20:15:24.446007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.445989 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" path="/var/lib/kubelet/pods/8fb65533-38aa-48c4-a374-fcf2f5e96f99/volumes" Oct 30 20:15:24.446598 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.446580 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee32ddbf-9640-4efd-b2a5-babcfb306088" path="/var/lib/kubelet/pods/ee32ddbf-9640-4efd-b2a5-babcfb306088/volumes" Oct 30 20:15:24.447580 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.447562 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" path="/var/lib/kubelet/pods/f93e44d7-d7d5-4beb-9332-403ddc9bf336/volumes" Oct 30 20:15:24.939876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.939843 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" Oct 30 20:15:24.940380 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.939875 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28" event={"ID":"4eafdae9-7fcd-4579-824b-60bf84bb9b7f","Type":"ContainerDied","Data":"b45bf52a26b95fb23b7295ed0e00d7ad3ca78f459d4e9088f9e6e36c85ed883b"} Oct 30 20:15:24.940380 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.939932 5091 scope.go:122] "RemoveContainer" containerID="5c396d0d0d50df725b20e69f96a81af10c88a90e390b271fb2095f1661918b23" Oct 30 20:15:24.993168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.991503 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28"] Oct 30 20:15:25.001254 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:24.997424 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:25.043320 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.039026 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-68685b-29889855-h4v28"] Oct 30 20:15:25.326678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.326636 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" podUID="af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c" containerName="payload-processing" containerID="cri-o://787bb21fae10d114351909eaba5a774d7d5ffed55967fe65344373866492caa5" gracePeriod=25 Oct 30 20:15:25.622970 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.622944 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:25.647503 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.647458 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-kube-api-access-8kbmj\" (UniqueName: \"kubernetes.io/projected/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-kube-api-access-8kbmj\") pod \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " Oct 30 20:15:25.647656 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.647564 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-routing-overlay-volume\") pod \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " Oct 30 20:15:25.647716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.647664 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-provider-credentials\") pod \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " Oct 30 20:15:25.647716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.647703 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-plugins-config-volume\") pod \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\" (UID: \"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c\") " Oct 30 20:15:25.648884 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.648839 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-routing-overlay-volume" pod "af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c" (UID: "af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:25.649020 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.648890 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-plugins-config-volume" pod "af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c" (UID: "af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:25.651378 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.651349 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-provider-credentials" pod "af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c" (UID: "af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:25.651846 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.651809 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-kube-api-access-8kbmj" pod "af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c" (UID: "af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c"). InnerVolumeSpecName "kube-api-access-8kbmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:25.750867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.750821 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:25.750867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.750864 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:25.751093 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.750879 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kbmj\" (UniqueName: \"kubernetes.io/projected/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-kube-api-access-8kbmj\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:25.751093 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.750893 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:25.799168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.794731 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:25.953137 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.953034 5091 generic.go:356] "Generic (PLEG): container finished" podID="af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c" containerID="787bb21fae10d114351909eaba5a774d7d5ffed55967fe65344373866492caa5" exitCode=0 Oct 30 20:15:25.953137 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.953080 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" Oct 30 20:15:25.953137 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.953113 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" event={"ID":"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c","Type":"ContainerDied","Data":"787bb21fae10d114351909eaba5a774d7d5ffed55967fe65344373866492caa5"} Oct 30 20:15:25.953780 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.953208 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt" event={"ID":"af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c","Type":"ContainerDied","Data":"0e3d46cf9f9c028e3c8e904c0ba6ab4013cf0c5e06695f8c54dc83b6e4364c79"} Oct 30 20:15:25.953780 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.953243 5091 scope.go:122] "RemoveContainer" containerID="787bb21fae10d114351909eaba5a774d7d5ffed55967fe65344373866492caa5" Oct 30 20:15:25.968222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.968201 5091 scope.go:122] "RemoveContainer" containerID="787bb21fae10d114351909eaba5a774d7d5ffed55967fe65344373866492caa5" Oct 30 20:15:25.968526 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:25.968507 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"787bb21fae10d114351909eaba5a774d7d5ffed55967fe65344373866492caa5\": container with ID starting with 787bb21fae10d114351909eaba5a774d7d5ffed55967fe65344373866492caa5 not found: ID does not exist" containerID="787bb21fae10d114351909eaba5a774d7d5ffed55967fe65344373866492caa5" Oct 30 20:15:25.968611 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.968535 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"787bb21fae10d114351909eaba5a774d7d5ffed55967fe65344373866492caa5"} err="failed to get container status \"787bb21fae10d114351909eaba5a774d7d5ffed55967fe65344373866492caa5\": rpc error: code = NotFound desc = could not find container \"787bb21fae10d114351909eaba5a774d7d5ffed55967fe65344373866492caa5\": container with ID starting with 787bb21fae10d114351909eaba5a774d7d5ffed55967fe65344373866492caa5 not found: ID does not exist" Oct 30 20:15:25.985406 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.985376 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt"] Oct 30 20:15:25.996213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:25.996177 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d97cc8/payload-processing-external-model-e2e-extmod-b-d97cc8-7b78sf7nt"] Oct 30 20:15:26.120591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:26.120554 5091 scope.go:122] "RemoveContainer" containerID="7fd889561d439776c6efdcdf06b465b6daa0962f7aa075c5296b2cfc20899976" Oct 30 20:15:26.132039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:26.132019 5091 scope.go:122] "RemoveContainer" containerID="da2c6817fcdb39609bd5391ce9f76c76fda163ae308f534945b36a3209ce3428" Oct 30 20:15:26.141906 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:26.141888 5091 scope.go:122] "RemoveContainer" containerID="2dbd901459f8c7013342894d59e0ad5def8614b3194e3b75da2df62f80f2b574" Oct 30 20:15:26.151568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:26.151544 5091 scope.go:122] "RemoveContainer" containerID="b0b24d1654175ca81f696fd899e53055d69a9620bea62f9f5b6e1c8ecfdb7ae9" Oct 30 20:15:26.448702 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:26.448671 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" path="/var/lib/kubelet/pods/4eafdae9-7fcd-4579-824b-60bf84bb9b7f/volumes" Oct 30 20:15:26.449292 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:26.449269 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c" path="/var/lib/kubelet/pods/af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c/volumes" Oct 30 20:15:26.668172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:26.668113 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:26.968776 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:26.968694 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice/crio-1ae08a3ca619d3c72e4eecfe6172f0a90cbfe89251b05e4988849b12b55b1843\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8a41b6_c1fd_4d9b_8f3d_c0a93eb8434c.slice/crio-5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397\": RecentStats: unable to find data in memory cache]" Oct 30 20:15:27.622925 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:27.614806 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:28.538166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.534227 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:28.698676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.698643 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7"] Oct 30 20:15:28.699391 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699374 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2a5e04ce-a8ad-4b44-bd00-624a49388708" containerName="cleanup" Oct 30 20:15:28.699391 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699391 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5e04ce-a8ad-4b44-bd00-624a49388708" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699401 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699407 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699417 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699422 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699429 5091 cpu_manager.go:414] "containerMap: removing container" podUID="755783df-2b63-4013-9133-d39c004e577f" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699433 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="755783df-2b63-4013-9133-d39c004e577f" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699443 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699449 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699459 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2ea189f9-f225-4fdb-8948-14cb53111cd5" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699464 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea189f9-f225-4fdb-8948-14cb53111cd5" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699473 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2ea189f9-f225-4fdb-8948-14cb53111cd5" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699478 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea189f9-f225-4fdb-8948-14cb53111cd5" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699488 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699493 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699503 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699507 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699515 5091 cpu_manager.go:414] "containerMap: removing container" podUID="67449cd5-2639-4625-b32e-ba3bec5a528b" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699520 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67449cd5-2639-4625-b32e-ba3bec5a528b" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699527 5091 cpu_manager.go:414] "containerMap: removing container" podUID="78e70c72-9983-4b2e-bd0c-176f50822feb" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699532 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e70c72-9983-4b2e-bd0c-176f50822feb" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699544 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699549 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699555 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2a5e04ce-a8ad-4b44-bd00-624a49388708" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699560 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5e04ce-a8ad-4b44-bd00-624a49388708" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699571 5091 cpu_manager.go:414] "containerMap: removing container" podUID="387559a4-9827-4a53-90ae-3f3f0852d25d" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699576 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387559a4-9827-4a53-90ae-3f3f0852d25d" containerName="cleanup" Oct 30 20:15:28.699566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699583 5091 cpu_manager.go:414] "containerMap: removing container" podUID="78e70c72-9983-4b2e-bd0c-176f50822feb" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699588 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e70c72-9983-4b2e-bd0c-176f50822feb" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699594 5091 cpu_manager.go:414] "containerMap: removing container" podUID="87e33682-f349-450b-ab65-1b60d304a6cf" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699599 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e33682-f349-450b-ab65-1b60d304a6cf" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699605 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699610 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699616 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699620 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699629 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b8336ff5-7043-4f2a-8ab1-60c4713596e9" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699633 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8336ff5-7043-4f2a-8ab1-60c4713596e9" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699639 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699643 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699650 5091 cpu_manager.go:414] "containerMap: removing container" podUID="87e33682-f349-450b-ab65-1b60d304a6cf" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699654 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e33682-f349-450b-ab65-1b60d304a6cf" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699665 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699669 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699675 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699679 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699686 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699690 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699700 5091 cpu_manager.go:414] "containerMap: removing container" podUID="af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c" containerName="payload-processing" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699704 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c" containerName="payload-processing" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699714 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699720 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699727 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699732 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699739 5091 cpu_manager.go:414] "containerMap: removing container" podUID="87e33682-f349-450b-ab65-1b60d304a6cf" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699743 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e33682-f349-450b-ab65-1b60d304a6cf" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699749 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ee32ddbf-9640-4efd-b2a5-babcfb306088" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699754 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee32ddbf-9640-4efd-b2a5-babcfb306088" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699761 5091 cpu_manager.go:414] "containerMap: removing container" podUID="387559a4-9827-4a53-90ae-3f3f0852d25d" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699765 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387559a4-9827-4a53-90ae-3f3f0852d25d" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699771 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d27d65d4-ebb6-4254-89c9-febcacbdf9f1" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699775 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27d65d4-ebb6-4254-89c9-febcacbdf9f1" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699784 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b8336ff5-7043-4f2a-8ab1-60c4713596e9" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699789 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8336ff5-7043-4f2a-8ab1-60c4713596e9" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699795 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699800 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699807 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ee32ddbf-9640-4efd-b2a5-babcfb306088" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699811 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee32ddbf-9640-4efd-b2a5-babcfb306088" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699817 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699822 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699832 5091 cpu_manager.go:414] "containerMap: removing container" podUID="67449cd5-2639-4625-b32e-ba3bec5a528b" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699836 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67449cd5-2639-4625-b32e-ba3bec5a528b" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699843 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ee32ddbf-9640-4efd-b2a5-babcfb306088" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699848 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee32ddbf-9640-4efd-b2a5-babcfb306088" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699855 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d27d65d4-ebb6-4254-89c9-febcacbdf9f1" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699860 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27d65d4-ebb6-4254-89c9-febcacbdf9f1" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699869 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d27d65d4-ebb6-4254-89c9-febcacbdf9f1" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699874 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d27d65d4-ebb6-4254-89c9-febcacbdf9f1" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699880 5091 cpu_manager.go:414] "containerMap: removing container" podUID="755783df-2b63-4013-9133-d39c004e577f" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699884 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="755783df-2b63-4013-9133-d39c004e577f" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699976 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27d65d4-ebb6-4254-89c9-febcacbdf9f1" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699983 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="755783df-2b63-4013-9133-d39c004e577f" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699990 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5e04ce-a8ad-4b44-bd00-624a49388708" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.699995 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700001 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700008 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="755783df-2b63-4013-9133-d39c004e577f" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700016 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700022 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea189f9-f225-4fdb-8948-14cb53111cd5" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700027 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea189f9-f225-4fdb-8948-14cb53111cd5" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700033 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700038 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700044 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="67449cd5-2639-4625-b32e-ba3bec5a528b" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700049 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e70c72-9983-4b2e-bd0c-176f50822feb" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700056 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700062 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5e04ce-a8ad-4b44-bd00-624a49388708" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700067 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="387559a4-9827-4a53-90ae-3f3f0852d25d" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700074 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700078 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e33682-f349-450b-ab65-1b60d304a6cf" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700084 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700089 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8336ff5-7043-4f2a-8ab1-60c4713596e9" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700094 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700099 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e33682-f349-450b-ab65-1b60d304a6cf" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700106 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700111 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700119 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700124 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="af6a7763-ca3a-48b9-8613-bc5e3e1bbd8c" containerName="payload-processing" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700131 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="387559a4-9827-4a53-90ae-3f3f0852d25d" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700137 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700158 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8336ff5-7043-4f2a-8ab1-60c4713596e9" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700167 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e33682-f349-450b-ab65-1b60d304a6cf" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700188 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee32ddbf-9640-4efd-b2a5-babcfb306088" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700194 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="387559a4-9827-4a53-90ae-3f3f0852d25d" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700199 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27d65d4-ebb6-4254-89c9-febcacbdf9f1" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700205 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8336ff5-7043-4f2a-8ab1-60c4713596e9" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700211 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700218 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee32ddbf-9640-4efd-b2a5-babcfb306088" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700225 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="67449cd5-2639-4625-b32e-ba3bec5a528b" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700230 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e70c72-9983-4b2e-bd0c-176f50822feb" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700237 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" containerName="cleanup" Oct 30 20:15:28.700984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.700242 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee32ddbf-9640-4efd-b2a5-babcfb306088" containerName="cleanup" Oct 30 20:15:28.707054 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.707035 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" Oct 30 20:15:28.717631 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.715392 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:28.717631 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.715547 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:15:28.718127 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.718106 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7"] Oct 30 20:15:28.801541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.801459 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtzf2\" (UniqueName: \"kubernetes.io/projected/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-kube-api-access-xtzf2\") pod \"maas-api-revoke-sub-90f64cb7-cjtb7\" (UID: \"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" Oct 30 20:15:28.801541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.801515 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90f64cb7-cjtb7\" (UID: \"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" Oct 30 20:15:28.801743 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.801673 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90f64cb7-cjtb7\" (UID: \"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" Oct 30 20:15:28.903247 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.903210 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtzf2\" (UniqueName: \"kubernetes.io/projected/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-kube-api-access-xtzf2\") pod \"maas-api-revoke-sub-90f64cb7-cjtb7\" (UID: \"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" Oct 30 20:15:28.903425 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.903263 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90f64cb7-cjtb7\" (UID: \"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" Oct 30 20:15:28.903425 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.903402 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90f64cb7-cjtb7\" (UID: \"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" Oct 30 20:15:28.903918 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.903896 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-90f64cb7-cjtb7\" (UID: \"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" Oct 30 20:15:28.905901 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.905849 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-90f64cb7-cjtb7\" (UID: \"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" Oct 30 20:15:28.926822 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:28.926790 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtzf2\" (UniqueName: \"kubernetes.io/projected/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-kube-api-access-xtzf2\") pod \"maas-api-revoke-sub-90f64cb7-cjtb7\" (UID: \"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" Oct 30 20:15:29.031587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.031550 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" Oct 30 20:15:29.348123 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.348095 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7"] Oct 30 20:15:29.348304 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:29.348279 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd906bfd5_8723_4c63_9f12_bb8f55a4e7e5.slice/crio-d8c711e5440b377d619e71947ad6683918a03cbbd550340c24a50e175fb94ef7 WatchSource:0}: Error finding container d8c711e5440b377d619e71947ad6683918a03cbbd550340c24a50e175fb94ef7: Status 404 returned error can't find the container with id d8c711e5440b377d619e71947ad6683918a03cbbd550340c24a50e175fb94ef7 Oct 30 20:15:29.496461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.496425 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7"] Oct 30 20:15:29.497208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497191 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" containerName="cleanup" Oct 30 20:15:29.497208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497208 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" containerName="cleanup" Oct 30 20:15:29.497307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497234 5091 cpu_manager.go:414] "containerMap: removing container" podUID="387559a4-9827-4a53-90ae-3f3f0852d25d" containerName="cleanup" Oct 30 20:15:29.497307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497239 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387559a4-9827-4a53-90ae-3f3f0852d25d" containerName="cleanup" Oct 30 20:15:29.497307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497249 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b8336ff5-7043-4f2a-8ab1-60c4713596e9" containerName="cleanup" Oct 30 20:15:29.497307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497254 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8336ff5-7043-4f2a-8ab1-60c4713596e9" containerName="cleanup" Oct 30 20:15:29.497307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497270 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" containerName="cleanup" Oct 30 20:15:29.497307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497276 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" containerName="cleanup" Oct 30 20:15:29.497307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497287 5091 cpu_manager.go:414] "containerMap: removing container" podUID="755783df-2b63-4013-9133-d39c004e577f" containerName="cleanup" Oct 30 20:15:29.497307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497292 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="755783df-2b63-4013-9133-d39c004e577f" containerName="cleanup" Oct 30 20:15:29.497307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497303 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2ea189f9-f225-4fdb-8948-14cb53111cd5" containerName="cleanup" Oct 30 20:15:29.497307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497310 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea189f9-f225-4fdb-8948-14cb53111cd5" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497320 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497327 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b69782b-78e9-4baf-8bdf-bcbf9b78767d" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497341 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497346 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497355 5091 cpu_manager.go:414] "containerMap: removing container" podUID="67449cd5-2639-4625-b32e-ba3bec5a528b" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497360 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67449cd5-2639-4625-b32e-ba3bec5a528b" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497374 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2a5e04ce-a8ad-4b44-bd00-624a49388708" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497378 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5e04ce-a8ad-4b44-bd00-624a49388708" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497390 5091 cpu_manager.go:414] "containerMap: removing container" podUID="78e70c72-9983-4b2e-bd0c-176f50822feb" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497394 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78e70c72-9983-4b2e-bd0c-176f50822feb" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497404 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497411 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497510 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb65533-38aa-48c4-a374-fcf2f5e96f99" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497517 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="78e70c72-9983-4b2e-bd0c-176f50822feb" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497527 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5e04ce-a8ad-4b44-bd00-624a49388708" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497535 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c8a41b6-c1fd-4d9b-8f3d-c0a93eb8434c" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497540 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d37c05b-93b8-4c8c-84d1-9e186d7ec265" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497548 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eafdae9-7fcd-4579-824b-60bf84bb9b7f" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497562 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="67449cd5-2639-4625-b32e-ba3bec5a528b" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497567 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93e44d7-d7d5-4beb-9332-403ddc9bf336" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497572 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d27d65d4-ebb6-4254-89c9-febcacbdf9f1" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497578 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="755783df-2b63-4013-9133-d39c004e577f" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497583 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7d9f0fd-4bb0-4773-8333-92f168e683c3" containerName="cleanup" Oct 30 20:15:29.497635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.497592 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea189f9-f225-4fdb-8948-14cb53111cd5" containerName="cleanup" Oct 30 20:15:29.507836 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.505372 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" Oct 30 20:15:29.529768 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.529736 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7"] Oct 30 20:15:29.624274 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.623319 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0308bc8c-4d95-4bb8-a116-5359e5df4d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c2cfa27-lqvt7\" (UID: \"0308bc8c-4d95-4bb8-a116-5359e5df4d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" Oct 30 20:15:29.624274 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.623374 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0308bc8c-4d95-4bb8-a116-5359e5df4d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c2cfa27-lqvt7\" (UID: \"0308bc8c-4d95-4bb8-a116-5359e5df4d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" Oct 30 20:15:29.624274 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.623481 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bscmq\" (UniqueName: \"kubernetes.io/projected/0308bc8c-4d95-4bb8-a116-5359e5df4d52-kube-api-access-bscmq\") pod \"maas-api-revoke-sub-7c2cfa27-lqvt7\" (UID: \"0308bc8c-4d95-4bb8-a116-5359e5df4d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" Oct 30 20:15:29.725193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.725104 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0308bc8c-4d95-4bb8-a116-5359e5df4d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c2cfa27-lqvt7\" (UID: \"0308bc8c-4d95-4bb8-a116-5359e5df4d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" Oct 30 20:15:29.725193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.725166 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0308bc8c-4d95-4bb8-a116-5359e5df4d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c2cfa27-lqvt7\" (UID: \"0308bc8c-4d95-4bb8-a116-5359e5df4d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" Oct 30 20:15:29.725454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.725265 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bscmq\" (UniqueName: \"kubernetes.io/projected/0308bc8c-4d95-4bb8-a116-5359e5df4d52-kube-api-access-bscmq\") pod \"maas-api-revoke-sub-7c2cfa27-lqvt7\" (UID: \"0308bc8c-4d95-4bb8-a116-5359e5df4d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" Oct 30 20:15:29.725765 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.725740 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0308bc8c-4d95-4bb8-a116-5359e5df4d52-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7c2cfa27-lqvt7\" (UID: \"0308bc8c-4d95-4bb8-a116-5359e5df4d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" Oct 30 20:15:29.727609 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.727588 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0308bc8c-4d95-4bb8-a116-5359e5df4d52-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7c2cfa27-lqvt7\" (UID: \"0308bc8c-4d95-4bb8-a116-5359e5df4d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" Oct 30 20:15:29.739550 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.739525 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bscmq\" (UniqueName: \"kubernetes.io/projected/0308bc8c-4d95-4bb8-a116-5359e5df4d52-kube-api-access-bscmq\") pod \"maas-api-revoke-sub-7c2cfa27-lqvt7\" (UID: \"0308bc8c-4d95-4bb8-a116-5359e5df4d52\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" Oct 30 20:15:29.831049 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.831018 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" Oct 30 20:15:29.930687 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.930655 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:29.993957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.993915 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" event={"ID":"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5","Type":"ContainerStarted","Data":"408a6022ae16bb52d52ad9e8664abc35750ca8556f82bd388b740f916584bad8"} Oct 30 20:15:29.994127 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:29.993986 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" event={"ID":"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5","Type":"ContainerStarted","Data":"d8c711e5440b377d619e71947ad6683918a03cbbd550340c24a50e175fb94ef7"} Oct 30 20:15:30.017878 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.017820 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" podStartSLOduration=2.017805954 podStartE2EDuration="2.017805954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:30.016430568 +0000 UTC m=+2649.711638527" watchObservedRunningTime="2026-10-30 20:15:30.017805954 +0000 UTC m=+2649.713013912" Oct 30 20:15:30.124662 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.123721 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7"] Oct 30 20:15:30.126394 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:30.126354 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0308bc8c_4d95_4bb8_a116_5359e5df4d52.slice/crio-9b7c9d85490833000b8bbc16bf0a0b2994b24f5d6d360f652dbd85a3f5dd353a WatchSource:0}: Error finding container 9b7c9d85490833000b8bbc16bf0a0b2994b24f5d6d360f652dbd85a3f5dd353a: Status 404 returned error can't find the container with id 9b7c9d85490833000b8bbc16bf0a0b2994b24f5d6d360f652dbd85a3f5dd353a Oct 30 20:15:30.498264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.494818 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:30.612654 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.612609 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c"] Oct 30 20:15:30.618499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.618476 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" Oct 30 20:15:30.630122 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.630091 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c"] Oct 30 20:15:30.656267 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.656227 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06ab6bf3-qlf7c\" (UID: \"d0a7eb30-eb93-41c4-aabf-ffda909ccb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" Oct 30 20:15:30.656564 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.656517 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06ab6bf3-qlf7c\" (UID: \"d0a7eb30-eb93-41c4-aabf-ffda909ccb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" Oct 30 20:15:30.656636 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.656619 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4nrv\" (UniqueName: \"kubernetes.io/projected/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-kube-api-access-q4nrv\") pod \"maas-api-revoke-sub-06ab6bf3-qlf7c\" (UID: \"d0a7eb30-eb93-41c4-aabf-ffda909ccb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" Oct 30 20:15:30.758511 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.758422 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06ab6bf3-qlf7c\" (UID: \"d0a7eb30-eb93-41c4-aabf-ffda909ccb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" Oct 30 20:15:30.758511 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.758491 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4nrv\" (UniqueName: \"kubernetes.io/projected/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-kube-api-access-q4nrv\") pod \"maas-api-revoke-sub-06ab6bf3-qlf7c\" (UID: \"d0a7eb30-eb93-41c4-aabf-ffda909ccb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" Oct 30 20:15:30.758687 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.758652 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06ab6bf3-qlf7c\" (UID: \"d0a7eb30-eb93-41c4-aabf-ffda909ccb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" Oct 30 20:15:30.759291 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.759262 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06ab6bf3-qlf7c\" (UID: \"d0a7eb30-eb93-41c4-aabf-ffda909ccb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" Oct 30 20:15:30.761299 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.761264 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06ab6bf3-qlf7c\" (UID: \"d0a7eb30-eb93-41c4-aabf-ffda909ccb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" Oct 30 20:15:30.776332 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.776304 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4nrv\" (UniqueName: \"kubernetes.io/projected/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-kube-api-access-q4nrv\") pod \"maas-api-revoke-sub-06ab6bf3-qlf7c\" (UID: \"d0a7eb30-eb93-41c4-aabf-ffda909ccb39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" Oct 30 20:15:30.936737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:30.936664 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" Oct 30 20:15:31.011534 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:31.011461 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" event={"ID":"0308bc8c-4d95-4bb8-a116-5359e5df4d52","Type":"ContainerStarted","Data":"84ebb82bac85eee2526f21b3fa24b02aa4ec4258c4136a67c625f2301694ca09"} Oct 30 20:15:31.011534 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:31.011528 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" event={"ID":"0308bc8c-4d95-4bb8-a116-5359e5df4d52","Type":"ContainerStarted","Data":"9b7c9d85490833000b8bbc16bf0a0b2994b24f5d6d360f652dbd85a3f5dd353a"} Oct 30 20:15:31.014033 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:31.013985 5091 generic.go:356] "Generic (PLEG): container finished" podID="d906bfd5-8723-4c63-9f12-bb8f55a4e7e5" containerID="408a6022ae16bb52d52ad9e8664abc35750ca8556f82bd388b740f916584bad8" exitCode=0 Oct 30 20:15:31.014423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:31.014355 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" event={"ID":"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5","Type":"ContainerDied","Data":"408a6022ae16bb52d52ad9e8664abc35750ca8556f82bd388b740f916584bad8"} Oct 30 20:15:31.039798 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:31.039741 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" podStartSLOduration=2.039726128 podStartE2EDuration="2.039726128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:31.035302327 +0000 UTC m=+2650.730510287" watchObservedRunningTime="2026-10-30 20:15:31.039726128 +0000 UTC m=+2650.734934086" Oct 30 20:15:31.225521 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:31.225485 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0a7eb30_eb93_41c4_aabf_ffda909ccb39.slice/crio-9e84a1a6eb9fee5fa5236af4494b93ab4cccf0355bb972e97fd29de4775bd91c WatchSource:0}: Error finding container 9e84a1a6eb9fee5fa5236af4494b93ab4cccf0355bb972e97fd29de4775bd91c: Status 404 returned error can't find the container with id 9e84a1a6eb9fee5fa5236af4494b93ab4cccf0355bb972e97fd29de4775bd91c Oct 30 20:15:31.225726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:31.225556 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c"] Oct 30 20:15:31.397184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:31.395892 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:32.024800 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.024761 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" event={"ID":"d0a7eb30-eb93-41c4-aabf-ffda909ccb39","Type":"ContainerStarted","Data":"c87e19f4c679b5acdcc9827ef816fe7cba3ea115d99dff5f1c940af63b1dfa7f"} Oct 30 20:15:32.025276 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.024827 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" event={"ID":"d0a7eb30-eb93-41c4-aabf-ffda909ccb39","Type":"ContainerStarted","Data":"9e84a1a6eb9fee5fa5236af4494b93ab4cccf0355bb972e97fd29de4775bd91c"} Oct 30 20:15:32.026517 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.026491 5091 generic.go:356] "Generic (PLEG): container finished" podID="0308bc8c-4d95-4bb8-a116-5359e5df4d52" containerID="84ebb82bac85eee2526f21b3fa24b02aa4ec4258c4136a67c625f2301694ca09" exitCode=0 Oct 30 20:15:32.026626 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.026566 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" event={"ID":"0308bc8c-4d95-4bb8-a116-5359e5df4d52","Type":"ContainerDied","Data":"84ebb82bac85eee2526f21b3fa24b02aa4ec4258c4136a67c625f2301694ca09"} Oct 30 20:15:32.052617 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.052546 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" podStartSLOduration=2.052529357 podStartE2EDuration="2.052529357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:32.051523765 +0000 UTC m=+2651.746731731" watchObservedRunningTime="2026-10-30 20:15:32.052529357 +0000 UTC m=+2651.747737317" Oct 30 20:15:32.289331 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.271299 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:32.289331 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.274345 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" Oct 30 20:15:32.431876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.431842 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-maas-api-xks-ca\") pod \"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5\" (UID: \"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5\") " Oct 30 20:15:32.431876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.431884 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-maas-api-openshift-ca\") pod \"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5\" (UID: \"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5\") " Oct 30 20:15:32.432201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.431920 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-kube-api-access-xtzf2\" (UniqueName: \"kubernetes.io/projected/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-kube-api-access-xtzf2\") pod \"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5\" (UID: \"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5\") " Oct 30 20:15:32.432376 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.432352 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-maas-api-openshift-ca" pod "d906bfd5-8723-4c63-9f12-bb8f55a4e7e5" (UID: "d906bfd5-8723-4c63-9f12-bb8f55a4e7e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:32.432527 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.432511 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:32.434396 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.434365 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-kube-api-access-xtzf2" pod "d906bfd5-8723-4c63-9f12-bb8f55a4e7e5" (UID: "d906bfd5-8723-4c63-9f12-bb8f55a4e7e5"). InnerVolumeSpecName "kube-api-access-xtzf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:32.434956 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.434928 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-maas-api-xks-ca" pod "d906bfd5-8723-4c63-9f12-bb8f55a4e7e5" (UID: "d906bfd5-8723-4c63-9f12-bb8f55a4e7e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:32.534107 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.534078 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:32.534284 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:32.534112 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtzf2\" (UniqueName: \"kubernetes.io/projected/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5-kube-api-access-xtzf2\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:33.050298 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.050261 5091 generic.go:356] "Generic (PLEG): container finished" podID="d0a7eb30-eb93-41c4-aabf-ffda909ccb39" containerID="c87e19f4c679b5acdcc9827ef816fe7cba3ea115d99dff5f1c940af63b1dfa7f" exitCode=0 Oct 30 20:15:33.050797 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.050347 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" event={"ID":"d0a7eb30-eb93-41c4-aabf-ffda909ccb39","Type":"ContainerDied","Data":"c87e19f4c679b5acdcc9827ef816fe7cba3ea115d99dff5f1c940af63b1dfa7f"} Oct 30 20:15:33.052239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.052216 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" Oct 30 20:15:33.052239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.052236 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7" event={"ID":"d906bfd5-8723-4c63-9f12-bb8f55a4e7e5","Type":"ContainerDied","Data":"d8c711e5440b377d619e71947ad6683918a03cbbd550340c24a50e175fb94ef7"} Oct 30 20:15:33.052441 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.052284 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8c711e5440b377d619e71947ad6683918a03cbbd550340c24a50e175fb94ef7" Oct 30 20:15:33.102765 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.102730 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:33.326568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.326484 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" Oct 30 20:15:33.455886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.455849 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0308bc8c-4d95-4bb8-a116-5359e5df4d52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0308bc8c-4d95-4bb8-a116-5359e5df4d52-maas-api-openshift-ca\") pod \"0308bc8c-4d95-4bb8-a116-5359e5df4d52\" (UID: \"0308bc8c-4d95-4bb8-a116-5359e5df4d52\") " Oct 30 20:15:33.456085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.455935 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0308bc8c-4d95-4bb8-a116-5359e5df4d52-kube-api-access-bscmq\" (UniqueName: \"kubernetes.io/projected/0308bc8c-4d95-4bb8-a116-5359e5df4d52-kube-api-access-bscmq\") pod \"0308bc8c-4d95-4bb8-a116-5359e5df4d52\" (UID: \"0308bc8c-4d95-4bb8-a116-5359e5df4d52\") " Oct 30 20:15:33.456085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.455970 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0308bc8c-4d95-4bb8-a116-5359e5df4d52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0308bc8c-4d95-4bb8-a116-5359e5df4d52-maas-api-xks-ca\") pod \"0308bc8c-4d95-4bb8-a116-5359e5df4d52\" (UID: \"0308bc8c-4d95-4bb8-a116-5359e5df4d52\") " Oct 30 20:15:33.456256 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.456229 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0308bc8c-4d95-4bb8-a116-5359e5df4d52-maas-api-openshift-ca" pod "0308bc8c-4d95-4bb8-a116-5359e5df4d52" (UID: "0308bc8c-4d95-4bb8-a116-5359e5df4d52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:33.456786 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.456759 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0308bc8c-4d95-4bb8-a116-5359e5df4d52-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:33.458320 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.458294 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0308bc8c-4d95-4bb8-a116-5359e5df4d52-kube-api-access-bscmq" pod "0308bc8c-4d95-4bb8-a116-5359e5df4d52" (UID: "0308bc8c-4d95-4bb8-a116-5359e5df4d52"). InnerVolumeSpecName "kube-api-access-bscmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:33.458484 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.458462 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0308bc8c-4d95-4bb8-a116-5359e5df4d52-maas-api-xks-ca" pod "0308bc8c-4d95-4bb8-a116-5359e5df4d52" (UID: "0308bc8c-4d95-4bb8-a116-5359e5df4d52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:33.558543 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.558494 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bscmq\" (UniqueName: \"kubernetes.io/projected/0308bc8c-4d95-4bb8-a116-5359e5df4d52-kube-api-access-bscmq\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:33.558543 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:33.558539 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0308bc8c-4d95-4bb8-a116-5359e5df4d52-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:34.034765 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:34.034722 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:34.091220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:34.091187 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" Oct 30 20:15:34.091220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:34.091214 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7" event={"ID":"0308bc8c-4d95-4bb8-a116-5359e5df4d52","Type":"ContainerDied","Data":"9b7c9d85490833000b8bbc16bf0a0b2994b24f5d6d360f652dbd85a3f5dd353a"} Oct 30 20:15:34.091755 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:34.091277 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b7c9d85490833000b8bbc16bf0a0b2994b24f5d6d360f652dbd85a3f5dd353a" Oct 30 20:15:34.298662 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:34.298595 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" Oct 30 20:15:34.386273 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:34.386234 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-maas-api-xks-ca\") pod \"d0a7eb30-eb93-41c4-aabf-ffda909ccb39\" (UID: \"d0a7eb30-eb93-41c4-aabf-ffda909ccb39\") " Oct 30 20:15:34.386482 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:34.386288 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-maas-api-openshift-ca\") pod \"d0a7eb30-eb93-41c4-aabf-ffda909ccb39\" (UID: \"d0a7eb30-eb93-41c4-aabf-ffda909ccb39\") " Oct 30 20:15:34.386674 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:34.386640 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-kube-api-access-q4nrv\" (UniqueName: \"kubernetes.io/projected/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-kube-api-access-q4nrv\") pod \"d0a7eb30-eb93-41c4-aabf-ffda909ccb39\" (UID: \"d0a7eb30-eb93-41c4-aabf-ffda909ccb39\") " Oct 30 20:15:34.386832 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:34.386728 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-maas-api-openshift-ca" pod "d0a7eb30-eb93-41c4-aabf-ffda909ccb39" (UID: "d0a7eb30-eb93-41c4-aabf-ffda909ccb39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:34.387413 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:34.387391 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:34.389029 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:34.389004 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-maas-api-xks-ca" pod "d0a7eb30-eb93-41c4-aabf-ffda909ccb39" (UID: "d0a7eb30-eb93-41c4-aabf-ffda909ccb39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:34.389138 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:34.389064 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-kube-api-access-q4nrv" pod "d0a7eb30-eb93-41c4-aabf-ffda909ccb39" (UID: "d0a7eb30-eb93-41c4-aabf-ffda909ccb39"). InnerVolumeSpecName "kube-api-access-q4nrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:34.488979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:34.488939 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:34.488979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:34.488970 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4nrv\" (UniqueName: \"kubernetes.io/projected/d0a7eb30-eb93-41c4-aabf-ffda909ccb39-kube-api-access-q4nrv\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:34.734134 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:34.734029 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8a41b6_c1fd_4d9b_8f3d_c0a93eb8434c.slice/crio-5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice/crio-1ae08a3ca619d3c72e4eecfe6172f0a90cbfe89251b05e4988849b12b55b1843\": RecentStats: unable to find data in memory cache]" Oct 30 20:15:35.100475 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:35.097604 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:35.138855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:35.135852 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" event={"ID":"d0a7eb30-eb93-41c4-aabf-ffda909ccb39","Type":"ContainerDied","Data":"9e84a1a6eb9fee5fa5236af4494b93ab4cccf0355bb972e97fd29de4775bd91c"} Oct 30 20:15:35.138855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:35.135938 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e84a1a6eb9fee5fa5236af4494b93ab4cccf0355bb972e97fd29de4775bd91c" Oct 30 20:15:35.138855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:35.136040 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c" Oct 30 20:15:36.048892 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:36.048854 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:36.818165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:36.817229 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:36.988911 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:36.988859 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8a41b6_c1fd_4d9b_8f3d_c0a93eb8434c.slice/crio-5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice/crio-1ae08a3ca619d3c72e4eecfe6172f0a90cbfe89251b05e4988849b12b55b1843\": RecentStats: unable to find data in memory cache]" Oct 30 20:15:37.309156 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:37.309114 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r"] Oct 30 20:15:37.309385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:37.309359 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerName="istio-proxy" containerID="cri-o://cd47aab2566f815246ded1d16b7fff7e4122935c0ce49879eaa31dd943fc0d0d" gracePeriod=30 Oct 30 20:15:37.723393 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:37.723304 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:38.598183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:38.592769 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:39.574209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:39.574172 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:39.886422 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:39.886332 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:40.072946 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.072896 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc"] Oct 30 20:15:40.079700 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.079667 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d906bfd5-8723-4c63-9f12-bb8f55a4e7e5" containerName="revoke-keys" Oct 30 20:15:40.079925 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.079910 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d906bfd5-8723-4c63-9f12-bb8f55a4e7e5" containerName="revoke-keys" Oct 30 20:15:40.080094 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.080083 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0308bc8c-4d95-4bb8-a116-5359e5df4d52" containerName="revoke-keys" Oct 30 20:15:40.080190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.080180 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0308bc8c-4d95-4bb8-a116-5359e5df4d52" containerName="revoke-keys" Oct 30 20:15:40.087169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.080310 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d0a7eb30-eb93-41c4-aabf-ffda909ccb39" containerName="revoke-keys" Oct 30 20:15:40.087403 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.087380 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0a7eb30-eb93-41c4-aabf-ffda909ccb39" containerName="revoke-keys" Oct 30 20:15:40.087881 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.087862 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0308bc8c-4d95-4bb8-a116-5359e5df4d52" containerName="revoke-keys" Oct 30 20:15:40.088053 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.088039 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0a7eb30-eb93-41c4-aabf-ffda909ccb39" containerName="revoke-keys" Oct 30 20:15:40.092230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.092206 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d906bfd5-8723-4c63-9f12-bb8f55a4e7e5" containerName="revoke-keys" Oct 30 20:15:40.194039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.193773 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc"] Oct 30 20:15:40.194039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.193899 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:40.200860 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.200831 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c06e1d87a941\"" Oct 30 20:15:40.201276 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.201254 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Oct 30 20:15:40.201416 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.201312 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Oct 30 20:15:40.201416 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.201381 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Oct 30 20:15:40.201551 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.201460 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Oct 30 20:15:40.211565 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.206266 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-mqtgc\"" Oct 30 20:15:40.255135 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.255099 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:40.394221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.388824 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72dc58a1-2629-4df0-b85b-e0271d9f171e-provider-credentials\") pod \"payload-processing-external-model-6d49c6d66d-wx6hc\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:40.417111 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.417074 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72dc58a1-2629-4df0-b85b-e0271d9f171e-plugins-config-volume\") pod \"payload-processing-external-model-6d49c6d66d-wx6hc\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:40.417514 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.417491 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72dc58a1-2629-4df0-b85b-e0271d9f171e-routing-overlay-volume\") pod \"payload-processing-external-model-6d49c6d66d-wx6hc\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:40.417702 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.417685 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcn7b\" (UniqueName: \"kubernetes.io/projected/72dc58a1-2629-4df0-b85b-e0271d9f171e-kube-api-access-dcn7b\") pod \"payload-processing-external-model-6d49c6d66d-wx6hc\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:40.529390 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.529350 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72dc58a1-2629-4df0-b85b-e0271d9f171e-provider-credentials\") pod \"payload-processing-external-model-6d49c6d66d-wx6hc\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:40.529701 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.529682 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72dc58a1-2629-4df0-b85b-e0271d9f171e-plugins-config-volume\") pod \"payload-processing-external-model-6d49c6d66d-wx6hc\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:40.529890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.529875 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72dc58a1-2629-4df0-b85b-e0271d9f171e-routing-overlay-volume\") pod \"payload-processing-external-model-6d49c6d66d-wx6hc\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:40.530015 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.529999 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcn7b\" (UniqueName: \"kubernetes.io/projected/72dc58a1-2629-4df0-b85b-e0271d9f171e-kube-api-access-dcn7b\") pod \"payload-processing-external-model-6d49c6d66d-wx6hc\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:40.531673 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.531645 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72dc58a1-2629-4df0-b85b-e0271d9f171e-plugins-config-volume\") pod \"payload-processing-external-model-6d49c6d66d-wx6hc\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:40.532575 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.532549 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72dc58a1-2629-4df0-b85b-e0271d9f171e-routing-overlay-volume\") pod \"payload-processing-external-model-6d49c6d66d-wx6hc\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:40.548444 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.548395 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72dc58a1-2629-4df0-b85b-e0271d9f171e-provider-credentials\") pod \"payload-processing-external-model-6d49c6d66d-wx6hc\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:40.568713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.568677 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcn7b\" (UniqueName: \"kubernetes.io/projected/72dc58a1-2629-4df0-b85b-e0271d9f171e-kube-api-access-dcn7b\") pod \"payload-processing-external-model-6d49c6d66d-wx6hc\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:40.820128 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:40.820050 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:41.056429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:41.056389 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:41.118409 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:41.118328 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc"] Oct 30 20:15:41.178206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:41.177284 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc"] Oct 30 20:15:41.183316 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:41.183278 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72dc58a1_2629_4df0_b85b_e0271d9f171e.slice/crio-53b2fe0e25f49db6d284672e6a1d3f0db02c87b409a886d522060f3bb1c1b9ed WatchSource:0}: Error finding container 53b2fe0e25f49db6d284672e6a1d3f0db02c87b409a886d522060f3bb1c1b9ed: Status 404 returned error can't find the container with id 53b2fe0e25f49db6d284672e6a1d3f0db02c87b409a886d522060f3bb1c1b9ed Oct 30 20:15:41.228429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:41.228389 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" event={"ID":"72dc58a1-2629-4df0-b85b-e0271d9f171e","Type":"ContainerStarted","Data":"53b2fe0e25f49db6d284672e6a1d3f0db02c87b409a886d522060f3bb1c1b9ed"} Oct 30 20:15:41.265178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:41.265129 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:41.757385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:41.757353 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k"] Oct 30 20:15:41.936504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:41.936435 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k"] Oct 30 20:15:41.936701 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:41.936557 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" Oct 30 20:15:41.950078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:41.950011 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:15:41.950187 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:41.950082 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:42.013295 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.013202 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k\" (UID: \"a8a71502-6ae1-4cb2-8181-b253e0cee4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" Oct 30 20:15:42.013457 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.013325 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k\" (UID: \"a8a71502-6ae1-4cb2-8181-b253e0cee4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" Oct 30 20:15:42.013457 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.013390 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqwkm\" (UniqueName: \"kubernetes.io/projected/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-kube-api-access-bqwkm\") pod \"maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k\" (UID: \"a8a71502-6ae1-4cb2-8181-b253e0cee4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" Oct 30 20:15:42.122138 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.122093 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k\" (UID: \"a8a71502-6ae1-4cb2-8181-b253e0cee4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" Oct 30 20:15:42.122651 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.122209 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k\" (UID: \"a8a71502-6ae1-4cb2-8181-b253e0cee4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" Oct 30 20:15:42.122651 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.122291 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqwkm\" (UniqueName: \"kubernetes.io/projected/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-kube-api-access-bqwkm\") pod \"maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k\" (UID: \"a8a71502-6ae1-4cb2-8181-b253e0cee4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" Oct 30 20:15:42.123225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.123192 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k\" (UID: \"a8a71502-6ae1-4cb2-8181-b253e0cee4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" Oct 30 20:15:42.125672 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.125647 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k\" (UID: \"a8a71502-6ae1-4cb2-8181-b253e0cee4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" Oct 30 20:15:42.142346 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.142322 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqwkm\" (UniqueName: \"kubernetes.io/projected/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-kube-api-access-bqwkm\") pod \"maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k\" (UID: \"a8a71502-6ae1-4cb2-8181-b253e0cee4fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" Oct 30 20:15:42.238681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.238627 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" event={"ID":"72dc58a1-2629-4df0-b85b-e0271d9f171e","Type":"ContainerStarted","Data":"735e6085edb05f351de43849ad85b3340b96daf525c5eeec4b0a8b48da443c0c"} Oct 30 20:15:42.238921 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.238902 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:42.258161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.258087 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" Oct 30 20:15:42.273368 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.271157 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" podStartSLOduration=3.271125844 podStartE2EDuration="3.271125844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:42.267577614 +0000 UTC m=+2661.962785574" watchObservedRunningTime="2026-10-30 20:15:42.271125844 +0000 UTC m=+2661.966333805" Oct 30 20:15:42.588088 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.587465 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k"] Oct 30 20:15:42.686473 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.670192 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:42.889401 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.886090 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:15:42.906567 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.894218 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:42.983749 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.983707 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-data\") pod \"2bcb265a-c599-404a-b848-b62b6e632f2a\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " Oct 30 20:15:42.983922 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.983840 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-credential-socket\") pod \"2bcb265a-c599-404a-b848-b62b6e632f2a\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " Oct 30 20:15:42.983972 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.983927 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bcb265a-c599-404a-b848-b62b6e632f2a-kube-api-access-9vxp4\" (UniqueName: \"kubernetes.io/projected/2bcb265a-c599-404a-b848-b62b6e632f2a-kube-api-access-9vxp4\") pod \"2bcb265a-c599-404a-b848-b62b6e632f2a\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " Oct 30 20:15:42.984015 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.983982 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-token\" (UniqueName: \"kubernetes.io/projected/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-token\") pod \"2bcb265a-c599-404a-b848-b62b6e632f2a\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " Oct 30 20:15:42.984057 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.984029 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-data" pod "2bcb265a-c599-404a-b848-b62b6e632f2a" (UID: "2bcb265a-c599-404a-b848-b62b6e632f2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:42.984209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.984181 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bcb265a-c599-404a-b848-b62b6e632f2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcb265a-c599-404a-b848-b62b6e632f2a-istiod-ca-cert\") pod \"2bcb265a-c599-404a-b848-b62b6e632f2a\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " Oct 30 20:15:42.984321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.984306 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-workload-socket\") pod \"2bcb265a-c599-404a-b848-b62b6e632f2a\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " Oct 30 20:15:42.984364 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.984346 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-workload-certs\") pod \"2bcb265a-c599-404a-b848-b62b6e632f2a\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " Oct 30 20:15:42.984414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.984401 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-envoy\") pod \"2bcb265a-c599-404a-b848-b62b6e632f2a\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " Oct 30 20:15:42.984463 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.984449 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-podinfo\") pod \"2bcb265a-c599-404a-b848-b62b6e632f2a\" (UID: \"2bcb265a-c599-404a-b848-b62b6e632f2a\") " Oct 30 20:15:42.984748 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.984717 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-workload-socket" pod "2bcb265a-c599-404a-b848-b62b6e632f2a" (UID: "2bcb265a-c599-404a-b848-b62b6e632f2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:42.985058 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.985030 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bcb265a-c599-404a-b848-b62b6e632f2a-istiod-ca-cert" pod "2bcb265a-c599-404a-b848-b62b6e632f2a" (UID: "2bcb265a-c599-404a-b848-b62b6e632f2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:42.985311 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.985287 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-workload-certs" pod "2bcb265a-c599-404a-b848-b62b6e632f2a" (UID: "2bcb265a-c599-404a-b848-b62b6e632f2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:42.985554 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.985528 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2bcb265a-c599-404a-b848-b62b6e632f2a-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:42.985634 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.985561 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:42.985634 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.985575 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:42.985634 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.985589 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:42.986477 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.986451 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-credential-socket" pod "2bcb265a-c599-404a-b848-b62b6e632f2a" (UID: "2bcb265a-c599-404a-b848-b62b6e632f2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:42.986856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.986829 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-token" pod "2bcb265a-c599-404a-b848-b62b6e632f2a" (UID: "2bcb265a-c599-404a-b848-b62b6e632f2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:42.987746 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.987711 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-envoy" pod "2bcb265a-c599-404a-b848-b62b6e632f2a" (UID: "2bcb265a-c599-404a-b848-b62b6e632f2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:15:42.988044 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.988019 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-podinfo" pod "2bcb265a-c599-404a-b848-b62b6e632f2a" (UID: "2bcb265a-c599-404a-b848-b62b6e632f2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:15:42.989057 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:42.989036 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcb265a-c599-404a-b848-b62b6e632f2a-kube-api-access-9vxp4" pod "2bcb265a-c599-404a-b848-b62b6e632f2a" (UID: "2bcb265a-c599-404a-b848-b62b6e632f2a"). InnerVolumeSpecName "kube-api-access-9vxp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:43.087946 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.087912 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:43.087946 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.087949 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:43.088189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.087965 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2bcb265a-c599-404a-b848-b62b6e632f2a-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:43.088189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.087979 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2bcb265a-c599-404a-b848-b62b6e632f2a-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:43.088189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.087994 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vxp4\" (UniqueName: \"kubernetes.io/projected/2bcb265a-c599-404a-b848-b62b6e632f2a-kube-api-access-9vxp4\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:43.337907 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.337850 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" event={"ID":"a8a71502-6ae1-4cb2-8181-b253e0cee4fb","Type":"ContainerStarted","Data":"479ebb6caf129bc33feeeda5da7e01dc23b261031af790869a92a89f82cfb782"} Oct 30 20:15:43.378169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.374207 5091 generic.go:356] "Generic (PLEG): container finished" podID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerID="cd47aab2566f815246ded1d16b7fff7e4122935c0ce49879eaa31dd943fc0d0d" exitCode=0 Oct 30 20:15:43.378169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.374515 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" Oct 30 20:15:43.378169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.377349 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" event={"ID":"2bcb265a-c599-404a-b848-b62b6e632f2a","Type":"ContainerDied","Data":"cd47aab2566f815246ded1d16b7fff7e4122935c0ce49879eaa31dd943fc0d0d"} Oct 30 20:15:43.378169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.377421 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r" event={"ID":"2bcb265a-c599-404a-b848-b62b6e632f2a","Type":"ContainerDied","Data":"a1cf944f327bee06f80248a99ba607a98f109a77c4acd76a47d9dd8bac2fcc79"} Oct 30 20:15:43.378169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.377453 5091 scope.go:122] "RemoveContainer" containerID="cd47aab2566f815246ded1d16b7fff7e4122935c0ce49879eaa31dd943fc0d0d" Oct 30 20:15:43.443783 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.440900 5091 scope.go:122] "RemoveContainer" containerID="b6ee1b25d608772915087d0d56222d6449e4450181ecb3053876f93e3f314c99" Oct 30 20:15:43.451296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.451117 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:43.467361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.463973 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr"] Oct 30 20:15:43.467361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.466444 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerName="istio-proxy" Oct 30 20:15:43.467361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.466469 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerName="istio-proxy" Oct 30 20:15:43.467361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.466549 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerName="istio-proxy" Oct 30 20:15:43.467361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.466558 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerName="istio-proxy" Oct 30 20:15:43.467361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.466767 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerName="istio-proxy" Oct 30 20:15:43.482747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.482321 5091 scope.go:122] "RemoveContainer" containerID="cd47aab2566f815246ded1d16b7fff7e4122935c0ce49879eaa31dd943fc0d0d" Oct 30 20:15:43.483925 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:43.483140 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd47aab2566f815246ded1d16b7fff7e4122935c0ce49879eaa31dd943fc0d0d\": container with ID starting with cd47aab2566f815246ded1d16b7fff7e4122935c0ce49879eaa31dd943fc0d0d not found: ID does not exist" containerID="cd47aab2566f815246ded1d16b7fff7e4122935c0ce49879eaa31dd943fc0d0d" Oct 30 20:15:43.483925 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.483655 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd47aab2566f815246ded1d16b7fff7e4122935c0ce49879eaa31dd943fc0d0d"} err="failed to get container status \"cd47aab2566f815246ded1d16b7fff7e4122935c0ce49879eaa31dd943fc0d0d\": rpc error: code = NotFound desc = could not find container \"cd47aab2566f815246ded1d16b7fff7e4122935c0ce49879eaa31dd943fc0d0d\": container with ID starting with cd47aab2566f815246ded1d16b7fff7e4122935c0ce49879eaa31dd943fc0d0d not found: ID does not exist" Oct 30 20:15:43.483925 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.483680 5091 scope.go:122] "RemoveContainer" containerID="b6ee1b25d608772915087d0d56222d6449e4450181ecb3053876f93e3f314c99" Oct 30 20:15:43.484472 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:43.484386 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ee1b25d608772915087d0d56222d6449e4450181ecb3053876f93e3f314c99\": container with ID starting with b6ee1b25d608772915087d0d56222d6449e4450181ecb3053876f93e3f314c99 not found: ID does not exist" containerID="b6ee1b25d608772915087d0d56222d6449e4450181ecb3053876f93e3f314c99" Oct 30 20:15:43.484472 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.484418 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ee1b25d608772915087d0d56222d6449e4450181ecb3053876f93e3f314c99"} err="failed to get container status \"b6ee1b25d608772915087d0d56222d6449e4450181ecb3053876f93e3f314c99\": rpc error: code = NotFound desc = could not find container \"b6ee1b25d608772915087d0d56222d6449e4450181ecb3053876f93e3f314c99\": container with ID starting with b6ee1b25d608772915087d0d56222d6449e4450181ecb3053876f93e3f314c99 not found: ID does not exist" Oct 30 20:15:43.486442 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.486382 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:43.488246 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.488222 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr"] Oct 30 20:15:43.500479 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.500458 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"routing-overlay\"" Oct 30 20:15:43.509253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.508072 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"kube-root-ca.crt\"" Oct 30 20:15:43.535189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.528566 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"openshift-service-ca.crt\"" Oct 30 20:15:43.535189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.528665 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c9bce8\"" Oct 30 20:15:43.535189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.529165 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"agc-cred-6e22e9b2ae2b\"" Oct 30 20:15:43.535189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.529296 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c9bce8\"/\"payload-processing-external-model-e2e--d955aedb-dockercfg-8b69h\"" Oct 30 20:15:43.615438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.569620 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhblb\" (UniqueName: \"kubernetes.io/projected/40738281-4bd0-4348-9bf2-ca0cccef3197-kube-api-access-nhblb\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:43.654052 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.654009 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40738281-4bd0-4348-9bf2-ca0cccef3197-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:43.683644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.665262 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40738281-4bd0-4348-9bf2-ca0cccef3197-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:43.683644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.665434 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40738281-4bd0-4348-9bf2-ca0cccef3197-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:43.693341 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.686249 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:43.795184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.791999 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhblb\" (UniqueName: \"kubernetes.io/projected/40738281-4bd0-4348-9bf2-ca0cccef3197-kube-api-access-nhblb\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:43.795184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.792107 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40738281-4bd0-4348-9bf2-ca0cccef3197-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:43.795184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.792256 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40738281-4bd0-4348-9bf2-ca0cccef3197-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:43.795184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.792340 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40738281-4bd0-4348-9bf2-ca0cccef3197-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:43.840193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.826058 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40738281-4bd0-4348-9bf2-ca0cccef3197-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:43.840193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.827195 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40738281-4bd0-4348-9bf2-ca0cccef3197-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:43.957169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.924836 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r"] Oct 30 20:15:43.957169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.926092 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhblb\" (UniqueName: \"kubernetes.io/projected/40738281-4bd0-4348-9bf2-ca0cccef3197-kube-api-access-nhblb\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:43.973167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.958200 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40738281-4bd0-4348-9bf2-ca0cccef3197-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:43.992182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:43.989754 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-75b361-openshift-default-d779c697c-c2k7r"] Oct 30 20:15:44.010443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.010404 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:44.380489 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.365205 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:44.468667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.467320 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" path="/var/lib/kubelet/pods/2bcb265a-c599-404a-b848-b62b6e632f2a/volumes" Oct 30 20:15:44.468667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.467858 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr"] Oct 30 20:15:44.485661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.485072 5091 generic.go:356] "Generic (PLEG): container finished" podID="a8a71502-6ae1-4cb2-8181-b253e0cee4fb" containerID="30afe6887606747bea451a55adb89fcd95a557fa76ee6bc71c9ebe46f7df1caf" exitCode=0 Oct 30 20:15:44.485661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.485186 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" event={"ID":"a8a71502-6ae1-4cb2-8181-b253e0cee4fb","Type":"ContainerDied","Data":"30afe6887606747bea451a55adb89fcd95a557fa76ee6bc71c9ebe46f7df1caf"} Oct 30 20:15:44.496574 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.496477 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj"] Oct 30 20:15:44.499203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.497956 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bcb265a-c599-404a-b848-b62b6e632f2a" containerName="istio-proxy" Oct 30 20:15:44.505247 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.505191 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:44.514183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.508836 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj"] Oct 30 20:15:44.523325 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.523084 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e56b6c793ba5\"" Oct 30 20:15:44.523325 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.523125 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-68vtr\"" Oct 30 20:15:44.548391 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.548360 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77872b35-7c24-4fa0-b13f-577261c3366c-routing-overlay-volume\") pod \"payload-processing-external-model-768c87d6b7-mr6cj\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:44.548541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.548524 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77872b35-7c24-4fa0-b13f-577261c3366c-plugins-config-volume\") pod \"payload-processing-external-model-768c87d6b7-mr6cj\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:44.548688 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.548673 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-provider-credentials\") pod \"payload-processing-external-model-768c87d6b7-mr6cj\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:44.548847 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.548831 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w28hx\" (UniqueName: \"kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-kube-api-access-w28hx\") pod \"payload-processing-external-model-768c87d6b7-mr6cj\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:44.598118 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.597388 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr"] Oct 30 20:15:44.618699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.618668 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:44.651046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.651016 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w28hx\" (UniqueName: \"kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-kube-api-access-w28hx\") pod \"payload-processing-external-model-768c87d6b7-mr6cj\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:44.651224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.651207 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77872b35-7c24-4fa0-b13f-577261c3366c-routing-overlay-volume\") pod \"payload-processing-external-model-768c87d6b7-mr6cj\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:44.651365 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.651350 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77872b35-7c24-4fa0-b13f-577261c3366c-plugins-config-volume\") pod \"payload-processing-external-model-768c87d6b7-mr6cj\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:44.651486 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.651466 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-provider-credentials\") pod \"payload-processing-external-model-768c87d6b7-mr6cj\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:44.651677 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:44.651657 5091 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e56b6c793ba5: secret "agc-cred-e56b6c793ba5" not found Oct 30 20:15:44.651733 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:44.651699 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj: secret "agc-cred-e56b6c793ba5" not found Oct 30 20:15:44.651797 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:44.651777 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-provider-credentials podName:77872b35-7c24-4fa0-b13f-577261c3366c nodeName:}" failed. No retries permitted until 2026-10-30 20:15:45.151740098 +0000 UTC m=+2664.846948051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-provider-credentials") pod "payload-processing-external-model-768c87d6b7-mr6cj" (UID: "77872b35-7c24-4fa0-b13f-577261c3366c") : secret "agc-cred-e56b6c793ba5" not found Oct 30 20:15:44.652262 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.652235 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77872b35-7c24-4fa0-b13f-577261c3366c-plugins-config-volume\") pod \"payload-processing-external-model-768c87d6b7-mr6cj\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:44.652791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.652770 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77872b35-7c24-4fa0-b13f-577261c3366c-routing-overlay-volume\") pod \"payload-processing-external-model-768c87d6b7-mr6cj\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:44.665210 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.665181 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w28hx\" (UniqueName: \"kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-kube-api-access-w28hx\") pod \"payload-processing-external-model-768c87d6b7-mr6cj\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:44.801377 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:44.801326 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:45.084239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:45.062360 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:45.184636 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:45.184551 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-provider-credentials\") pod \"payload-processing-external-model-768c87d6b7-mr6cj\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:45.185221 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:45.185196 5091 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e56b6c793ba5: secret "agc-cred-e56b6c793ba5" not found Oct 30 20:15:45.185221 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:45.185223 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj: secret "agc-cred-e56b6c793ba5" not found Oct 30 20:15:45.185524 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:45.185301 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-provider-credentials podName:77872b35-7c24-4fa0-b13f-577261c3366c nodeName:}" failed. No retries permitted until 2026-10-30 20:15:46.185271781 +0000 UTC m=+2665.880479731 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-provider-credentials") pod "payload-processing-external-model-768c87d6b7-mr6cj" (UID: "77872b35-7c24-4fa0-b13f-577261c3366c") : secret "agc-cred-e56b6c793ba5" not found Oct 30 20:15:45.239306 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:45.237773 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:45.611305 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:45.611069 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:45.620011 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:45.619968 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj"] Oct 30 20:15:45.622960 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:45.621682 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" event={"ID":"40738281-4bd0-4348-9bf2-ca0cccef3197","Type":"ContainerStarted","Data":"2f8a0cb031bb3cc906b66567c1caa274d23ed89a5203355e305d16d277fb4c9b"} Oct 30 20:15:45.622960 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:45.621739 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:45.622960 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:45.621768 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" event={"ID":"40738281-4bd0-4348-9bf2-ca0cccef3197","Type":"ContainerStarted","Data":"401cf01d6f378dc28469b12c4fd52f80577b9135f5f56f8ea215ca4fe42d5012"} Oct 30 20:15:45.622960 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:45.622493 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" podUID="77872b35-7c24-4fa0-b13f-577261c3366c" Oct 30 20:15:45.663185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:45.661688 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" podStartSLOduration=2.661669738 podStartE2EDuration="2.661669738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:45.65952836 +0000 UTC m=+2665.354736321" watchObservedRunningTime="2026-10-30 20:15:45.661669738 +0000 UTC m=+2665.356877699" Oct 30 20:15:45.859168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:45.857464 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:46.133255 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:46.128983 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:15:46.156374 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.156331 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" Oct 30 20:15:46.194851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.193720 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-kube-api-access-bqwkm\" (UniqueName: \"kubernetes.io/projected/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-kube-api-access-bqwkm\") pod \"a8a71502-6ae1-4cb2-8181-b253e0cee4fb\" (UID: \"a8a71502-6ae1-4cb2-8181-b253e0cee4fb\") " Oct 30 20:15:46.194851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.193768 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-maas-api-xks-ca\") pod \"a8a71502-6ae1-4cb2-8181-b253e0cee4fb\" (UID: \"a8a71502-6ae1-4cb2-8181-b253e0cee4fb\") " Oct 30 20:15:46.194851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.193825 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-maas-api-openshift-ca\") pod \"a8a71502-6ae1-4cb2-8181-b253e0cee4fb\" (UID: \"a8a71502-6ae1-4cb2-8181-b253e0cee4fb\") " Oct 30 20:15:46.194851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.194297 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-provider-credentials\") pod \"payload-processing-external-model-768c87d6b7-mr6cj\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:46.198205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.195864 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-maas-api-openshift-ca" pod "a8a71502-6ae1-4cb2-8181-b253e0cee4fb" (UID: "a8a71502-6ae1-4cb2-8181-b253e0cee4fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:46.198205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.198066 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-maas-api-xks-ca" pod "a8a71502-6ae1-4cb2-8181-b253e0cee4fb" (UID: "a8a71502-6ae1-4cb2-8181-b253e0cee4fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:46.198917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.198788 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-provider-credentials\") pod \"payload-processing-external-model-768c87d6b7-mr6cj\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:46.200053 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.200026 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-kube-api-access-bqwkm" pod "a8a71502-6ae1-4cb2-8181-b253e0cee4fb" (UID: "a8a71502-6ae1-4cb2-8181-b253e0cee4fb"). InnerVolumeSpecName "kube-api-access-bqwkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:46.295918 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.295841 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqwkm\" (UniqueName: \"kubernetes.io/projected/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-kube-api-access-bqwkm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:46.295918 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.295875 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:46.295918 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.295888 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8a71502-6ae1-4cb2-8181-b253e0cee4fb-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:46.630283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.630204 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" Oct 30 20:15:46.630680 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.630203 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k" event={"ID":"a8a71502-6ae1-4cb2-8181-b253e0cee4fb","Type":"ContainerDied","Data":"479ebb6caf129bc33feeeda5da7e01dc23b261031af790869a92a89f82cfb782"} Oct 30 20:15:46.630680 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.630346 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="479ebb6caf129bc33feeeda5da7e01dc23b261031af790869a92a89f82cfb782" Oct 30 20:15:46.630680 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.630368 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:46.639514 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.639493 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:46.702486 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.702454 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-kube-api-access-w28hx\" (UniqueName: \"kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-kube-api-access-w28hx\") pod \"77872b35-7c24-4fa0-b13f-577261c3366c\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " Oct 30 20:15:46.702667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.702501 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-provider-credentials\") pod \"77872b35-7c24-4fa0-b13f-577261c3366c\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " Oct 30 20:15:46.702667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.702654 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77872b35-7c24-4fa0-b13f-577261c3366c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77872b35-7c24-4fa0-b13f-577261c3366c-plugins-config-volume\") pod \"77872b35-7c24-4fa0-b13f-577261c3366c\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " Oct 30 20:15:46.702770 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.702751 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77872b35-7c24-4fa0-b13f-577261c3366c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77872b35-7c24-4fa0-b13f-577261c3366c-routing-overlay-volume\") pod \"77872b35-7c24-4fa0-b13f-577261c3366c\" (UID: \"77872b35-7c24-4fa0-b13f-577261c3366c\") " Oct 30 20:15:46.703046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.703017 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77872b35-7c24-4fa0-b13f-577261c3366c-plugins-config-volume" pod "77872b35-7c24-4fa0-b13f-577261c3366c" (UID: "77872b35-7c24-4fa0-b13f-577261c3366c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:46.703224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.703092 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77872b35-7c24-4fa0-b13f-577261c3366c-routing-overlay-volume" pod "77872b35-7c24-4fa0-b13f-577261c3366c" (UID: "77872b35-7c24-4fa0-b13f-577261c3366c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:46.703422 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.703401 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77872b35-7c24-4fa0-b13f-577261c3366c-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:46.703510 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.703424 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/77872b35-7c24-4fa0-b13f-577261c3366c-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:46.704733 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.704709 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-kube-api-access-w28hx" pod "77872b35-7c24-4fa0-b13f-577261c3366c" (UID: "77872b35-7c24-4fa0-b13f-577261c3366c"). InnerVolumeSpecName "kube-api-access-w28hx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:46.704834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.704741 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-provider-credentials" pod "77872b35-7c24-4fa0-b13f-577261c3366c" (UID: "77872b35-7c24-4fa0-b13f-577261c3366c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:46.800174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.798341 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:46.817170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.811538 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w28hx\" (UniqueName: \"kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-kube-api-access-w28hx\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:46.817170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.811573 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/77872b35-7c24-4fa0-b13f-577261c3366c-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:46.944392 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.944318 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh"] Oct 30 20:15:46.945105 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.945089 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a8a71502-6ae1-4cb2-8181-b253e0cee4fb" containerName="revoke-keys" Oct 30 20:15:46.945209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.945107 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8a71502-6ae1-4cb2-8181-b253e0cee4fb" containerName="revoke-keys" Oct 30 20:15:46.945282 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.945268 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8a71502-6ae1-4cb2-8181-b253e0cee4fb" containerName="revoke-keys" Oct 30 20:15:46.949328 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.949308 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:46.964049 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.964020 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh"] Oct 30 20:15:46.964762 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.964741 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-wnqnd\"" Oct 30 20:15:46.983167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:46.983126 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:47.013595 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.013568 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrsls\" (UniqueName: \"kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-kube-api-access-xrsls\") pod \"payload-processing-external-model-768c87d6b7-cjddh\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:47.013755 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.013630 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-provider-credentials\") pod \"payload-processing-external-model-768c87d6b7-cjddh\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:47.013804 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.013779 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-routing-overlay-volume\") pod \"payload-processing-external-model-768c87d6b7-cjddh\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:47.013886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.013871 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-plugins-config-volume\") pod \"payload-processing-external-model-768c87d6b7-cjddh\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:47.109171 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.086165 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice/crio-1ae08a3ca619d3c72e4eecfe6172f0a90cbfe89251b05e4988849b12b55b1843\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8a41b6_c1fd_4d9b_8f3d_c0a93eb8434c.slice/crio-5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice\": RecentStats: unable to find data in memory cache]" Oct 30 20:15:47.176170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.149209 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-routing-overlay-volume\") pod \"payload-processing-external-model-768c87d6b7-cjddh\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:47.176170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.149325 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-plugins-config-volume\") pod \"payload-processing-external-model-768c87d6b7-cjddh\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:47.176170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.149549 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrsls\" (UniqueName: \"kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-kube-api-access-xrsls\") pod \"payload-processing-external-model-768c87d6b7-cjddh\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:47.176170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.149617 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-provider-credentials\") pod \"payload-processing-external-model-768c87d6b7-cjddh\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:47.176170 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.149860 5091 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e56b6c793ba5: secret "agc-cred-e56b6c793ba5" not found Oct 30 20:15:47.176170 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.149880 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh: secret "agc-cred-e56b6c793ba5" not found Oct 30 20:15:47.176170 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.149943 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-provider-credentials podName:39f81e95-43c7-45e3-b02f-6931c527f4dd nodeName:}" failed. No retries permitted until 2026-10-30 20:15:47.649921308 +0000 UTC m=+2667.345129265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-provider-credentials") pod "payload-processing-external-model-768c87d6b7-cjddh" (UID: "39f81e95-43c7-45e3-b02f-6931c527f4dd") : secret "agc-cred-e56b6c793ba5" not found Oct 30 20:15:47.176170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.150960 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-routing-overlay-volume\") pod \"payload-processing-external-model-768c87d6b7-cjddh\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:47.176170 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.151037 5091 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 30 20:15:47.176170 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.151090 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-plugins-config-volume podName:39f81e95-43c7-45e3-b02f-6931c527f4dd nodeName:}" failed. No retries permitted until 2026-10-30 20:15:47.651070515 +0000 UTC m=+2667.346278471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-plugins-config-volume") pod "payload-processing-external-model-768c87d6b7-cjddh" (UID: "39f81e95-43c7-45e3-b02f-6931c527f4dd") : configmap "payload-processing-external-model-plugins" not found Oct 30 20:15:47.176170 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.158729 5091 projected.go:196] Error preparing data for projected volume kube-api-access-xrsls for pod models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 20:15:47.176170 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.158789 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-kube-api-access-xrsls podName:39f81e95-43c7-45e3-b02f-6931c527f4dd nodeName:}" failed. No retries permitted until 2026-10-30 20:15:47.658771598 +0000 UTC m=+2667.353979552 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-xrsls" (UniqueName: "kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-kube-api-access-xrsls") pod "payload-processing-external-model-768c87d6b7-cjddh" (UID: "39f81e95-43c7-45e3-b02f-6931c527f4dd") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 20:15:47.228628 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.228519 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh"] Oct 30 20:15:47.229662 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.229612 5091 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-xrsls plugins-config-volume provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" podUID="39f81e95-43c7-45e3-b02f-6931c527f4dd" Oct 30 20:15:47.300446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.300128 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:47.304080 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.304037 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" podUID="72dc58a1-2629-4df0-b85b-e0271d9f171e" containerName="payload-processing" containerID="cri-o://735e6085edb05f351de43849ad85b3340b96daf525c5eeec4b0a8b48da443c0c" gracePeriod=25 Oct 30 20:15:47.550118 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.550030 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:47.753516 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.753472 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-plugins-config-volume\") pod \"payload-processing-external-model-768c87d6b7-cjddh\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:47.754419 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.754390 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrsls\" (UniqueName: \"kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-kube-api-access-xrsls\") pod \"payload-processing-external-model-768c87d6b7-cjddh\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:47.754951 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.754930 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-provider-credentials\") pod \"payload-processing-external-model-768c87d6b7-cjddh\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:47.756459 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.754130 5091 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 30 20:15:47.756649 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.756635 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-plugins-config-volume podName:39f81e95-43c7-45e3-b02f-6931c527f4dd nodeName:}" failed. No retries permitted until 2026-10-30 20:15:48.756612995 +0000 UTC m=+2668.451820935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-plugins-config-volume") pod "payload-processing-external-model-768c87d6b7-cjddh" (UID: "39f81e95-43c7-45e3-b02f-6931c527f4dd") : configmap "payload-processing-external-model-plugins" not found Oct 30 20:15:47.757115 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.755134 5091 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e56b6c793ba5: secret "agc-cred-e56b6c793ba5" not found Oct 30 20:15:47.757276 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.757264 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh: secret "agc-cred-e56b6c793ba5" not found Oct 30 20:15:47.757436 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.757424 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-provider-credentials podName:39f81e95-43c7-45e3-b02f-6931c527f4dd nodeName:}" failed. No retries permitted until 2026-10-30 20:15:48.757407288 +0000 UTC m=+2668.452615242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-provider-credentials") pod "payload-processing-external-model-768c87d6b7-cjddh" (UID: "39f81e95-43c7-45e3-b02f-6931c527f4dd") : secret "agc-cred-e56b6c793ba5" not found Oct 30 20:15:47.811218 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.801802 5091 projected.go:196] Error preparing data for projected volume kube-api-access-xrsls for pod models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 20:15:47.811218 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:47.801887 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-kube-api-access-xrsls podName:39f81e95-43c7-45e3-b02f-6931c527f4dd nodeName:}" failed. No retries permitted until 2026-10-30 20:15:48.801864299 +0000 UTC m=+2668.497072235 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-xrsls" (UniqueName: "kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-kube-api-access-xrsls") pod "payload-processing-external-model-768c87d6b7-cjddh" (UID: "39f81e95-43c7-45e3-b02f-6931c527f4dd") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 20:15:47.868282 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.859797 5091 generic.go:356] "Generic (PLEG): container finished" podID="72dc58a1-2629-4df0-b85b-e0271d9f171e" containerID="735e6085edb05f351de43849ad85b3340b96daf525c5eeec4b0a8b48da443c0c" exitCode=0 Oct 30 20:15:47.868282 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.860053 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:47.868282 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.860686 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" event={"ID":"72dc58a1-2629-4df0-b85b-e0271d9f171e","Type":"ContainerDied","Data":"735e6085edb05f351de43849ad85b3340b96daf525c5eeec4b0a8b48da443c0c"} Oct 30 20:15:47.868282 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.860772 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj" Oct 30 20:15:47.985579 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:47.984350 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:48.107304 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.101140 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-routing-overlay-volume\") pod \"39f81e95-43c7-45e3-b02f-6931c527f4dd\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " Oct 30 20:15:48.113928 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.113843 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-routing-overlay-volume" pod "39f81e95-43c7-45e3-b02f-6931c527f4dd" (UID: "39f81e95-43c7-45e3-b02f-6931c527f4dd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:48.117800 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.117691 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:48.190286 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.190248 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj"] Oct 30 20:15:48.193576 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.193557 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-768c87d6b7-mr6cj"] Oct 30 20:15:48.203446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.203405 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.248899 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.248873 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:48.298595 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.298559 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw"] Oct 30 20:15:48.299742 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.299710 5091 cpu_manager.go:414] "containerMap: removing container" podUID="72dc58a1-2629-4df0-b85b-e0271d9f171e" containerName="payload-processing" Oct 30 20:15:48.299742 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.299740 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72dc58a1-2629-4df0-b85b-e0271d9f171e" containerName="payload-processing" Oct 30 20:15:48.300013 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.299997 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="72dc58a1-2629-4df0-b85b-e0271d9f171e" containerName="payload-processing" Oct 30 20:15:48.304516 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.304494 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" Oct 30 20:15:48.304939 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.304907 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72dc58a1-2629-4df0-b85b-e0271d9f171e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72dc58a1-2629-4df0-b85b-e0271d9f171e-plugins-config-volume\") pod \"72dc58a1-2629-4df0-b85b-e0271d9f171e\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " Oct 30 20:15:48.305207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.305187 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72dc58a1-2629-4df0-b85b-e0271d9f171e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72dc58a1-2629-4df0-b85b-e0271d9f171e-routing-overlay-volume\") pod \"72dc58a1-2629-4df0-b85b-e0271d9f171e\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " Oct 30 20:15:48.305265 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.305247 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72dc58a1-2629-4df0-b85b-e0271d9f171e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/72dc58a1-2629-4df0-b85b-e0271d9f171e-provider-credentials\") pod \"72dc58a1-2629-4df0-b85b-e0271d9f171e\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " Oct 30 20:15:48.305425 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.305377 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72dc58a1-2629-4df0-b85b-e0271d9f171e-kube-api-access-dcn7b\" (UniqueName: \"kubernetes.io/projected/72dc58a1-2629-4df0-b85b-e0271d9f171e-kube-api-access-dcn7b\") pod \"72dc58a1-2629-4df0-b85b-e0271d9f171e\" (UID: \"72dc58a1-2629-4df0-b85b-e0271d9f171e\") " Oct 30 20:15:48.305698 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.305669 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72dc58a1-2629-4df0-b85b-e0271d9f171e-plugins-config-volume" pod "72dc58a1-2629-4df0-b85b-e0271d9f171e" (UID: "72dc58a1-2629-4df0-b85b-e0271d9f171e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:48.306447 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.306420 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72dc58a1-2629-4df0-b85b-e0271d9f171e-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.306620 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.306473 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72dc58a1-2629-4df0-b85b-e0271d9f171e-routing-overlay-volume" pod "72dc58a1-2629-4df0-b85b-e0271d9f171e" (UID: "72dc58a1-2629-4df0-b85b-e0271d9f171e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:48.309124 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.309095 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72dc58a1-2629-4df0-b85b-e0271d9f171e-provider-credentials" pod "72dc58a1-2629-4df0-b85b-e0271d9f171e" (UID: "72dc58a1-2629-4df0-b85b-e0271d9f171e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:48.310801 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.310777 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw"] Oct 30 20:15:48.311629 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.311606 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:48.311725 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.311696 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:15:48.315112 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.315080 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72dc58a1-2629-4df0-b85b-e0271d9f171e-kube-api-access-dcn7b" pod "72dc58a1-2629-4df0-b85b-e0271d9f171e" (UID: "72dc58a1-2629-4df0-b85b-e0271d9f171e"). InnerVolumeSpecName "kube-api-access-dcn7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:48.407466 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.407385 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75ft2\" (UniqueName: \"kubernetes.io/projected/70b8ac16-887e-4b69-b088-5a71eac646ee-kube-api-access-75ft2\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw\" (UID: \"70b8ac16-887e-4b69-b088-5a71eac646ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" Oct 30 20:15:48.407635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.407617 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b8ac16-887e-4b69-b088-5a71eac646ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw\" (UID: \"70b8ac16-887e-4b69-b088-5a71eac646ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" Oct 30 20:15:48.407683 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.407647 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b8ac16-887e-4b69-b088-5a71eac646ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw\" (UID: \"70b8ac16-887e-4b69-b088-5a71eac646ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" Oct 30 20:15:48.407844 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.407820 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72dc58a1-2629-4df0-b85b-e0271d9f171e-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.407915 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.407845 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72dc58a1-2629-4df0-b85b-e0271d9f171e-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.407915 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.407855 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcn7b\" (UniqueName: \"kubernetes.io/projected/72dc58a1-2629-4df0-b85b-e0271d9f171e-kube-api-access-dcn7b\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:48.442762 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.442729 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77872b35-7c24-4fa0-b13f-577261c3366c" path="/var/lib/kubelet/pods/77872b35-7c24-4fa0-b13f-577261c3366c/volumes" Oct 30 20:15:48.509565 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.509531 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75ft2\" (UniqueName: \"kubernetes.io/projected/70b8ac16-887e-4b69-b088-5a71eac646ee-kube-api-access-75ft2\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw\" (UID: \"70b8ac16-887e-4b69-b088-5a71eac646ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" Oct 30 20:15:48.509873 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.509854 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b8ac16-887e-4b69-b088-5a71eac646ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw\" (UID: \"70b8ac16-887e-4b69-b088-5a71eac646ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" Oct 30 20:15:48.509981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.509914 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b8ac16-887e-4b69-b088-5a71eac646ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw\" (UID: \"70b8ac16-887e-4b69-b088-5a71eac646ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" Oct 30 20:15:48.510689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.510663 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b8ac16-887e-4b69-b088-5a71eac646ee-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw\" (UID: \"70b8ac16-887e-4b69-b088-5a71eac646ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" Oct 30 20:15:48.512344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.512314 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b8ac16-887e-4b69-b088-5a71eac646ee-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw\" (UID: \"70b8ac16-887e-4b69-b088-5a71eac646ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" Oct 30 20:15:48.528889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.528863 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75ft2\" (UniqueName: \"kubernetes.io/projected/70b8ac16-887e-4b69-b088-5a71eac646ee-kube-api-access-75ft2\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw\" (UID: \"70b8ac16-887e-4b69-b088-5a71eac646ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" Oct 30 20:15:48.661991 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.661914 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" Oct 30 20:15:48.815359 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.815320 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrsls\" (UniqueName: \"kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-kube-api-access-xrsls\") pod \"payload-processing-external-model-768c87d6b7-cjddh\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:48.815723 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.815393 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-provider-credentials\") pod \"payload-processing-external-model-768c87d6b7-cjddh\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:48.815723 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:48.815482 5091 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e56b6c793ba5: secret "agc-cred-e56b6c793ba5" not found Oct 30 20:15:48.815723 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:48.815507 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh: secret "agc-cred-e56b6c793ba5" not found Oct 30 20:15:48.815723 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.815531 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-plugins-config-volume\") pod \"payload-processing-external-model-768c87d6b7-cjddh\" (UID: \"39f81e95-43c7-45e3-b02f-6931c527f4dd\") " pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:48.815723 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:48.815565 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-provider-credentials podName:39f81e95-43c7-45e3-b02f-6931c527f4dd nodeName:}" failed. No retries permitted until 2026-10-30 20:15:50.815546001 +0000 UTC m=+2670.510753961 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-provider-credentials") pod "payload-processing-external-model-768c87d6b7-cjddh" (UID: "39f81e95-43c7-45e3-b02f-6931c527f4dd") : secret "agc-cred-e56b6c793ba5" not found Oct 30 20:15:48.815723 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:48.815590 5091 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Oct 30 20:15:48.815723 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:48.815653 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-plugins-config-volume podName:39f81e95-43c7-45e3-b02f-6931c527f4dd nodeName:}" failed. No retries permitted until 2026-10-30 20:15:50.815637105 +0000 UTC m=+2670.510845059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-plugins-config-volume") pod "payload-processing-external-model-768c87d6b7-cjddh" (UID: "39f81e95-43c7-45e3-b02f-6931c527f4dd") : configmap "payload-processing-external-model-plugins" not found Oct 30 20:15:48.842635 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:48.842610 5091 projected.go:196] Error preparing data for projected volume kube-api-access-xrsls for pod models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh: failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 20:15:48.842771 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:48.842669 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-kube-api-access-xrsls podName:39f81e95-43c7-45e3-b02f-6931c527f4dd nodeName:}" failed. No retries permitted until 2026-10-30 20:15:50.842655151 +0000 UTC m=+2670.537863094 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-xrsls" (UniqueName: "kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-kube-api-access-xrsls") pod "payload-processing-external-model-768c87d6b7-cjddh" (UID: "39f81e95-43c7-45e3-b02f-6931c527f4dd") : failed to fetch token: serviceaccounts "payload-processing-external-model" not found Oct 30 20:15:48.871292 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.871252 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" Oct 30 20:15:48.871292 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.871272 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc" event={"ID":"72dc58a1-2629-4df0-b85b-e0271d9f171e","Type":"ContainerDied","Data":"53b2fe0e25f49db6d284672e6a1d3f0db02c87b409a886d522060f3bb1c1b9ed"} Oct 30 20:15:48.871547 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.871342 5091 scope.go:122] "RemoveContainer" containerID="735e6085edb05f351de43849ad85b3340b96daf525c5eeec4b0a8b48da443c0c" Oct 30 20:15:48.871547 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.871406 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh" Oct 30 20:15:48.925711 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.925628 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc"] Oct 30 20:15:48.932058 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.932029 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d49c6d66d-wx6hc"] Oct 30 20:15:48.969440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.969413 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw"] Oct 30 20:15:48.970245 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:48.970221 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b8ac16_887e_4b69_b088_5a71eac646ee.slice/crio-6f614b336cc114b050d84c3bb07347755b5bb4cb168866beba5e68ee5d90940f WatchSource:0}: Error finding container 6f614b336cc114b050d84c3bb07347755b5bb4cb168866beba5e68ee5d90940f: Status 404 returned error can't find the container with id 6f614b336cc114b050d84c3bb07347755b5bb4cb168866beba5e68ee5d90940f Oct 30 20:15:48.976636 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.976579 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh"] Oct 30 20:15:48.982160 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:48.982124 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-768c87d6b7-cjddh"] Oct 30 20:15:49.022025 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:49.021990 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrsls\" (UniqueName: \"kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-kube-api-access-xrsls\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:49.022025 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:49.022017 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/39f81e95-43c7-45e3-b02f-6931c527f4dd-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:49.022025 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:49.022029 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/39f81e95-43c7-45e3-b02f-6931c527f4dd-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:49.445038 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:49.440353 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:49.672184 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:49.672120 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8a41b6_c1fd_4d9b_8f3d_c0a93eb8434c.slice/crio-5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice/crio-1ae08a3ca619d3c72e4eecfe6172f0a90cbfe89251b05e4988849b12b55b1843\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice\": RecentStats: unable to find data in memory cache]" Oct 30 20:15:49.886296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:49.886191 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" event={"ID":"70b8ac16-887e-4b69-b088-5a71eac646ee","Type":"ContainerStarted","Data":"5c6e70682bb09bae6664f9e5835acfe4c62e4cbadd23687f5c608de27b9ad9c9"} Oct 30 20:15:49.886296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:49.886264 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" event={"ID":"70b8ac16-887e-4b69-b088-5a71eac646ee","Type":"ContainerStarted","Data":"6f614b336cc114b050d84c3bb07347755b5bb4cb168866beba5e68ee5d90940f"} Oct 30 20:15:49.924465 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:49.924415 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" podStartSLOduration=1.9244013290000002 podStartE2EDuration="1.924401329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:49.919890326 +0000 UTC m=+2669.615098286" watchObservedRunningTime="2026-10-30 20:15:49.924401329 +0000 UTC m=+2669.619609323" Oct 30 20:15:50.445738 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:50.445643 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39f81e95-43c7-45e3-b02f-6931c527f4dd" path="/var/lib/kubelet/pods/39f81e95-43c7-45e3-b02f-6931c527f4dd/volumes" Oct 30 20:15:50.446173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:50.446131 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72dc58a1-2629-4df0-b85b-e0271d9f171e" path="/var/lib/kubelet/pods/72dc58a1-2629-4df0-b85b-e0271d9f171e/volumes" Oct 30 20:15:50.525167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:50.524962 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:50.621759 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:50.621714 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" podUID="40738281-4bd0-4348-9bf2-ca0cccef3197" containerName="payload-processing" containerID="cri-o://2f8a0cb031bb3cc906b66567c1caa274d23ed89a5203355e305d16d277fb4c9b" gracePeriod=25 Oct 30 20:15:51.102749 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.102709 5091 generic.go:356] "Generic (PLEG): container finished" podID="70b8ac16-887e-4b69-b088-5a71eac646ee" containerID="5c6e70682bb09bae6664f9e5835acfe4c62e4cbadd23687f5c608de27b9ad9c9" exitCode=0 Oct 30 20:15:51.103917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.103748 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" event={"ID":"70b8ac16-887e-4b69-b088-5a71eac646ee","Type":"ContainerDied","Data":"5c6e70682bb09bae6664f9e5835acfe4c62e4cbadd23687f5c608de27b9ad9c9"} Oct 30 20:15:51.115538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.112406 5091 generic.go:356] "Generic (PLEG): container finished" podID="40738281-4bd0-4348-9bf2-ca0cccef3197" containerID="2f8a0cb031bb3cc906b66567c1caa274d23ed89a5203355e305d16d277fb4c9b" exitCode=0 Oct 30 20:15:51.115538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.112459 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" event={"ID":"40738281-4bd0-4348-9bf2-ca0cccef3197","Type":"ContainerDied","Data":"2f8a0cb031bb3cc906b66567c1caa274d23ed89a5203355e305d16d277fb4c9b"} Oct 30 20:15:51.163475 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.163442 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:51.185886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.185855 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40738281-4bd0-4348-9bf2-ca0cccef3197-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40738281-4bd0-4348-9bf2-ca0cccef3197-routing-overlay-volume\") pod \"40738281-4bd0-4348-9bf2-ca0cccef3197\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " Oct 30 20:15:51.186035 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.185916 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40738281-4bd0-4348-9bf2-ca0cccef3197-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40738281-4bd0-4348-9bf2-ca0cccef3197-plugins-config-volume\") pod \"40738281-4bd0-4348-9bf2-ca0cccef3197\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " Oct 30 20:15:51.186035 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.185950 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40738281-4bd0-4348-9bf2-ca0cccef3197-kube-api-access-nhblb\" (UniqueName: \"kubernetes.io/projected/40738281-4bd0-4348-9bf2-ca0cccef3197-kube-api-access-nhblb\") pod \"40738281-4bd0-4348-9bf2-ca0cccef3197\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " Oct 30 20:15:51.186260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.186236 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40738281-4bd0-4348-9bf2-ca0cccef3197-provider-credentials\" (UniqueName: \"kubernetes.io/projected/40738281-4bd0-4348-9bf2-ca0cccef3197-provider-credentials\") pod \"40738281-4bd0-4348-9bf2-ca0cccef3197\" (UID: \"40738281-4bd0-4348-9bf2-ca0cccef3197\") " Oct 30 20:15:51.186403 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.186262 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40738281-4bd0-4348-9bf2-ca0cccef3197-routing-overlay-volume" pod "40738281-4bd0-4348-9bf2-ca0cccef3197" (UID: "40738281-4bd0-4348-9bf2-ca0cccef3197"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:51.186403 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.186287 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40738281-4bd0-4348-9bf2-ca0cccef3197-plugins-config-volume" pod "40738281-4bd0-4348-9bf2-ca0cccef3197" (UID: "40738281-4bd0-4348-9bf2-ca0cccef3197"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:51.186883 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.186862 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40738281-4bd0-4348-9bf2-ca0cccef3197-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:51.186991 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.186884 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40738281-4bd0-4348-9bf2-ca0cccef3197-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:51.188229 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.188202 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40738281-4bd0-4348-9bf2-ca0cccef3197-kube-api-access-nhblb" pod "40738281-4bd0-4348-9bf2-ca0cccef3197" (UID: "40738281-4bd0-4348-9bf2-ca0cccef3197"). InnerVolumeSpecName "kube-api-access-nhblb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:51.188423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.188400 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40738281-4bd0-4348-9bf2-ca0cccef3197-provider-credentials" pod "40738281-4bd0-4348-9bf2-ca0cccef3197" (UID: "40738281-4bd0-4348-9bf2-ca0cccef3197"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:51.288882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.288844 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40738281-4bd0-4348-9bf2-ca0cccef3197-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:51.288882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.288873 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhblb\" (UniqueName: \"kubernetes.io/projected/40738281-4bd0-4348-9bf2-ca0cccef3197-kube-api-access-nhblb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:51.637177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:51.636065 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:52.165882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.165073 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:52.176367 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.176336 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" Oct 30 20:15:52.179394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.179298 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr" event={"ID":"40738281-4bd0-4348-9bf2-ca0cccef3197","Type":"ContainerDied","Data":"401cf01d6f378dc28469b12c4fd52f80577b9135f5f56f8ea215ca4fe42d5012"} Oct 30 20:15:52.179394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.179365 5091 scope.go:122] "RemoveContainer" containerID="2f8a0cb031bb3cc906b66567c1caa274d23ed89a5203355e305d16d277fb4c9b" Oct 30 20:15:52.315393 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.315323 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr"] Oct 30 20:15:52.331593 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.327368 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:52.431503 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.431417 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c9bce8/payload-processing-external-model-e2e-extmod-a-c9bce8-5bbcfr2zr"] Oct 30 20:15:52.585912 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.585833 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:52.588612 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.588578 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40738281-4bd0-4348-9bf2-ca0cccef3197" path="/var/lib/kubelet/pods/40738281-4bd0-4348-9bf2-ca0cccef3197/volumes" Oct 30 20:15:52.689196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.689119 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" Oct 30 20:15:52.692809 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.692782 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k"] Oct 30 20:15:52.698739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.698713 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-68685b-cmg9k"] Oct 30 20:15:52.743827 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.743798 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70b8ac16-887e-4b69-b088-5a71eac646ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b8ac16-887e-4b69-b088-5a71eac646ee-maas-api-xks-ca\") pod \"70b8ac16-887e-4b69-b088-5a71eac646ee\" (UID: \"70b8ac16-887e-4b69-b088-5a71eac646ee\") " Oct 30 20:15:52.744022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.743947 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b8ac16-887e-4b69-b088-5a71eac646ee-kube-api-access-75ft2\" (UniqueName: \"kubernetes.io/projected/70b8ac16-887e-4b69-b088-5a71eac646ee-kube-api-access-75ft2\") pod \"70b8ac16-887e-4b69-b088-5a71eac646ee\" (UID: \"70b8ac16-887e-4b69-b088-5a71eac646ee\") " Oct 30 20:15:52.744097 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.744074 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70b8ac16-887e-4b69-b088-5a71eac646ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b8ac16-887e-4b69-b088-5a71eac646ee-maas-api-openshift-ca\") pod \"70b8ac16-887e-4b69-b088-5a71eac646ee\" (UID: \"70b8ac16-887e-4b69-b088-5a71eac646ee\") " Oct 30 20:15:52.744975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.744946 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b8ac16-887e-4b69-b088-5a71eac646ee-maas-api-openshift-ca" pod "70b8ac16-887e-4b69-b088-5a71eac646ee" (UID: "70b8ac16-887e-4b69-b088-5a71eac646ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:52.745738 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.745702 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:52.746734 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.746708 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70b8ac16-887e-4b69-b088-5a71eac646ee-maas-api-xks-ca" pod "70b8ac16-887e-4b69-b088-5a71eac646ee" (UID: "70b8ac16-887e-4b69-b088-5a71eac646ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:52.746856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.746745 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b8ac16-887e-4b69-b088-5a71eac646ee-kube-api-access-75ft2" pod "70b8ac16-887e-4b69-b088-5a71eac646ee" (UID: "70b8ac16-887e-4b69-b088-5a71eac646ee"). InnerVolumeSpecName "kube-api-access-75ft2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:52.846272 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.846232 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/70b8ac16-887e-4b69-b088-5a71eac646ee-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:52.846500 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.846479 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/70b8ac16-887e-4b69-b088-5a71eac646ee-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:52.846602 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.846503 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75ft2\" (UniqueName: \"kubernetes.io/projected/70b8ac16-887e-4b69-b088-5a71eac646ee-kube-api-access-75ft2\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:52.913067 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:52.913024 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:53.187166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:53.187127 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" Oct 30 20:15:53.187655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:53.187166 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw" event={"ID":"70b8ac16-887e-4b69-b088-5a71eac646ee","Type":"ContainerDied","Data":"6f614b336cc114b050d84c3bb07347755b5bb4cb168866beba5e68ee5d90940f"} Oct 30 20:15:53.187655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:53.187227 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f614b336cc114b050d84c3bb07347755b5bb4cb168866beba5e68ee5d90940f" Oct 30 20:15:53.197994 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:53.197960 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:53.500176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:53.500056 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:53.985685 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:53.985598 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:54.140492 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.140452 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:54.296203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.296116 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:54.447659 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.447610 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8a71502-6ae1-4cb2-8181-b253e0cee4fb" path="/var/lib/kubelet/pods/a8a71502-6ae1-4cb2-8181-b253e0cee4fb/volumes" Oct 30 20:15:54.448002 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.447985 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9"] Oct 30 20:15:54.448707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.448686 5091 cpu_manager.go:414] "containerMap: removing container" podUID="40738281-4bd0-4348-9bf2-ca0cccef3197" containerName="payload-processing" Oct 30 20:15:54.448707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.448705 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40738281-4bd0-4348-9bf2-ca0cccef3197" containerName="payload-processing" Oct 30 20:15:54.448924 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.448748 5091 cpu_manager.go:414] "containerMap: removing container" podUID="70b8ac16-887e-4b69-b088-5a71eac646ee" containerName="revoke-keys" Oct 30 20:15:54.448924 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.448754 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b8ac16-887e-4b69-b088-5a71eac646ee" containerName="revoke-keys" Oct 30 20:15:54.449045 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.448948 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="40738281-4bd0-4348-9bf2-ca0cccef3197" containerName="payload-processing" Oct 30 20:15:54.449045 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.448967 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b8ac16-887e-4b69-b088-5a71eac646ee" containerName="revoke-keys" Oct 30 20:15:54.453415 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.453391 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" Oct 30 20:15:54.472664 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.472440 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:15:54.472664 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.472492 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:15:54.489902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.489785 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:54.503182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.498544 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9"] Oct 30 20:15:54.662031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.653254 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p42rd\" (UniqueName: \"kubernetes.io/projected/88837fc6-f0b1-4573-90de-0183fa4024a3-kube-api-access-p42rd\") pod \"maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9\" (UID: \"88837fc6-f0b1-4573-90de-0183fa4024a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" Oct 30 20:15:54.662031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.653594 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88837fc6-f0b1-4573-90de-0183fa4024a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9\" (UID: \"88837fc6-f0b1-4573-90de-0183fa4024a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" Oct 30 20:15:54.662031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.653641 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88837fc6-f0b1-4573-90de-0183fa4024a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9\" (UID: \"88837fc6-f0b1-4573-90de-0183fa4024a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" Oct 30 20:15:54.763544 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.763486 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88837fc6-f0b1-4573-90de-0183fa4024a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9\" (UID: \"88837fc6-f0b1-4573-90de-0183fa4024a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" Oct 30 20:15:54.763752 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.763569 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88837fc6-f0b1-4573-90de-0183fa4024a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9\" (UID: \"88837fc6-f0b1-4573-90de-0183fa4024a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" Oct 30 20:15:54.763752 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.763724 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p42rd\" (UniqueName: \"kubernetes.io/projected/88837fc6-f0b1-4573-90de-0183fa4024a3-kube-api-access-p42rd\") pod \"maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9\" (UID: \"88837fc6-f0b1-4573-90de-0183fa4024a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" Oct 30 20:15:54.824525 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.824196 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88837fc6-f0b1-4573-90de-0183fa4024a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9\" (UID: \"88837fc6-f0b1-4573-90de-0183fa4024a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" Oct 30 20:15:54.827850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.827312 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p42rd\" (UniqueName: \"kubernetes.io/projected/88837fc6-f0b1-4573-90de-0183fa4024a3-kube-api-access-p42rd\") pod \"maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9\" (UID: \"88837fc6-f0b1-4573-90de-0183fa4024a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" Oct 30 20:15:54.830482 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.830185 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:54.842297 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:54.842267 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88837fc6-f0b1-4573-90de-0183fa4024a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9\" (UID: \"88837fc6-f0b1-4573-90de-0183fa4024a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" Oct 30 20:15:55.100855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:55.100824 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" Oct 30 20:15:55.278829 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:55.278770 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:55.443181 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:55.443119 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88837fc6_f0b1_4573_90de_0183fa4024a3.slice/crio-7722c24917325e0052a560228ff009f5ebd41c3cef0e4f1f00ee9f3116137f02 WatchSource:0}: Error finding container 7722c24917325e0052a560228ff009f5ebd41c3cef0e4f1f00ee9f3116137f02: Status 404 returned error can't find the container with id 7722c24917325e0052a560228ff009f5ebd41c3cef0e4f1f00ee9f3116137f02 Oct 30 20:15:55.444125 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:55.444092 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9"] Oct 30 20:15:56.226523 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:56.222546 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:56.260985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:56.256961 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" event={"ID":"88837fc6-f0b1-4573-90de-0183fa4024a3","Type":"ContainerStarted","Data":"f5500571352ec05d957229ba49b52690a8bcf87429b3fd7d69e75c6809606fb6"} Oct 30 20:15:56.260985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:56.257036 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" event={"ID":"88837fc6-f0b1-4573-90de-0183fa4024a3","Type":"ContainerStarted","Data":"7722c24917325e0052a560228ff009f5ebd41c3cef0e4f1f00ee9f3116137f02"} Oct 30 20:15:56.309798 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:56.309727 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" podStartSLOduration=2.309706824 podStartE2EDuration="2.309706824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:15:56.303546963 +0000 UTC m=+2675.998754922" watchObservedRunningTime="2026-10-30 20:15:56.309706824 +0000 UTC m=+2676.004914783" Oct 30 20:15:57.271960 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:57.271922 5091 generic.go:356] "Generic (PLEG): container finished" podID="88837fc6-f0b1-4573-90de-0183fa4024a3" containerID="f5500571352ec05d957229ba49b52690a8bcf87429b3fd7d69e75c6809606fb6" exitCode=0 Oct 30 20:15:57.272390 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:57.272024 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" event={"ID":"88837fc6-f0b1-4573-90de-0183fa4024a3","Type":"ContainerDied","Data":"f5500571352ec05d957229ba49b52690a8bcf87429b3fd7d69e75c6809606fb6"} Oct 30 20:15:57.349169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:57.334004 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:57.463263 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:15:57.463220 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice/crio-1ae08a3ca619d3c72e4eecfe6172f0a90cbfe89251b05e4988849b12b55b1843\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8a41b6_c1fd_4d9b_8f3d_c0a93eb8434c.slice/crio-5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397\": RecentStats: unable to find data in memory cache]" Oct 30 20:15:57.670562 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:57.670470 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:57.816274 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:57.816231 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:58.458438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:58.458411 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" Oct 30 20:15:58.551317 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:58.542925 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88837fc6-f0b1-4573-90de-0183fa4024a3-kube-api-access-p42rd\" (UniqueName: \"kubernetes.io/projected/88837fc6-f0b1-4573-90de-0183fa4024a3-kube-api-access-p42rd\") pod \"88837fc6-f0b1-4573-90de-0183fa4024a3\" (UID: \"88837fc6-f0b1-4573-90de-0183fa4024a3\") " Oct 30 20:15:58.556001 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:58.555955 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:58.581523 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:58.581382 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88837fc6-f0b1-4573-90de-0183fa4024a3-kube-api-access-p42rd" pod "88837fc6-f0b1-4573-90de-0183fa4024a3" (UID: "88837fc6-f0b1-4573-90de-0183fa4024a3"). InnerVolumeSpecName "kube-api-access-p42rd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:15:58.645056 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:58.645019 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88837fc6-f0b1-4573-90de-0183fa4024a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88837fc6-f0b1-4573-90de-0183fa4024a3-maas-api-xks-ca\") pod \"88837fc6-f0b1-4573-90de-0183fa4024a3\" (UID: \"88837fc6-f0b1-4573-90de-0183fa4024a3\") " Oct 30 20:15:58.645357 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:58.645338 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88837fc6-f0b1-4573-90de-0183fa4024a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88837fc6-f0b1-4573-90de-0183fa4024a3-maas-api-openshift-ca\") pod \"88837fc6-f0b1-4573-90de-0183fa4024a3\" (UID: \"88837fc6-f0b1-4573-90de-0183fa4024a3\") " Oct 30 20:15:58.645794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:58.645768 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88837fc6-f0b1-4573-90de-0183fa4024a3-maas-api-openshift-ca" pod "88837fc6-f0b1-4573-90de-0183fa4024a3" (UID: "88837fc6-f0b1-4573-90de-0183fa4024a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:15:58.646062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:58.646045 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p42rd\" (UniqueName: \"kubernetes.io/projected/88837fc6-f0b1-4573-90de-0183fa4024a3-kube-api-access-p42rd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:58.646115 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:58.646067 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88837fc6-f0b1-4573-90de-0183fa4024a3-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:58.647638 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:58.647611 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88837fc6-f0b1-4573-90de-0183fa4024a3-maas-api-xks-ca" pod "88837fc6-f0b1-4573-90de-0183fa4024a3" (UID: "88837fc6-f0b1-4573-90de-0183fa4024a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:15:58.709842 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:58.709805 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:58.747114 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:58.747084 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88837fc6-f0b1-4573-90de-0183fa4024a3-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:15:59.170122 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.170086 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:59.198958 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.198921 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6"] Oct 30 20:15:59.200054 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.200022 5091 cpu_manager.go:414] "containerMap: removing container" podUID="88837fc6-f0b1-4573-90de-0183fa4024a3" containerName="revoke-keys" Oct 30 20:15:59.200054 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.200045 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88837fc6-f0b1-4573-90de-0183fa4024a3" containerName="revoke-keys" Oct 30 20:15:59.200235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.200201 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="88837fc6-f0b1-4573-90de-0183fa4024a3" containerName="revoke-keys" Oct 30 20:15:59.206756 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.206734 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" Oct 30 20:15:59.213924 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.213900 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c257e7\"" Oct 30 20:15:59.216270 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.216240 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6"] Oct 30 20:15:59.218221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.218202 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c257e7-dockercfg-hsmh4\"" Oct 30 20:15:59.260050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.260011 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6gmr\" (UniqueName: \"kubernetes.io/projected/5776e63e-a778-4589-9e2a-10548456ff14-kube-api-access-p6gmr\") pod \"payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6\" (UID: \"5776e63e-a778-4589-9e2a-10548456ff14\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" Oct 30 20:15:59.260258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.260106 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5776e63e-a778-4589-9e2a-10548456ff14-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6\" (UID: \"5776e63e-a778-4589-9e2a-10548456ff14\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" Oct 30 20:15:59.296587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.296552 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" Oct 30 20:15:59.296587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.296577 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9" event={"ID":"88837fc6-f0b1-4573-90de-0183fa4024a3","Type":"ContainerDied","Data":"7722c24917325e0052a560228ff009f5ebd41c3cef0e4f1f00ee9f3116137f02"} Oct 30 20:15:59.296839 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.296631 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7722c24917325e0052a560228ff009f5ebd41c3cef0e4f1f00ee9f3116137f02" Oct 30 20:15:59.366173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.364550 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6gmr\" (UniqueName: \"kubernetes.io/projected/5776e63e-a778-4589-9e2a-10548456ff14-kube-api-access-p6gmr\") pod \"payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6\" (UID: \"5776e63e-a778-4589-9e2a-10548456ff14\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" Oct 30 20:15:59.382239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.380240 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5776e63e-a778-4589-9e2a-10548456ff14-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6\" (UID: \"5776e63e-a778-4589-9e2a-10548456ff14\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" Oct 30 20:15:59.382239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.381382 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5776e63e-a778-4589-9e2a-10548456ff14-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6\" (UID: \"5776e63e-a778-4589-9e2a-10548456ff14\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" Oct 30 20:15:59.386501 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.386458 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:15:59.444260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.444169 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6gmr\" (UniqueName: \"kubernetes.io/projected/5776e63e-a778-4589-9e2a-10548456ff14-kube-api-access-p6gmr\") pod \"payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6\" (UID: \"5776e63e-a778-4589-9e2a-10548456ff14\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" Oct 30 20:15:59.527833 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.527537 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" Oct 30 20:15:59.957185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:15:59.956253 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6"] Oct 30 20:15:59.957185 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:15:59.956420 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5776e63e_a778_4589_9e2a_10548456ff14.slice/crio-34da5e0512c93d4b0c53d8736a5f76e654c1e7f0872a1f2a282f099c69c6b9a9 WatchSource:0}: Error finding container 34da5e0512c93d4b0c53d8736a5f76e654c1e7f0872a1f2a282f099c69c6b9a9: Status 404 returned error can't find the container with id 34da5e0512c93d4b0c53d8736a5f76e654c1e7f0872a1f2a282f099c69c6b9a9 Oct 30 20:16:00.029682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:00.020558 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:00.308764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:00.308670 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" event={"ID":"5776e63e-a778-4589-9e2a-10548456ff14","Type":"ContainerStarted","Data":"e38c2e5f865210eb2106a39e84b987795b89e2eabb1499bd2ed5ba9c3cd28595"} Oct 30 20:16:00.308907 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:00.308884 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" Oct 30 20:16:00.308947 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:00.308918 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" event={"ID":"5776e63e-a778-4589-9e2a-10548456ff14","Type":"ContainerStarted","Data":"34da5e0512c93d4b0c53d8736a5f76e654c1e7f0872a1f2a282f099c69c6b9a9"} Oct 30 20:16:00.335584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:00.335534 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" podStartSLOduration=1.335509917 podStartE2EDuration="1.335509917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:15:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:00.331800383 +0000 UTC m=+2680.027008341" watchObservedRunningTime="2026-10-30 20:16:00.335509917 +0000 UTC m=+2680.030717875" Oct 30 20:16:00.998185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:00.996315 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:01.755253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:01.755215 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:01.907756 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:01.907719 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:02.164861 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.164766 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:02.536680 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.536650 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm"] Oct 30 20:16:02.542066 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.542040 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" Oct 30 20:16:02.555208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.555178 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:16:02.557645 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.557619 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:16:02.558057 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.558031 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm"] Oct 30 20:16:02.705457 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.705424 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pd2b\" (UniqueName: \"kubernetes.io/projected/b40287fc-d491-46a7-848f-51bdf16806ba-kube-api-access-7pd2b\") pod \"maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm\" (UID: \"b40287fc-d491-46a7-848f-51bdf16806ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" Oct 30 20:16:02.705682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.705657 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40287fc-d491-46a7-848f-51bdf16806ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm\" (UID: \"b40287fc-d491-46a7-848f-51bdf16806ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" Oct 30 20:16:02.705748 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.705707 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40287fc-d491-46a7-848f-51bdf16806ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm\" (UID: \"b40287fc-d491-46a7-848f-51bdf16806ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" Oct 30 20:16:02.806890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.806812 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pd2b\" (UniqueName: \"kubernetes.io/projected/b40287fc-d491-46a7-848f-51bdf16806ba-kube-api-access-7pd2b\") pod \"maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm\" (UID: \"b40287fc-d491-46a7-848f-51bdf16806ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" Oct 30 20:16:02.807069 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.807051 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40287fc-d491-46a7-848f-51bdf16806ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm\" (UID: \"b40287fc-d491-46a7-848f-51bdf16806ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" Oct 30 20:16:02.807165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.807110 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40287fc-d491-46a7-848f-51bdf16806ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm\" (UID: \"b40287fc-d491-46a7-848f-51bdf16806ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" Oct 30 20:16:02.807764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.807740 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40287fc-d491-46a7-848f-51bdf16806ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm\" (UID: \"b40287fc-d491-46a7-848f-51bdf16806ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" Oct 30 20:16:02.809453 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.809425 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40287fc-d491-46a7-848f-51bdf16806ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm\" (UID: \"b40287fc-d491-46a7-848f-51bdf16806ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" Oct 30 20:16:02.820977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.820956 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pd2b\" (UniqueName: \"kubernetes.io/projected/b40287fc-d491-46a7-848f-51bdf16806ba-kube-api-access-7pd2b\") pod \"maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm\" (UID: \"b40287fc-d491-46a7-848f-51bdf16806ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" Oct 30 20:16:02.862268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:02.862243 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" Oct 30 20:16:03.063216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:03.063111 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:03.150291 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:03.150259 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm"] Oct 30 20:16:03.151246 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:16:03.151210 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40287fc_d491_46a7_848f_51bdf16806ba.slice/crio-c6b8a0442b1a2237d3450f3bda632d072846b907d618f8ec26d46290ca7f8ad1 WatchSource:0}: Error finding container c6b8a0442b1a2237d3450f3bda632d072846b907d618f8ec26d46290ca7f8ad1: Status 404 returned error can't find the container with id c6b8a0442b1a2237d3450f3bda632d072846b907d618f8ec26d46290ca7f8ad1 Oct 30 20:16:03.348536 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:03.348314 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" event={"ID":"b40287fc-d491-46a7-848f-51bdf16806ba","Type":"ContainerStarted","Data":"c6b8a0442b1a2237d3450f3bda632d072846b907d618f8ec26d46290ca7f8ad1"} Oct 30 20:16:04.280172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:04.279374 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:04.361578 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:04.361483 5091 generic.go:356] "Generic (PLEG): container finished" podID="b40287fc-d491-46a7-848f-51bdf16806ba" containerID="65ef639f736dfb39823d8fe6dd72ebfe8e9afbd4bb4c8e26eba904e62988cb33" exitCode=0 Oct 30 20:16:04.361578 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:04.361544 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" event={"ID":"b40287fc-d491-46a7-848f-51bdf16806ba","Type":"ContainerDied","Data":"65ef639f736dfb39823d8fe6dd72ebfe8e9afbd4bb4c8e26eba904e62988cb33"} Oct 30 20:16:04.685540 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:04.685442 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:04.746159 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:04.746101 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8a41b6_c1fd_4d9b_8f3d_c0a93eb8434c.slice/crio-5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice/crio-1ae08a3ca619d3c72e4eecfe6172f0a90cbfe89251b05e4988849b12b55b1843\": RecentStats: unable to find data in memory cache]" Oct 30 20:16:04.893170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:04.882439 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:05.268172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:05.266184 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:05.553480 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:05.553449 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" Oct 30 20:16:05.633273 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:05.633235 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40287fc-d491-46a7-848f-51bdf16806ba-kube-api-access-7pd2b\" (UniqueName: \"kubernetes.io/projected/b40287fc-d491-46a7-848f-51bdf16806ba-kube-api-access-7pd2b\") pod \"b40287fc-d491-46a7-848f-51bdf16806ba\" (UID: \"b40287fc-d491-46a7-848f-51bdf16806ba\") " Oct 30 20:16:05.634023 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:05.633994 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b40287fc-d491-46a7-848f-51bdf16806ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40287fc-d491-46a7-848f-51bdf16806ba-maas-api-xks-ca\") pod \"b40287fc-d491-46a7-848f-51bdf16806ba\" (UID: \"b40287fc-d491-46a7-848f-51bdf16806ba\") " Oct 30 20:16:05.634205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:05.634041 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40287fc-d491-46a7-848f-51bdf16806ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40287fc-d491-46a7-848f-51bdf16806ba-maas-api-openshift-ca\") pod \"b40287fc-d491-46a7-848f-51bdf16806ba\" (UID: \"b40287fc-d491-46a7-848f-51bdf16806ba\") " Oct 30 20:16:05.634528 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:05.634499 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40287fc-d491-46a7-848f-51bdf16806ba-maas-api-openshift-ca" pod "b40287fc-d491-46a7-848f-51bdf16806ba" (UID: "b40287fc-d491-46a7-848f-51bdf16806ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:05.634983 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:05.634941 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b40287fc-d491-46a7-848f-51bdf16806ba-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:05.635696 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:05.635674 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40287fc-d491-46a7-848f-51bdf16806ba-kube-api-access-7pd2b" pod "b40287fc-d491-46a7-848f-51bdf16806ba" (UID: "b40287fc-d491-46a7-848f-51bdf16806ba"). InnerVolumeSpecName "kube-api-access-7pd2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:05.636083 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:05.636062 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40287fc-d491-46a7-848f-51bdf16806ba-maas-api-xks-ca" pod "b40287fc-d491-46a7-848f-51bdf16806ba" (UID: "b40287fc-d491-46a7-848f-51bdf16806ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:16:05.736865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:05.736837 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pd2b\" (UniqueName: \"kubernetes.io/projected/b40287fc-d491-46a7-848f-51bdf16806ba-kube-api-access-7pd2b\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:05.736865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:05.736865 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b40287fc-d491-46a7-848f-51bdf16806ba-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:06.115186 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:06.109116 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:06.384889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:06.384804 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" Oct 30 20:16:06.384889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:06.384828 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm" event={"ID":"b40287fc-d491-46a7-848f-51bdf16806ba","Type":"ContainerDied","Data":"c6b8a0442b1a2237d3450f3bda632d072846b907d618f8ec26d46290ca7f8ad1"} Oct 30 20:16:06.384889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:06.384878 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6b8a0442b1a2237d3450f3bda632d072846b907d618f8ec26d46290ca7f8ad1" Oct 30 20:16:07.035043 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:07.033932 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:07.481617 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:07.481522 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice/crio-1ae08a3ca619d3c72e4eecfe6172f0a90cbfe89251b05e4988849b12b55b1843\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8a41b6_c1fd_4d9b_8f3d_c0a93eb8434c.slice/crio-5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice\": RecentStats: unable to find data in memory cache]" Oct 30 20:16:07.994165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:07.994003 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:08.968198 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:08.964087 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:09.656571 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:09.656526 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:09.869043 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:09.869000 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:09.994257 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:09.994164 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw"] Oct 30 20:16:10.017568 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:10.015471 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c8241-fvlxw"] Oct 30 20:16:10.029071 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:10.029017 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:10.205371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:10.205127 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:10.425706 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:10.425656 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:10.442907 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:10.442879 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70b8ac16-887e-4b69-b088-5a71eac646ee" path="/var/lib/kubelet/pods/70b8ac16-887e-4b69-b088-5a71eac646ee/volumes" Oct 30 20:16:11.102404 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:11.102365 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:11.320352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:11.320322 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" Oct 30 20:16:11.921719 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:11.921683 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice/crio-1ae08a3ca619d3c72e4eecfe6172f0a90cbfe89251b05e4988849b12b55b1843\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8a41b6_c1fd_4d9b_8f3d_c0a93eb8434c.slice/crio-5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397\": RecentStats: unable to find data in memory cache]" Oct 30 20:16:11.921924 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:11.921689 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice/crio-1ae08a3ca619d3c72e4eecfe6172f0a90cbfe89251b05e4988849b12b55b1843\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8a41b6_c1fd_4d9b_8f3d_c0a93eb8434c.slice/crio-5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397\": RecentStats: unable to find data in memory cache]" Oct 30 20:16:12.048180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:12.041194 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:12.742547 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:12.742509 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t"] Oct 30 20:16:12.864927 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:12.864887 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:13.047902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:13.047865 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:13.208378 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:13.208342 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:14.099052 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:14.099013 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:14.819126 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:14.819091 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:15.203118 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:15.203029 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:15.986871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:15.986826 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:16.205383 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:16.205339 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:16:16.228294 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:16.228260 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:16.793176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:16.792259 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:17.758476 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:17.758232 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" podUID="8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8" containerName="payload-processing" containerID="cri-o://d939db0cf44c4331b56e4046a78ab5b8905ccf9cdc9ed0403b3b71339fdf8045" gracePeriod=25 Oct 30 20:16:18.400212 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:18.400175 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:18.866329 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:18.862562 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice/crio-1ae08a3ca619d3c72e4eecfe6172f0a90cbfe89251b05e4988849b12b55b1843\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8a41b6_c1fd_4d9b_8f3d_c0a93eb8434c.slice/crio-5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397\": RecentStats: unable to find data in memory cache]" Oct 30 20:16:18.955358 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:18.955321 5091 generic.go:356] "Generic (PLEG): container finished" podID="8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8" containerID="d939db0cf44c4331b56e4046a78ab5b8905ccf9cdc9ed0403b3b71339fdf8045" exitCode=0 Oct 30 20:16:18.955881 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:18.955849 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" event={"ID":"8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8","Type":"ContainerDied","Data":"d939db0cf44c4331b56e4046a78ab5b8905ccf9cdc9ed0403b3b71339fdf8045"} Oct 30 20:16:18.964304 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:18.964280 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" Oct 30 20:16:19.050248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:19.050204 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8-kube-api-access-v7p4t\" (UniqueName: \"kubernetes.io/projected/8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8-kube-api-access-v7p4t\") pod \"8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8\" (UID: \"8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8\") " Oct 30 20:16:19.051494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:19.051465 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8-plugins-config-volume\") pod \"8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8\" (UID: \"8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8\") " Oct 30 20:16:19.070170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:19.067724 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8-plugins-config-volume" pod "8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8" (UID: "8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:19.072747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:19.072713 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:19.072918 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:19.072742 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8-kube-api-access-v7p4t" pod "8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8" (UID: "8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8"). InnerVolumeSpecName "kube-api-access-v7p4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:19.174575 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:19.174498 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7p4t\" (UniqueName: \"kubernetes.io/projected/8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8-kube-api-access-v7p4t\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:19.580318 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:19.580264 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:19.665649 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:19.665607 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c8a41b6_c1fd_4d9b_8f3d_c0a93eb8434c.slice/crio-5c34fcdef7aa187e953393469427e19f329907aba8d6f369d9a71cdae7db6397\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d9f0fd_4bb0_4773_8333_92f168e683c3.slice/crio-1ae08a3ca619d3c72e4eecfe6172f0a90cbfe89251b05e4988849b12b55b1843\": RecentStats: unable to find data in memory cache]" Oct 30 20:16:19.966966 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:19.966888 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" Oct 30 20:16:19.966966 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:19.966912 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t" event={"ID":"8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8","Type":"ContainerDied","Data":"5df2cd3b813371a3d8a401235adc4a60ba31708341187f8627630b0633aa706d"} Oct 30 20:16:19.967472 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:19.966972 5091 scope.go:122] "RemoveContainer" containerID="d939db0cf44c4331b56e4046a78ab5b8905ccf9cdc9ed0403b3b71339fdf8045" Oct 30 20:16:19.998451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:19.998424 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t"] Oct 30 20:16:20.002566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:20.002543 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-2a2286-866c7dfb84-ktc6t"] Oct 30 20:16:20.391960 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:20.391926 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:20.445504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:20.445471 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8" path="/var/lib/kubelet/pods/8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8/volumes" Oct 30 20:16:20.638810 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:20.638737 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:20.814339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:20.814303 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9"] Oct 30 20:16:20.830607 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:20.830574 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c9bce8-phjv9"] Oct 30 20:16:20.844119 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:20.844072 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:21.074517 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.074421 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:21.304619 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.304571 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp"] Oct 30 20:16:21.304875 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.304842 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" podUID="18e11004-d729-4eec-8626-28eba4b01e90" containerName="istio-proxy" containerID="cri-o://f2c891affe98aa1c7fdad0779c85e054d7b9c3c4044839d33366360f7a45190e" gracePeriod=30 Oct 30 20:16:21.315477 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.315437 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:21.499223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.496573 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:21.732239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.732208 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 20:16:21.732811 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.732784 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 20:16:21.873275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.873160 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj"] Oct 30 20:16:21.874131 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.874109 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b40287fc-d491-46a7-848f-51bdf16806ba" containerName="revoke-keys" Oct 30 20:16:21.874240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.874133 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40287fc-d491-46a7-848f-51bdf16806ba" containerName="revoke-keys" Oct 30 20:16:21.874240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.874193 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8" containerName="payload-processing" Oct 30 20:16:21.874240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.874200 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8" containerName="payload-processing" Oct 30 20:16:21.874349 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.874327 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40287fc-d491-46a7-848f-51bdf16806ba" containerName="revoke-keys" Oct 30 20:16:21.874390 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.874349 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1d2799-2a52-4d5e-9ddc-a4e84b4e26e8" containerName="payload-processing" Oct 30 20:16:21.879580 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.879559 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" Oct 30 20:16:21.889407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.889381 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:16:21.889694 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.889668 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:16:21.891998 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:21.891441 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj"] Oct 30 20:16:22.014010 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.002748 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:22.053135 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.053057 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8c85f0-5a0d-4306-a911-5d872a620026-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-969878-sllwj\" (UID: \"4c8c85f0-5a0d-4306-a911-5d872a620026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" Oct 30 20:16:22.053135 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.053111 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2pnm\" (UniqueName: \"kubernetes.io/projected/4c8c85f0-5a0d-4306-a911-5d872a620026-kube-api-access-g2pnm\") pod \"maas-api-revoke-keys-e2e-extmod-b-969878-sllwj\" (UID: \"4c8c85f0-5a0d-4306-a911-5d872a620026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" Oct 30 20:16:22.053500 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.053471 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8c85f0-5a0d-4306-a911-5d872a620026-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-969878-sllwj\" (UID: \"4c8c85f0-5a0d-4306-a911-5d872a620026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" Oct 30 20:16:22.155385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.155299 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8c85f0-5a0d-4306-a911-5d872a620026-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-969878-sllwj\" (UID: \"4c8c85f0-5a0d-4306-a911-5d872a620026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" Oct 30 20:16:22.155808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.155386 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8c85f0-5a0d-4306-a911-5d872a620026-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-969878-sllwj\" (UID: \"4c8c85f0-5a0d-4306-a911-5d872a620026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" Oct 30 20:16:22.155808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.155409 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2pnm\" (UniqueName: \"kubernetes.io/projected/4c8c85f0-5a0d-4306-a911-5d872a620026-kube-api-access-g2pnm\") pod \"maas-api-revoke-keys-e2e-extmod-b-969878-sllwj\" (UID: \"4c8c85f0-5a0d-4306-a911-5d872a620026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" Oct 30 20:16:22.156078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.156056 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8c85f0-5a0d-4306-a911-5d872a620026-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-969878-sllwj\" (UID: \"4c8c85f0-5a0d-4306-a911-5d872a620026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" Oct 30 20:16:22.157827 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.157802 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8c85f0-5a0d-4306-a911-5d872a620026-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-969878-sllwj\" (UID: \"4c8c85f0-5a0d-4306-a911-5d872a620026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" Oct 30 20:16:22.174215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.174118 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2pnm\" (UniqueName: \"kubernetes.io/projected/4c8c85f0-5a0d-4306-a911-5d872a620026-kube-api-access-g2pnm\") pod \"maas-api-revoke-keys-e2e-extmod-b-969878-sllwj\" (UID: \"4c8c85f0-5a0d-4306-a911-5d872a620026\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" Oct 30 20:16:22.199806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.199776 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" Oct 30 20:16:22.444606 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.444508 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88837fc6-f0b1-4573-90de-0183fa4024a3" path="/var/lib/kubelet/pods/88837fc6-f0b1-4573-90de-0183fa4024a3/volumes" Oct 30 20:16:22.495904 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:16:22.495866 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c8c85f0_5a0d_4306_a911_5d872a620026.slice/crio-a2502faf5a77a32b79b155670c35886192c486c9405625fef31389c9c8cd51b7 WatchSource:0}: Error finding container a2502faf5a77a32b79b155670c35886192c486c9405625fef31389c9c8cd51b7: Status 404 returned error can't find the container with id a2502faf5a77a32b79b155670c35886192c486c9405625fef31389c9c8cd51b7 Oct 30 20:16:22.496408 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.496384 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj"] Oct 30 20:16:22.531475 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.531451 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn"] Oct 30 20:16:22.537539 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.537517 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" Oct 30 20:16:22.555189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.546135 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn"] Oct 30 20:16:22.662844 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.662809 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4t6sd\" (UniqueName: \"kubernetes.io/projected/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-kube-api-access-4t6sd\") pod \"maas-api-revoke-sub-8d9e4d9d-zdtjn\" (UID: \"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" Oct 30 20:16:22.663030 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.662966 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d9e4d9d-zdtjn\" (UID: \"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" Oct 30 20:16:22.663191 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.663136 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d9e4d9d-zdtjn\" (UID: \"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" Oct 30 20:16:22.764441 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.764407 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d9e4d9d-zdtjn\" (UID: \"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" Oct 30 20:16:22.764637 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.764542 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4t6sd\" (UniqueName: \"kubernetes.io/projected/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-kube-api-access-4t6sd\") pod \"maas-api-revoke-sub-8d9e4d9d-zdtjn\" (UID: \"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" Oct 30 20:16:22.764709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.764653 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d9e4d9d-zdtjn\" (UID: \"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" Oct 30 20:16:22.765161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.765126 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8d9e4d9d-zdtjn\" (UID: \"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" Oct 30 20:16:22.767009 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.766982 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8d9e4d9d-zdtjn\" (UID: \"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" Oct 30 20:16:22.781793 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.781764 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4t6sd\" (UniqueName: \"kubernetes.io/projected/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-kube-api-access-4t6sd\") pod \"maas-api-revoke-sub-8d9e4d9d-zdtjn\" (UID: \"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" Oct 30 20:16:22.863928 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:22.863896 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" Oct 30 20:16:23.038717 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:23.038622 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" event={"ID":"4c8c85f0-5a0d-4306-a911-5d872a620026","Type":"ContainerStarted","Data":"cc9181cb9d8515085f21a54ce5fe846609ba287c01c04aaa2576e114756c8dc4"} Oct 30 20:16:23.038717 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:23.038679 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" event={"ID":"4c8c85f0-5a0d-4306-a911-5d872a620026","Type":"ContainerStarted","Data":"a2502faf5a77a32b79b155670c35886192c486c9405625fef31389c9c8cd51b7"} Oct 30 20:16:23.041999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:23.041970 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:23.155746 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:23.155694 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" podStartSLOduration=2.155677348 podStartE2EDuration="2.155677348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:23.104039338 +0000 UTC m=+2702.799247297" watchObservedRunningTime="2026-10-30 20:16:23.155677348 +0000 UTC m=+2702.850885305" Oct 30 20:16:23.162960 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:23.161597 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn"] Oct 30 20:16:24.005173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:24.004901 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:24.066273 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:24.066233 5091 generic.go:356] "Generic (PLEG): container finished" podID="4c8c85f0-5a0d-4306-a911-5d872a620026" containerID="cc9181cb9d8515085f21a54ce5fe846609ba287c01c04aaa2576e114756c8dc4" exitCode=0 Oct 30 20:16:24.066427 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:24.066308 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" event={"ID":"4c8c85f0-5a0d-4306-a911-5d872a620026","Type":"ContainerDied","Data":"cc9181cb9d8515085f21a54ce5fe846609ba287c01c04aaa2576e114756c8dc4"} Oct 30 20:16:24.068105 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:24.068073 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" event={"ID":"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d","Type":"ContainerStarted","Data":"65923c0ed3f4a05f36d31e7b80f60b41cf1124cc3445b79c3531ac312b5da3a2"} Oct 30 20:16:24.068253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:24.068117 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" event={"ID":"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d","Type":"ContainerStarted","Data":"864a88edd574c122991018daf11a9899420ea6bbff3255e7bd0a33b187ffdcc0"} Oct 30 20:16:24.116195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:24.115105 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" podStartSLOduration=2.115087627 podStartE2EDuration="2.115087627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:24.114596584 +0000 UTC m=+2703.809804542" watchObservedRunningTime="2026-10-30 20:16:24.115087627 +0000 UTC m=+2703.810295587" Oct 30 20:16:24.164814 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:24.164773 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:24.774405 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:24.774363 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:16:24.774588 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:24.774436 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" podUID="18e11004-d729-4eec-8626-28eba4b01e90" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:16:25.077870 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:25.077782 5091 generic.go:356] "Generic (PLEG): container finished" podID="2f8bc28a-95aa-4ceb-97f9-91a2f881d50d" containerID="65923c0ed3f4a05f36d31e7b80f60b41cf1124cc3445b79c3531ac312b5da3a2" exitCode=0 Oct 30 20:16:25.078027 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:25.077864 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" event={"ID":"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d","Type":"ContainerDied","Data":"65923c0ed3f4a05f36d31e7b80f60b41cf1124cc3445b79c3531ac312b5da3a2"} Oct 30 20:16:25.209176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:25.209045 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:25.352858 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:25.352796 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" Oct 30 20:16:25.427273 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:25.427228 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c8c85f0-5a0d-4306-a911-5d872a620026-kube-api-access-g2pnm\" (UniqueName: \"kubernetes.io/projected/4c8c85f0-5a0d-4306-a911-5d872a620026-kube-api-access-g2pnm\") pod \"4c8c85f0-5a0d-4306-a911-5d872a620026\" (UID: \"4c8c85f0-5a0d-4306-a911-5d872a620026\") " Oct 30 20:16:25.427474 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:25.427343 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c8c85f0-5a0d-4306-a911-5d872a620026-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8c85f0-5a0d-4306-a911-5d872a620026-maas-api-openshift-ca\") pod \"4c8c85f0-5a0d-4306-a911-5d872a620026\" (UID: \"4c8c85f0-5a0d-4306-a911-5d872a620026\") " Oct 30 20:16:25.427767 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:25.427735 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c8c85f0-5a0d-4306-a911-5d872a620026-maas-api-openshift-ca" pod "4c8c85f0-5a0d-4306-a911-5d872a620026" (UID: "4c8c85f0-5a0d-4306-a911-5d872a620026"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:25.428321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:25.428296 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c8c85f0-5a0d-4306-a911-5d872a620026-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:25.430272 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:25.430242 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c8c85f0-5a0d-4306-a911-5d872a620026-kube-api-access-g2pnm" pod "4c8c85f0-5a0d-4306-a911-5d872a620026" (UID: "4c8c85f0-5a0d-4306-a911-5d872a620026"). InnerVolumeSpecName "kube-api-access-g2pnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:25.529435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:25.529391 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c8c85f0-5a0d-4306-a911-5d872a620026-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8c85f0-5a0d-4306-a911-5d872a620026-maas-api-xks-ca\") pod \"4c8c85f0-5a0d-4306-a911-5d872a620026\" (UID: \"4c8c85f0-5a0d-4306-a911-5d872a620026\") " Oct 30 20:16:25.530362 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:25.530298 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2pnm\" (UniqueName: \"kubernetes.io/projected/4c8c85f0-5a0d-4306-a911-5d872a620026-kube-api-access-g2pnm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:25.531785 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:25.531757 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c8c85f0-5a0d-4306-a911-5d872a620026-maas-api-xks-ca" pod "4c8c85f0-5a0d-4306-a911-5d872a620026" (UID: "4c8c85f0-5a0d-4306-a911-5d872a620026"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:16:25.631936 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:25.631849 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c8c85f0-5a0d-4306-a911-5d872a620026-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:26.087984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.087950 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" Oct 30 20:16:26.088162 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.087974 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj" event={"ID":"4c8c85f0-5a0d-4306-a911-5d872a620026","Type":"ContainerDied","Data":"a2502faf5a77a32b79b155670c35886192c486c9405625fef31389c9c8cd51b7"} Oct 30 20:16:26.088162 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.088049 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2502faf5a77a32b79b155670c35886192c486c9405625fef31389c9c8cd51b7" Oct 30 20:16:26.304025 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.303977 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:26.331207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.329317 5091 scope.go:122] "RemoveContainer" containerID="cf7dbbcce71b02fc1961cb32691c13c9f94e8dbed35688aa87d48db10a3a5538" Oct 30 20:16:26.422483 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.422458 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" Oct 30 20:16:26.568753 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.568717 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-kube-api-access-4t6sd\" (UniqueName: \"kubernetes.io/projected/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-kube-api-access-4t6sd\") pod \"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d\" (UID: \"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d\") " Oct 30 20:16:26.568920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.568804 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-maas-api-openshift-ca\") pod \"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d\" (UID: \"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d\") " Oct 30 20:16:26.568920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.568890 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-maas-api-xks-ca\") pod \"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d\" (UID: \"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d\") " Oct 30 20:16:26.569707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.569232 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-maas-api-openshift-ca" pod "2f8bc28a-95aa-4ceb-97f9-91a2f881d50d" (UID: "2f8bc28a-95aa-4ceb-97f9-91a2f881d50d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:26.569707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.569661 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:26.571269 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.571245 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-maas-api-xks-ca" pod "2f8bc28a-95aa-4ceb-97f9-91a2f881d50d" (UID: "2f8bc28a-95aa-4ceb-97f9-91a2f881d50d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:16:26.572393 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.572370 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-kube-api-access-4t6sd" pod "2f8bc28a-95aa-4ceb-97f9-91a2f881d50d" (UID: "2f8bc28a-95aa-4ceb-97f9-91a2f881d50d"). InnerVolumeSpecName "kube-api-access-4t6sd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:26.632635 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.632605 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:16:26.670536 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.670510 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e11004-d729-4eec-8626-28eba4b01e90-kube-api-access-tlxp8\" (UniqueName: \"kubernetes.io/projected/18e11004-d729-4eec-8626-28eba4b01e90-kube-api-access-tlxp8\") pod \"18e11004-d729-4eec-8626-28eba4b01e90\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " Oct 30 20:16:26.670662 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.670548 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-istio-data\") pod \"18e11004-d729-4eec-8626-28eba4b01e90\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " Oct 30 20:16:26.670662 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.670632 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e11004-d729-4eec-8626-28eba4b01e90-istio-token\" (UniqueName: \"kubernetes.io/projected/18e11004-d729-4eec-8626-28eba4b01e90-istio-token\") pod \"18e11004-d729-4eec-8626-28eba4b01e90\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " Oct 30 20:16:26.670800 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.670711 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-workload-socket\") pod \"18e11004-d729-4eec-8626-28eba4b01e90\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " Oct 30 20:16:26.670870 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.670807 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e11004-d729-4eec-8626-28eba4b01e90-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e11004-d729-4eec-8626-28eba4b01e90-istiod-ca-cert\") pod \"18e11004-d729-4eec-8626-28eba4b01e90\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " Oct 30 20:16:26.670870 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.670839 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-credential-socket\") pod \"18e11004-d729-4eec-8626-28eba4b01e90\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " Oct 30 20:16:26.670979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.670899 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-istio-data" pod "18e11004-d729-4eec-8626-28eba4b01e90" (UID: "18e11004-d729-4eec-8626-28eba4b01e90"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:26.670979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.670962 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18e11004-d729-4eec-8626-28eba4b01e90-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e11004-d729-4eec-8626-28eba4b01e90-istio-podinfo\") pod \"18e11004-d729-4eec-8626-28eba4b01e90\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " Oct 30 20:16:26.671101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.671057 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-workload-certs\") pod \"18e11004-d729-4eec-8626-28eba4b01e90\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " Oct 30 20:16:26.671101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.671074 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-workload-socket" pod "18e11004-d729-4eec-8626-28eba4b01e90" (UID: "18e11004-d729-4eec-8626-28eba4b01e90"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:26.671240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.671105 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-istio-envoy\") pod \"18e11004-d729-4eec-8626-28eba4b01e90\" (UID: \"18e11004-d729-4eec-8626-28eba4b01e90\") " Oct 30 20:16:26.671240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.671211 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-credential-socket" pod "18e11004-d729-4eec-8626-28eba4b01e90" (UID: "18e11004-d729-4eec-8626-28eba4b01e90"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:26.671640 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.671612 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-workload-certs" pod "18e11004-d729-4eec-8626-28eba4b01e90" (UID: "18e11004-d729-4eec-8626-28eba4b01e90"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:26.671859 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.671832 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e11004-d729-4eec-8626-28eba4b01e90-istiod-ca-cert" pod "18e11004-d729-4eec-8626-28eba4b01e90" (UID: "18e11004-d729-4eec-8626-28eba4b01e90"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:26.671991 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.671971 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4t6sd\" (UniqueName: \"kubernetes.io/projected/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-kube-api-access-4t6sd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:26.672063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.671996 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:26.672063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.672011 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:26.672063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.672026 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:26.672063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.672041 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:26.672063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.672054 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e11004-d729-4eec-8626-28eba4b01e90-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:26.672379 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.672068 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:26.673201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.673177 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e11004-d729-4eec-8626-28eba4b01e90-istio-token" pod "18e11004-d729-4eec-8626-28eba4b01e90" (UID: "18e11004-d729-4eec-8626-28eba4b01e90"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:26.673318 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.673242 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e11004-d729-4eec-8626-28eba4b01e90-kube-api-access-tlxp8" pod "18e11004-d729-4eec-8626-28eba4b01e90" (UID: "18e11004-d729-4eec-8626-28eba4b01e90"). InnerVolumeSpecName "kube-api-access-tlxp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:26.673661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.673632 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18e11004-d729-4eec-8626-28eba4b01e90-istio-podinfo" pod "18e11004-d729-4eec-8626-28eba4b01e90" (UID: "18e11004-d729-4eec-8626-28eba4b01e90"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:16:26.673959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.673919 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-istio-envoy" pod "18e11004-d729-4eec-8626-28eba4b01e90" (UID: "18e11004-d729-4eec-8626-28eba4b01e90"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:26.774086 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.774049 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e11004-d729-4eec-8626-28eba4b01e90-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:26.774086 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.774083 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e11004-d729-4eec-8626-28eba4b01e90-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:26.774086 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.774093 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e11004-d729-4eec-8626-28eba4b01e90-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:26.774372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:26.774103 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlxp8\" (UniqueName: \"kubernetes.io/projected/18e11004-d729-4eec-8626-28eba4b01e90-kube-api-access-tlxp8\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:27.101974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.101930 5091 generic.go:356] "Generic (PLEG): container finished" podID="18e11004-d729-4eec-8626-28eba4b01e90" containerID="f2c891affe98aa1c7fdad0779c85e054d7b9c3c4044839d33366360f7a45190e" exitCode=0 Oct 30 20:16:27.103319 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.101994 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" Oct 30 20:16:27.103319 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.102006 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" event={"ID":"18e11004-d729-4eec-8626-28eba4b01e90","Type":"ContainerDied","Data":"f2c891affe98aa1c7fdad0779c85e054d7b9c3c4044839d33366360f7a45190e"} Oct 30 20:16:27.103319 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.102072 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp" event={"ID":"18e11004-d729-4eec-8626-28eba4b01e90","Type":"ContainerDied","Data":"fd51b63d7390c04a6480d04510e8bc8e50a2ea0d7a5308a649c6745d08f321d6"} Oct 30 20:16:27.103319 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.102107 5091 scope.go:122] "RemoveContainer" containerID="f2c891affe98aa1c7fdad0779c85e054d7b9c3c4044839d33366360f7a45190e" Oct 30 20:16:27.105484 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.105435 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" Oct 30 20:16:27.105484 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.105466 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn" event={"ID":"2f8bc28a-95aa-4ceb-97f9-91a2f881d50d","Type":"ContainerDied","Data":"864a88edd574c122991018daf11a9899420ea6bbff3255e7bd0a33b187ffdcc0"} Oct 30 20:16:27.105725 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.105507 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="864a88edd574c122991018daf11a9899420ea6bbff3255e7bd0a33b187ffdcc0" Oct 30 20:16:27.122130 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.122109 5091 scope.go:122] "RemoveContainer" containerID="b832152b92057244bcc5e2d9da58874ecd2cee48a3954953a6a67e39f106d133" Oct 30 20:16:27.137374 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.137353 5091 scope.go:122] "RemoveContainer" containerID="f2c891affe98aa1c7fdad0779c85e054d7b9c3c4044839d33366360f7a45190e" Oct 30 20:16:27.137679 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:27.137656 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c891affe98aa1c7fdad0779c85e054d7b9c3c4044839d33366360f7a45190e\": container with ID starting with f2c891affe98aa1c7fdad0779c85e054d7b9c3c4044839d33366360f7a45190e not found: ID does not exist" containerID="f2c891affe98aa1c7fdad0779c85e054d7b9c3c4044839d33366360f7a45190e" Oct 30 20:16:27.137793 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.137690 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c891affe98aa1c7fdad0779c85e054d7b9c3c4044839d33366360f7a45190e"} err="failed to get container status \"f2c891affe98aa1c7fdad0779c85e054d7b9c3c4044839d33366360f7a45190e\": rpc error: code = NotFound desc = could not find container \"f2c891affe98aa1c7fdad0779c85e054d7b9c3c4044839d33366360f7a45190e\": container with ID starting with f2c891affe98aa1c7fdad0779c85e054d7b9c3c4044839d33366360f7a45190e not found: ID does not exist" Oct 30 20:16:27.137793 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.137714 5091 scope.go:122] "RemoveContainer" containerID="b832152b92057244bcc5e2d9da58874ecd2cee48a3954953a6a67e39f106d133" Oct 30 20:16:27.137953 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:27.137936 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b832152b92057244bcc5e2d9da58874ecd2cee48a3954953a6a67e39f106d133\": container with ID starting with b832152b92057244bcc5e2d9da58874ecd2cee48a3954953a6a67e39f106d133 not found: ID does not exist" containerID="b832152b92057244bcc5e2d9da58874ecd2cee48a3954953a6a67e39f106d133" Oct 30 20:16:27.138019 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.137961 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b832152b92057244bcc5e2d9da58874ecd2cee48a3954953a6a67e39f106d133"} err="failed to get container status \"b832152b92057244bcc5e2d9da58874ecd2cee48a3954953a6a67e39f106d133\": rpc error: code = NotFound desc = could not find container \"b832152b92057244bcc5e2d9da58874ecd2cee48a3954953a6a67e39f106d133\": container with ID starting with b832152b92057244bcc5e2d9da58874ecd2cee48a3954953a6a67e39f106d133 not found: ID does not exist" Oct 30 20:16:27.162627 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.162585 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp"] Oct 30 20:16:27.172030 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.171984 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-8c8241-openshift-default-5c9478b8bf-jnmkp"] Oct 30 20:16:27.294856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.294807 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:27.556172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.551129 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:27.691218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.691167 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm"] Oct 30 20:16:27.701196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.700543 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-2a2286-j24cm"] Oct 30 20:16:27.773170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:27.765160 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:28.276513 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:28.276473 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:28.443782 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:28.443750 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e11004-d729-4eec-8626-28eba4b01e90" path="/var/lib/kubelet/pods/18e11004-d729-4eec-8626-28eba4b01e90/volumes" Oct 30 20:16:28.444306 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:28.444285 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b40287fc-d491-46a7-848f-51bdf16806ba" path="/var/lib/kubelet/pods/b40287fc-d491-46a7-848f-51bdf16806ba/volumes" Oct 30 20:16:28.891169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:28.890711 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:29.724202 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:29.720578 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:29.890038 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:29.890006 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:30.053021 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.052964 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:30.200727 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.200680 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz"] Oct 30 20:16:30.200958 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.200931 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerName="istio-proxy" containerID="cri-o://01e6ec8e203e65d17a3c0b1202e25b0a44ec2b522c011cf7b84a2bd1c7503c29" gracePeriod=30 Oct 30 20:16:30.209266 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.209156 5091 patch_prober.go:29] interesting pod/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:16:30.209266 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.209212 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:16:30.260280 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.260238 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:30.710247 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.710209 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:30.762203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.757368 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml"] Oct 30 20:16:30.762464 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.758655 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2f8bc28a-95aa-4ceb-97f9-91a2f881d50d" containerName="revoke-keys" Oct 30 20:16:30.762587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.762559 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f8bc28a-95aa-4ceb-97f9-91a2f881d50d" containerName="revoke-keys" Oct 30 20:16:30.762759 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.762613 5091 cpu_manager.go:414] "containerMap: removing container" podUID="18e11004-d729-4eec-8626-28eba4b01e90" containerName="istio-proxy" Oct 30 20:16:30.762759 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.762628 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e11004-d729-4eec-8626-28eba4b01e90" containerName="istio-proxy" Oct 30 20:16:30.762759 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.762688 5091 cpu_manager.go:414] "containerMap: removing container" podUID="18e11004-d729-4eec-8626-28eba4b01e90" containerName="istio-proxy" Oct 30 20:16:30.762759 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.762697 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e11004-d729-4eec-8626-28eba4b01e90" containerName="istio-proxy" Oct 30 20:16:30.762759 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.762756 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4c8c85f0-5a0d-4306-a911-5d872a620026" containerName="revoke-keys" Oct 30 20:16:30.763060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.762764 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c8c85f0-5a0d-4306-a911-5d872a620026" containerName="revoke-keys" Oct 30 20:16:30.763060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.763041 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e11004-d729-4eec-8626-28eba4b01e90" containerName="istio-proxy" Oct 30 20:16:30.763208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.763077 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c8c85f0-5a0d-4306-a911-5d872a620026" containerName="revoke-keys" Oct 30 20:16:30.763208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.763097 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f8bc28a-95aa-4ceb-97f9-91a2f881d50d" containerName="revoke-keys" Oct 30 20:16:30.810309 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.810273 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml"] Oct 30 20:16:30.810521 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.810398 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" Oct 30 20:16:30.816667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.816638 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:16:30.816809 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.816736 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:16:30.883493 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.883452 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6192cc65-0103-4b75-b51c-3969262634eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml\" (UID: \"6192cc65-0103-4b75-b51c-3969262634eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" Oct 30 20:16:30.883693 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.883563 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6192cc65-0103-4b75-b51c-3969262634eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml\" (UID: \"6192cc65-0103-4b75-b51c-3969262634eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" Oct 30 20:16:30.883790 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.883628 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzgvx\" (UniqueName: \"kubernetes.io/projected/6192cc65-0103-4b75-b51c-3969262634eb-kube-api-access-pzgvx\") pod \"maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml\" (UID: \"6192cc65-0103-4b75-b51c-3969262634eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" Oct 30 20:16:30.986107 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.986016 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6192cc65-0103-4b75-b51c-3969262634eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml\" (UID: \"6192cc65-0103-4b75-b51c-3969262634eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" Oct 30 20:16:30.986302 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.986138 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6192cc65-0103-4b75-b51c-3969262634eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml\" (UID: \"6192cc65-0103-4b75-b51c-3969262634eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" Oct 30 20:16:30.986302 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.986227 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzgvx\" (UniqueName: \"kubernetes.io/projected/6192cc65-0103-4b75-b51c-3969262634eb-kube-api-access-pzgvx\") pod \"maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml\" (UID: \"6192cc65-0103-4b75-b51c-3969262634eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" Oct 30 20:16:30.986940 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.986856 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6192cc65-0103-4b75-b51c-3969262634eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml\" (UID: \"6192cc65-0103-4b75-b51c-3969262634eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" Oct 30 20:16:30.988831 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:30.988807 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6192cc65-0103-4b75-b51c-3969262634eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml\" (UID: \"6192cc65-0103-4b75-b51c-3969262634eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" Oct 30 20:16:31.001068 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:31.001036 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzgvx\" (UniqueName: \"kubernetes.io/projected/6192cc65-0103-4b75-b51c-3969262634eb-kube-api-access-pzgvx\") pod \"maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml\" (UID: \"6192cc65-0103-4b75-b51c-3969262634eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" Oct 30 20:16:31.127741 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:31.127702 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" Oct 30 20:16:31.417406 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:31.417370 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml"] Oct 30 20:16:31.417796 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:16:31.417767 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6192cc65_0103_4b75_b51c_3969262634eb.slice/crio-05df951be66a0d16ae3be2e471994597ae6e23a26a90dbd36f05344ff7b02304 WatchSource:0}: Error finding container 05df951be66a0d16ae3be2e471994597ae6e23a26a90dbd36f05344ff7b02304: Status 404 returned error can't find the container with id 05df951be66a0d16ae3be2e471994597ae6e23a26a90dbd36f05344ff7b02304 Oct 30 20:16:31.827958 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:31.827922 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:32.149411 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:32.149292 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:32.162978 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:32.155051 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" event={"ID":"6192cc65-0103-4b75-b51c-3969262634eb","Type":"ContainerStarted","Data":"a7a03f85339cec74afb7a2b34c8934d2c51683bf8bdec844f756acf684bc6f42"} Oct 30 20:16:32.162978 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:32.155111 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" event={"ID":"6192cc65-0103-4b75-b51c-3969262634eb","Type":"ContainerStarted","Data":"05df951be66a0d16ae3be2e471994597ae6e23a26a90dbd36f05344ff7b02304"} Oct 30 20:16:32.186139 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:32.186084 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" podStartSLOduration=2.186071029 podStartE2EDuration="2.186071029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:32.178221941 +0000 UTC m=+2711.873429900" watchObservedRunningTime="2026-10-30 20:16:32.186071029 +0000 UTC m=+2711.881278988" Oct 30 20:16:32.286728 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:32.286693 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:32.704443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:32.699138 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:33.058202 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.058157 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g"] Oct 30 20:16:33.059031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.059015 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e11004-d729-4eec-8626-28eba4b01e90" containerName="istio-proxy" Oct 30 20:16:33.062705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.062687 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" Oct 30 20:16:33.072606 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.072570 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g"] Oct 30 20:16:33.147466 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.147436 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e16466-369a-423c-82c1-cbb82fd66fca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce9920af-qzp4g\" (UID: \"77e16466-369a-423c-82c1-cbb82fd66fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" Oct 30 20:16:33.147638 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.147623 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e16466-369a-423c-82c1-cbb82fd66fca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce9920af-qzp4g\" (UID: \"77e16466-369a-423c-82c1-cbb82fd66fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" Oct 30 20:16:33.147689 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.147674 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wp4d\" (UniqueName: \"kubernetes.io/projected/77e16466-369a-423c-82c1-cbb82fd66fca-kube-api-access-2wp4d\") pod \"maas-api-revoke-sub-ce9920af-qzp4g\" (UID: \"77e16466-369a-423c-82c1-cbb82fd66fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" Oct 30 20:16:33.166094 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.166057 5091 generic.go:356] "Generic (PLEG): container finished" podID="6192cc65-0103-4b75-b51c-3969262634eb" containerID="a7a03f85339cec74afb7a2b34c8934d2c51683bf8bdec844f756acf684bc6f42" exitCode=0 Oct 30 20:16:33.166522 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.166139 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" event={"ID":"6192cc65-0103-4b75-b51c-3969262634eb","Type":"ContainerDied","Data":"a7a03f85339cec74afb7a2b34c8934d2c51683bf8bdec844f756acf684bc6f42"} Oct 30 20:16:33.248952 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.248915 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e16466-369a-423c-82c1-cbb82fd66fca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce9920af-qzp4g\" (UID: \"77e16466-369a-423c-82c1-cbb82fd66fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" Oct 30 20:16:33.249126 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.248980 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wp4d\" (UniqueName: \"kubernetes.io/projected/77e16466-369a-423c-82c1-cbb82fd66fca-kube-api-access-2wp4d\") pod \"maas-api-revoke-sub-ce9920af-qzp4g\" (UID: \"77e16466-369a-423c-82c1-cbb82fd66fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" Oct 30 20:16:33.249230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.249183 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e16466-369a-423c-82c1-cbb82fd66fca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce9920af-qzp4g\" (UID: \"77e16466-369a-423c-82c1-cbb82fd66fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" Oct 30 20:16:33.249867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.249836 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e16466-369a-423c-82c1-cbb82fd66fca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce9920af-qzp4g\" (UID: \"77e16466-369a-423c-82c1-cbb82fd66fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" Oct 30 20:16:33.251629 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.251605 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e16466-369a-423c-82c1-cbb82fd66fca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce9920af-qzp4g\" (UID: \"77e16466-369a-423c-82c1-cbb82fd66fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" Oct 30 20:16:33.263595 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.263569 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wp4d\" (UniqueName: \"kubernetes.io/projected/77e16466-369a-423c-82c1-cbb82fd66fca-kube-api-access-2wp4d\") pod \"maas-api-revoke-sub-ce9920af-qzp4g\" (UID: \"77e16466-369a-423c-82c1-cbb82fd66fca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" Oct 30 20:16:33.383071 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.382981 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" Oct 30 20:16:33.643823 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.643655 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:33.693376 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:16:33.693335 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77e16466_369a_423c_82c1_cbb82fd66fca.slice/crio-24db5b3e703d41e610b6abbd391791a6262aa18a21fe09947fbd5f6315898859 WatchSource:0}: Error finding container 24db5b3e703d41e610b6abbd391791a6262aa18a21fe09947fbd5f6315898859: Status 404 returned error can't find the container with id 24db5b3e703d41e610b6abbd391791a6262aa18a21fe09947fbd5f6315898859 Oct 30 20:16:33.694844 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:33.694812 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g"] Oct 30 20:16:34.176651 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:34.176610 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" event={"ID":"77e16466-369a-423c-82c1-cbb82fd66fca","Type":"ContainerStarted","Data":"2f6b27a94c5903e94dfacfb80b8216c9ba171be0ea76ffe58f5101ba88956c8b"} Oct 30 20:16:34.177121 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:34.176683 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" event={"ID":"77e16466-369a-423c-82c1-cbb82fd66fca","Type":"ContainerStarted","Data":"24db5b3e703d41e610b6abbd391791a6262aa18a21fe09947fbd5f6315898859"} Oct 30 20:16:34.212239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:34.212177 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" podStartSLOduration=1.212160401 podStartE2EDuration="1.212160401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:34.210257713 +0000 UTC m=+2713.905465675" watchObservedRunningTime="2026-10-30 20:16:34.212160401 +0000 UTC m=+2713.907368354" Oct 30 20:16:34.355200 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:34.355173 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" Oct 30 20:16:34.477805 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:34.476906 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:34.504172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:34.490967 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6192cc65-0103-4b75-b51c-3969262634eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6192cc65-0103-4b75-b51c-3969262634eb-maas-api-openshift-ca\") pod \"6192cc65-0103-4b75-b51c-3969262634eb\" (UID: \"6192cc65-0103-4b75-b51c-3969262634eb\") " Oct 30 20:16:34.504172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:34.491090 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6192cc65-0103-4b75-b51c-3969262634eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6192cc65-0103-4b75-b51c-3969262634eb-maas-api-xks-ca\") pod \"6192cc65-0103-4b75-b51c-3969262634eb\" (UID: \"6192cc65-0103-4b75-b51c-3969262634eb\") " Oct 30 20:16:34.504172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:34.491303 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6192cc65-0103-4b75-b51c-3969262634eb-kube-api-access-pzgvx\" (UniqueName: \"kubernetes.io/projected/6192cc65-0103-4b75-b51c-3969262634eb-kube-api-access-pzgvx\") pod \"6192cc65-0103-4b75-b51c-3969262634eb\" (UID: \"6192cc65-0103-4b75-b51c-3969262634eb\") " Oct 30 20:16:34.504172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:34.499547 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6192cc65-0103-4b75-b51c-3969262634eb-maas-api-openshift-ca" pod "6192cc65-0103-4b75-b51c-3969262634eb" (UID: "6192cc65-0103-4b75-b51c-3969262634eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:34.522173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:34.512319 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6192cc65-0103-4b75-b51c-3969262634eb-kube-api-access-pzgvx" pod "6192cc65-0103-4b75-b51c-3969262634eb" (UID: "6192cc65-0103-4b75-b51c-3969262634eb"). InnerVolumeSpecName "kube-api-access-pzgvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:34.556314 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:34.556281 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6192cc65-0103-4b75-b51c-3969262634eb-maas-api-xks-ca" pod "6192cc65-0103-4b75-b51c-3969262634eb" (UID: "6192cc65-0103-4b75-b51c-3969262634eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:16:34.593991 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:34.593900 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6192cc65-0103-4b75-b51c-3969262634eb-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:34.593991 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:34.593939 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6192cc65-0103-4b75-b51c-3969262634eb-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:34.593991 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:34.593955 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzgvx\" (UniqueName: \"kubernetes.io/projected/6192cc65-0103-4b75-b51c-3969262634eb-kube-api-access-pzgvx\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:35.196336 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.196283 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" event={"ID":"6192cc65-0103-4b75-b51c-3969262634eb","Type":"ContainerDied","Data":"05df951be66a0d16ae3be2e471994597ae6e23a26a90dbd36f05344ff7b02304"} Oct 30 20:16:35.196831 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.196404 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05df951be66a0d16ae3be2e471994597ae6e23a26a90dbd36f05344ff7b02304" Oct 30 20:16:35.196831 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.196510 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml" Oct 30 20:16:35.198724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.198692 5091 generic.go:356] "Generic (PLEG): container finished" podID="77e16466-369a-423c-82c1-cbb82fd66fca" containerID="2f6b27a94c5903e94dfacfb80b8216c9ba171be0ea76ffe58f5101ba88956c8b" exitCode=0 Oct 30 20:16:35.198862 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.198739 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" event={"ID":"77e16466-369a-423c-82c1-cbb82fd66fca","Type":"ContainerDied","Data":"2f6b27a94c5903e94dfacfb80b8216c9ba171be0ea76ffe58f5101ba88956c8b"} Oct 30 20:16:35.303688 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.303340 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:35.490294 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.490088 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:35.571136 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.571102 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:16:35.631342 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.631311 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:35.745055 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.744966 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-data\") pod \"374daf82-137e-4a07-ad5f-b205bcb571fc\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " Oct 30 20:16:35.745277 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.745081 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-workload-certs\") pod \"374daf82-137e-4a07-ad5f-b205bcb571fc\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " Oct 30 20:16:35.745277 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.745245 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/374daf82-137e-4a07-ad5f-b205bcb571fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374daf82-137e-4a07-ad5f-b205bcb571fc-istiod-ca-cert\") pod \"374daf82-137e-4a07-ad5f-b205bcb571fc\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " Oct 30 20:16:35.745410 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.745328 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-envoy\") pod \"374daf82-137e-4a07-ad5f-b205bcb571fc\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " Oct 30 20:16:35.745410 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.745366 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-credential-socket\") pod \"374daf82-137e-4a07-ad5f-b205bcb571fc\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " Oct 30 20:16:35.745525 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.745511 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-token\" (UniqueName: \"kubernetes.io/projected/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-token\") pod \"374daf82-137e-4a07-ad5f-b205bcb571fc\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " Oct 30 20:16:35.745580 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.745550 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-workload-socket\") pod \"374daf82-137e-4a07-ad5f-b205bcb571fc\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " Oct 30 20:16:35.746188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.745641 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/374daf82-137e-4a07-ad5f-b205bcb571fc-kube-api-access-wxzqk\" (UniqueName: \"kubernetes.io/projected/374daf82-137e-4a07-ad5f-b205bcb571fc-kube-api-access-wxzqk\") pod \"374daf82-137e-4a07-ad5f-b205bcb571fc\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " Oct 30 20:16:35.746188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.745693 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-podinfo\") pod \"374daf82-137e-4a07-ad5f-b205bcb571fc\" (UID: \"374daf82-137e-4a07-ad5f-b205bcb571fc\") " Oct 30 20:16:35.746188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.745716 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-data" pod "374daf82-137e-4a07-ad5f-b205bcb571fc" (UID: "374daf82-137e-4a07-ad5f-b205bcb571fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:35.751505 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.746832 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-workload-socket" pod "374daf82-137e-4a07-ad5f-b205bcb571fc" (UID: "374daf82-137e-4a07-ad5f-b205bcb571fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:35.751505 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.747295 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-workload-certs" pod "374daf82-137e-4a07-ad5f-b205bcb571fc" (UID: "374daf82-137e-4a07-ad5f-b205bcb571fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:35.751505 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.747626 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-credential-socket" pod "374daf82-137e-4a07-ad5f-b205bcb571fc" (UID: "374daf82-137e-4a07-ad5f-b205bcb571fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:35.751505 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.748529 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/374daf82-137e-4a07-ad5f-b205bcb571fc-istiod-ca-cert" pod "374daf82-137e-4a07-ad5f-b205bcb571fc" (UID: "374daf82-137e-4a07-ad5f-b205bcb571fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:35.753824 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.753724 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-podinfo" pod "374daf82-137e-4a07-ad5f-b205bcb571fc" (UID: "374daf82-137e-4a07-ad5f-b205bcb571fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:16:35.753824 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.753783 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374daf82-137e-4a07-ad5f-b205bcb571fc-kube-api-access-wxzqk" pod "374daf82-137e-4a07-ad5f-b205bcb571fc" (UID: "374daf82-137e-4a07-ad5f-b205bcb571fc"). InnerVolumeSpecName "kube-api-access-wxzqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:35.753988 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.753812 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-token" pod "374daf82-137e-4a07-ad5f-b205bcb571fc" (UID: "374daf82-137e-4a07-ad5f-b205bcb571fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:35.754246 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.754218 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-envoy" pod "374daf82-137e-4a07-ad5f-b205bcb571fc" (UID: "374daf82-137e-4a07-ad5f-b205bcb571fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:35.756681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.756516 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxzqk\" (UniqueName: \"kubernetes.io/projected/374daf82-137e-4a07-ad5f-b205bcb571fc-kube-api-access-wxzqk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:35.756681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.756545 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:35.756681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.756561 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:35.756681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.756575 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:35.756681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.756590 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/374daf82-137e-4a07-ad5f-b205bcb571fc-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:35.756681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.756612 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:35.756681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.756625 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:35.756681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.756638 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/374daf82-137e-4a07-ad5f-b205bcb571fc-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:35.756681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.756652 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/374daf82-137e-4a07-ad5f-b205bcb571fc-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:35.850087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:35.850049 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:36.209433 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.209400 5091 generic.go:356] "Generic (PLEG): container finished" podID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerID="01e6ec8e203e65d17a3c0b1202e25b0a44ec2b522c011cf7b84a2bd1c7503c29" exitCode=0 Oct 30 20:16:36.209957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.209453 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" Oct 30 20:16:36.209957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.209495 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" event={"ID":"374daf82-137e-4a07-ad5f-b205bcb571fc","Type":"ContainerDied","Data":"01e6ec8e203e65d17a3c0b1202e25b0a44ec2b522c011cf7b84a2bd1c7503c29"} Oct 30 20:16:36.209957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.209559 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz" event={"ID":"374daf82-137e-4a07-ad5f-b205bcb571fc","Type":"ContainerDied","Data":"0d561ce8724f5a45e8534792b347e897d604f46b6b120ff2515836e820a5db70"} Oct 30 20:16:36.209957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.209595 5091 scope.go:122] "RemoveContainer" containerID="01e6ec8e203e65d17a3c0b1202e25b0a44ec2b522c011cf7b84a2bd1c7503c29" Oct 30 20:16:36.221961 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.221939 5091 scope.go:122] "RemoveContainer" containerID="f22c1b8878712b749f290f1e15da4c381305985e674b688ad3285f6b18ec360a" Oct 30 20:16:36.233704 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.233684 5091 scope.go:122] "RemoveContainer" containerID="01e6ec8e203e65d17a3c0b1202e25b0a44ec2b522c011cf7b84a2bd1c7503c29" Oct 30 20:16:36.234004 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:36.233983 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e6ec8e203e65d17a3c0b1202e25b0a44ec2b522c011cf7b84a2bd1c7503c29\": container with ID starting with 01e6ec8e203e65d17a3c0b1202e25b0a44ec2b522c011cf7b84a2bd1c7503c29 not found: ID does not exist" containerID="01e6ec8e203e65d17a3c0b1202e25b0a44ec2b522c011cf7b84a2bd1c7503c29" Oct 30 20:16:36.234065 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.234014 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e6ec8e203e65d17a3c0b1202e25b0a44ec2b522c011cf7b84a2bd1c7503c29"} err="failed to get container status \"01e6ec8e203e65d17a3c0b1202e25b0a44ec2b522c011cf7b84a2bd1c7503c29\": rpc error: code = NotFound desc = could not find container \"01e6ec8e203e65d17a3c0b1202e25b0a44ec2b522c011cf7b84a2bd1c7503c29\": container with ID starting with 01e6ec8e203e65d17a3c0b1202e25b0a44ec2b522c011cf7b84a2bd1c7503c29 not found: ID does not exist" Oct 30 20:16:36.234065 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.234032 5091 scope.go:122] "RemoveContainer" containerID="f22c1b8878712b749f290f1e15da4c381305985e674b688ad3285f6b18ec360a" Oct 30 20:16:36.234352 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:36.234332 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22c1b8878712b749f290f1e15da4c381305985e674b688ad3285f6b18ec360a\": container with ID starting with f22c1b8878712b749f290f1e15da4c381305985e674b688ad3285f6b18ec360a not found: ID does not exist" containerID="f22c1b8878712b749f290f1e15da4c381305985e674b688ad3285f6b18ec360a" Oct 30 20:16:36.234400 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.234358 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22c1b8878712b749f290f1e15da4c381305985e674b688ad3285f6b18ec360a"} err="failed to get container status \"f22c1b8878712b749f290f1e15da4c381305985e674b688ad3285f6b18ec360a\": rpc error: code = NotFound desc = could not find container \"f22c1b8878712b749f290f1e15da4c381305985e674b688ad3285f6b18ec360a\": container with ID starting with f22c1b8878712b749f290f1e15da4c381305985e674b688ad3285f6b18ec360a not found: ID does not exist" Oct 30 20:16:36.243949 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.243252 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz"] Oct 30 20:16:36.250280 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.250257 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c9bce8-openshift-default-5c5b54bfd5-9s2qz"] Oct 30 20:16:36.364085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.319780 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz"] Oct 30 20:16:36.364085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.321026 5091 cpu_manager.go:414] "containerMap: removing container" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerName="istio-proxy" Oct 30 20:16:36.364085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.321052 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerName="istio-proxy" Oct 30 20:16:36.364085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.321069 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6192cc65-0103-4b75-b51c-3969262634eb" containerName="revoke-keys" Oct 30 20:16:36.364085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.321077 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6192cc65-0103-4b75-b51c-3969262634eb" containerName="revoke-keys" Oct 30 20:16:36.364085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.321113 5091 cpu_manager.go:414] "containerMap: removing container" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerName="istio-proxy" Oct 30 20:16:36.364085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.321120 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerName="istio-proxy" Oct 30 20:16:36.364085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.321309 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerName="istio-proxy" Oct 30 20:16:36.364085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.321383 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6192cc65-0103-4b75-b51c-3969262634eb" containerName="revoke-keys" Oct 30 20:16:36.364085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.321409 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" containerName="istio-proxy" Oct 30 20:16:36.378393 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.375807 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz"] Oct 30 20:16:36.378393 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.376006 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" Oct 30 20:16:36.409504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.409465 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:36.445706 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.445672 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="374daf82-137e-4a07-ad5f-b205bcb571fc" path="/var/lib/kubelet/pods/374daf82-137e-4a07-ad5f-b205bcb571fc/volumes" Oct 30 20:16:36.472751 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.472672 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12da7406-84d4-4eb6-83ce-49b324d67d71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz\" (UID: \"12da7406-84d4-4eb6-83ce-49b324d67d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" Oct 30 20:16:36.472751 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.472743 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12da7406-84d4-4eb6-83ce-49b324d67d71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz\" (UID: \"12da7406-84d4-4eb6-83ce-49b324d67d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" Oct 30 20:16:36.473194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.473161 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nxd4\" (UniqueName: \"kubernetes.io/projected/12da7406-84d4-4eb6-83ce-49b324d67d71-kube-api-access-2nxd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz\" (UID: \"12da7406-84d4-4eb6-83ce-49b324d67d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" Oct 30 20:16:36.485366 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.485342 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" Oct 30 20:16:36.574648 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.574609 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77e16466-369a-423c-82c1-cbb82fd66fca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e16466-369a-423c-82c1-cbb82fd66fca-maas-api-xks-ca\") pod \"77e16466-369a-423c-82c1-cbb82fd66fca\" (UID: \"77e16466-369a-423c-82c1-cbb82fd66fca\") " Oct 30 20:16:36.574848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.574719 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77e16466-369a-423c-82c1-cbb82fd66fca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e16466-369a-423c-82c1-cbb82fd66fca-maas-api-openshift-ca\") pod \"77e16466-369a-423c-82c1-cbb82fd66fca\" (UID: \"77e16466-369a-423c-82c1-cbb82fd66fca\") " Oct 30 20:16:36.574981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.574964 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e16466-369a-423c-82c1-cbb82fd66fca-kube-api-access-2wp4d\" (UniqueName: \"kubernetes.io/projected/77e16466-369a-423c-82c1-cbb82fd66fca-kube-api-access-2wp4d\") pod \"77e16466-369a-423c-82c1-cbb82fd66fca\" (UID: \"77e16466-369a-423c-82c1-cbb82fd66fca\") " Oct 30 20:16:36.575072 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.575050 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e16466-369a-423c-82c1-cbb82fd66fca-maas-api-openshift-ca" pod "77e16466-369a-423c-82c1-cbb82fd66fca" (UID: "77e16466-369a-423c-82c1-cbb82fd66fca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:36.575530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.575503 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nxd4\" (UniqueName: \"kubernetes.io/projected/12da7406-84d4-4eb6-83ce-49b324d67d71-kube-api-access-2nxd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz\" (UID: \"12da7406-84d4-4eb6-83ce-49b324d67d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" Oct 30 20:16:36.575702 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.575684 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12da7406-84d4-4eb6-83ce-49b324d67d71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz\" (UID: \"12da7406-84d4-4eb6-83ce-49b324d67d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" Oct 30 20:16:36.576223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.576197 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12da7406-84d4-4eb6-83ce-49b324d67d71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz\" (UID: \"12da7406-84d4-4eb6-83ce-49b324d67d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" Oct 30 20:16:36.577089 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.576434 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e16466-369a-423c-82c1-cbb82fd66fca-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:36.577233 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.577124 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12da7406-84d4-4eb6-83ce-49b324d67d71-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz\" (UID: \"12da7406-84d4-4eb6-83ce-49b324d67d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" Oct 30 20:16:36.577305 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.577262 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e16466-369a-423c-82c1-cbb82fd66fca-maas-api-xks-ca" pod "77e16466-369a-423c-82c1-cbb82fd66fca" (UID: "77e16466-369a-423c-82c1-cbb82fd66fca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:16:36.577525 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.577500 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e16466-369a-423c-82c1-cbb82fd66fca-kube-api-access-2wp4d" pod "77e16466-369a-423c-82c1-cbb82fd66fca" (UID: "77e16466-369a-423c-82c1-cbb82fd66fca"). InnerVolumeSpecName "kube-api-access-2wp4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:36.578283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.578264 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12da7406-84d4-4eb6-83ce-49b324d67d71-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz\" (UID: \"12da7406-84d4-4eb6-83ce-49b324d67d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" Oct 30 20:16:36.589484 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.589458 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nxd4\" (UniqueName: \"kubernetes.io/projected/12da7406-84d4-4eb6-83ce-49b324d67d71-kube-api-access-2nxd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz\" (UID: \"12da7406-84d4-4eb6-83ce-49b324d67d71\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" Oct 30 20:16:36.678448 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.678419 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wp4d\" (UniqueName: \"kubernetes.io/projected/77e16466-369a-423c-82c1-cbb82fd66fca-kube-api-access-2wp4d\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:36.678448 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.678447 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e16466-369a-423c-82c1-cbb82fd66fca-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:36.712029 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:36.712000 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" Oct 30 20:16:37.001537 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:16:37.001499 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12da7406_84d4_4eb6_83ce_49b324d67d71.slice/crio-4807e77646fb1c66623edf210fec4278b1473c384cb233de3b1b5375151070fa WatchSource:0}: Error finding container 4807e77646fb1c66623edf210fec4278b1473c384cb233de3b1b5375151070fa: Status 404 returned error can't find the container with id 4807e77646fb1c66623edf210fec4278b1473c384cb233de3b1b5375151070fa Oct 30 20:16:37.003044 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:37.003015 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz"] Oct 30 20:16:37.219940 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:37.219900 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" event={"ID":"12da7406-84d4-4eb6-83ce-49b324d67d71","Type":"ContainerStarted","Data":"4807e77646fb1c66623edf210fec4278b1473c384cb233de3b1b5375151070fa"} Oct 30 20:16:37.222696 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:37.222673 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" Oct 30 20:16:37.222696 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:37.222681 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g" event={"ID":"77e16466-369a-423c-82c1-cbb82fd66fca","Type":"ContainerDied","Data":"24db5b3e703d41e610b6abbd391791a6262aa18a21fe09947fbd5f6315898859"} Oct 30 20:16:37.222917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:37.222719 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24db5b3e703d41e610b6abbd391791a6262aa18a21fe09947fbd5f6315898859" Oct 30 20:16:37.325283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:37.325242 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:38.231596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:38.231565 5091 generic.go:356] "Generic (PLEG): container finished" podID="12da7406-84d4-4eb6-83ce-49b324d67d71" containerID="ad27fda62ec07b14de265f7ad372e74b6acb9c367c8783c7fa899e56506d4989" exitCode=0 Oct 30 20:16:38.232018 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:38.231642 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" event={"ID":"12da7406-84d4-4eb6-83ce-49b324d67d71","Type":"ContainerDied","Data":"ad27fda62ec07b14de265f7ad372e74b6acb9c367c8783c7fa899e56506d4989"} Oct 30 20:16:38.384440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:38.384347 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:39.149518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:39.149474 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:39.436709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:39.436682 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" Oct 30 20:16:39.533206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:39.533171 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12da7406-84d4-4eb6-83ce-49b324d67d71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12da7406-84d4-4eb6-83ce-49b324d67d71-maas-api-openshift-ca\") pod \"12da7406-84d4-4eb6-83ce-49b324d67d71\" (UID: \"12da7406-84d4-4eb6-83ce-49b324d67d71\") " Oct 30 20:16:39.533376 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:39.533365 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12da7406-84d4-4eb6-83ce-49b324d67d71-kube-api-access-2nxd4\" (UniqueName: \"kubernetes.io/projected/12da7406-84d4-4eb6-83ce-49b324d67d71-kube-api-access-2nxd4\") pod \"12da7406-84d4-4eb6-83ce-49b324d67d71\" (UID: \"12da7406-84d4-4eb6-83ce-49b324d67d71\") " Oct 30 20:16:39.533531 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:39.533497 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12da7406-84d4-4eb6-83ce-49b324d67d71-maas-api-openshift-ca" pod "12da7406-84d4-4eb6-83ce-49b324d67d71" (UID: "12da7406-84d4-4eb6-83ce-49b324d67d71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:39.533625 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:39.533588 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12da7406-84d4-4eb6-83ce-49b324d67d71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12da7406-84d4-4eb6-83ce-49b324d67d71-maas-api-xks-ca\") pod \"12da7406-84d4-4eb6-83ce-49b324d67d71\" (UID: \"12da7406-84d4-4eb6-83ce-49b324d67d71\") " Oct 30 20:16:39.534951 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:39.534920 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12da7406-84d4-4eb6-83ce-49b324d67d71-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:39.535884 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:39.535859 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12da7406-84d4-4eb6-83ce-49b324d67d71-maas-api-xks-ca" pod "12da7406-84d4-4eb6-83ce-49b324d67d71" (UID: "12da7406-84d4-4eb6-83ce-49b324d67d71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:16:39.536004 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:39.535982 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12da7406-84d4-4eb6-83ce-49b324d67d71-kube-api-access-2nxd4" pod "12da7406-84d4-4eb6-83ce-49b324d67d71" (UID: "12da7406-84d4-4eb6-83ce-49b324d67d71"). InnerVolumeSpecName "kube-api-access-2nxd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:39.636931 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:39.636898 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nxd4\" (UniqueName: \"kubernetes.io/projected/12da7406-84d4-4eb6-83ce-49b324d67d71-kube-api-access-2nxd4\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:39.636931 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:39.636929 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12da7406-84d4-4eb6-83ce-49b324d67d71-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:39.959168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:39.955336 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:40.249451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:40.249407 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" event={"ID":"12da7406-84d4-4eb6-83ce-49b324d67d71","Type":"ContainerDied","Data":"4807e77646fb1c66623edf210fec4278b1473c384cb233de3b1b5375151070fa"} Oct 30 20:16:40.249614 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:40.249473 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4807e77646fb1c66623edf210fec4278b1473c384cb233de3b1b5375151070fa" Oct 30 20:16:40.249614 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:40.249432 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz" Oct 30 20:16:40.840639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:40.840547 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:41.006141 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:41.006104 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4"] Oct 30 20:16:41.099919 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:41.099832 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:41.417287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:41.407887 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:41.983738 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:41.983677 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:42.790169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:42.789736 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:43.562824 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:43.562792 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:43.773861 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:43.773821 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:43.854294 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:43.854207 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj"] Oct 30 20:16:43.928165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:43.924673 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:43.936177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:43.928520 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-969878-sllwj"] Oct 30 20:16:44.095170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.088726 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:44.339190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.337728 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:44.444543 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.444507 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c8c85f0-5a0d-4306-a911-5d872a620026" path="/var/lib/kubelet/pods/4c8c85f0-5a0d-4306-a911-5d872a620026/volumes" Oct 30 20:16:44.783858 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.783822 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn"] Oct 30 20:16:44.784897 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.784876 5091 cpu_manager.go:414] "containerMap: removing container" podUID="77e16466-369a-423c-82c1-cbb82fd66fca" containerName="revoke-keys" Oct 30 20:16:44.784959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.784900 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e16466-369a-423c-82c1-cbb82fd66fca" containerName="revoke-keys" Oct 30 20:16:44.784959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.784919 5091 cpu_manager.go:414] "containerMap: removing container" podUID="12da7406-84d4-4eb6-83ce-49b324d67d71" containerName="revoke-keys" Oct 30 20:16:44.784959 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.784928 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12da7406-84d4-4eb6-83ce-49b324d67d71" containerName="revoke-keys" Oct 30 20:16:44.785196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.785180 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e16466-369a-423c-82c1-cbb82fd66fca" containerName="revoke-keys" Oct 30 20:16:44.785248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.785200 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="12da7406-84d4-4eb6-83ce-49b324d67d71" containerName="revoke-keys" Oct 30 20:16:44.790027 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.790005 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:44.798071 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.798037 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn"] Oct 30 20:16:44.802446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.802358 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:44.966739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.966686 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lh9vn\" (UniqueName: \"kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-kube-api-access-lh9vn\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:44.967219 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.967188 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2eb7ef3-adf6-4710-9760-f952a4b003ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:44.967545 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.967525 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2eb7ef3-adf6-4710-9760-f952a4b003ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:44.967721 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.967697 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:44.975567 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:44.975307 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:45.069174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.069064 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2eb7ef3-adf6-4710-9760-f952a4b003ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:45.069174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.069118 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:45.069432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.069317 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lh9vn\" (UniqueName: \"kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-kube-api-access-lh9vn\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:45.069496 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.069445 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2eb7ef3-adf6-4710-9760-f952a4b003ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:45.070034 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.070011 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2eb7ef3-adf6-4710-9760-f952a4b003ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:45.070252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.070225 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2eb7ef3-adf6-4710-9760-f952a4b003ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:45.071931 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.071908 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:45.083582 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.083557 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lh9vn\" (UniqueName: \"kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-kube-api-access-lh9vn\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:45.157280 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.157250 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:45.175176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.162226 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks"] Oct 30 20:16:45.197712 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.195878 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:45.221167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.219223 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks"] Oct 30 20:16:45.312707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.308366 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzl6\" (UniqueName: \"kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-kube-api-access-6pzl6\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8lskks\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:45.312707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.308542 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5688c73b-8d9e-49d2-88b3-15efc5942caa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8lskks\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:45.312707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.308751 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5688c73b-8d9e-49d2-88b3-15efc5942caa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8lskks\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:45.312707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.308839 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8lskks\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:45.416992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.416843 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5688c73b-8d9e-49d2-88b3-15efc5942caa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8lskks\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:45.418702 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.418662 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5688c73b-8d9e-49d2-88b3-15efc5942caa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8lskks\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:45.418998 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.418973 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8lskks\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:45.419326 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.419293 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzl6\" (UniqueName: \"kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-kube-api-access-6pzl6\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8lskks\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:45.420579 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.420556 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5688c73b-8d9e-49d2-88b3-15efc5942caa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8lskks\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:45.424929 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.424902 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5688c73b-8d9e-49d2-88b3-15efc5942caa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8lskks\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:45.428453 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.428428 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8lskks\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:45.442934 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.442797 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzl6\" (UniqueName: \"kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-kube-api-access-6pzl6\") pod \"payload-processing-external-model-e2e-extmod-a-222480-68f8lskks\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:45.648449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.648168 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:45.862061 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.861384 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:45.984594 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:45.981878 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn"] Oct 30 20:16:46.029659 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:46.029259 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" podUID="d973dcd0-1e61-4929-a1a0-648efb94577c" containerName="payload-pre-processing" containerID="cri-o://d62a88067ef970155a179d2d94cbde6d036b01a624d783e8292398fe3a802857" gracePeriod=25 Oct 30 20:16:46.071211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:46.069886 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:46.099056 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:46.098967 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:16:46.181125 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:46.181096 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks"] Oct 30 20:16:46.182511 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:16:46.182485 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5688c73b_8d9e_49d2_88b3_15efc5942caa.slice/crio-6bb7ed55ea77c50a4c3726a18f0c9920dcf9f6b496755a21b7cde734dd7be772 WatchSource:0}: Error finding container 6bb7ed55ea77c50a4c3726a18f0c9920dcf9f6b496755a21b7cde734dd7be772: Status 404 returned error can't find the container with id 6bb7ed55ea77c50a4c3726a18f0c9920dcf9f6b496755a21b7cde734dd7be772 Oct 30 20:16:46.411037 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:46.410939 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" event={"ID":"5688c73b-8d9e-49d2-88b3-15efc5942caa","Type":"ContainerStarted","Data":"f3933c7fc908700a7eaf5f24c652bf1ac7b16226dcb4ae40f87d78a7d751dc3d"} Oct 30 20:16:46.411037 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:46.411004 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" event={"ID":"5688c73b-8d9e-49d2-88b3-15efc5942caa","Type":"ContainerStarted","Data":"6bb7ed55ea77c50a4c3726a18f0c9920dcf9f6b496755a21b7cde734dd7be772"} Oct 30 20:16:46.411335 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:46.411049 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:46.413070 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:46.413045 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" event={"ID":"f2eb7ef3-adf6-4710-9760-f952a4b003ba","Type":"ContainerStarted","Data":"bec50fb0545d9ba6394926d201138d8769d85499870016561cfb89c403230d5a"} Oct 30 20:16:46.413239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:46.413086 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" event={"ID":"f2eb7ef3-adf6-4710-9760-f952a4b003ba","Type":"ContainerStarted","Data":"a30bce97e7f4cc95272374392e8c0730f6c43caac1736c1d990648099bb8f223"} Oct 30 20:16:46.413239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:46.413187 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:46.420761 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:46.420735 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx"] Oct 30 20:16:46.448696 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:46.448627 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" podStartSLOduration=1.448606146 podStartE2EDuration="1.448606146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:46.442372021 +0000 UTC m=+2726.137579979" watchObservedRunningTime="2026-10-30 20:16:46.448606146 +0000 UTC m=+2726.143814100" Oct 30 20:16:46.451644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:46.451317 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5"] Oct 30 20:16:46.478078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:46.478007 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" podStartSLOduration=2.477988299 podStartE2EDuration="2.477988299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:46.471589297 +0000 UTC m=+2726.166797256" watchObservedRunningTime="2026-10-30 20:16:46.477988299 +0000 UTC m=+2726.173196258" Oct 30 20:16:46.570065 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:46.569903 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:46.739253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:46.738169 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:47.088475 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.088429 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:47.284963 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.275061 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:47.390353 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.390269 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks"] Oct 30 20:16:47.427044 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.427014 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v"] Oct 30 20:16:47.431929 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.431909 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:16:47.445525 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.445499 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v"] Oct 30 20:16:47.592452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.592403 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsxql\" (UniqueName: \"kubernetes.io/projected/19ec9497-2d39-4468-8165-921ef5727351-kube-api-access-rsxql\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:16:47.592645 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.592494 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19ec9497-2d39-4468-8165-921ef5727351-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:16:47.594723 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.593045 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ec9497-2d39-4468-8165-921ef5727351-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:16:47.594723 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.593170 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19ec9497-2d39-4468-8165-921ef5727351-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:16:47.594723 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:47.593823 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-222480/agc-cred-bd77348f7e09: secret "agc-cred-bd77348f7e09" not found Oct 30 20:16:47.594723 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:47.593849 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks: secret "agc-cred-bd77348f7e09" not found Oct 30 20:16:47.594723 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:47.593910 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-provider-credentials podName:5688c73b-8d9e-49d2-88b3-15efc5942caa nodeName:}" failed. No retries permitted until 2026-10-30 20:16:48.093890144 +0000 UTC m=+2727.789098079 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" (UID: "5688c73b-8d9e-49d2-88b3-15efc5942caa") : secret "agc-cred-bd77348f7e09" not found Oct 30 20:16:47.595116 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:47.594733 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4d5758/agc-cred-20d4e7c9cc7c: secret "agc-cred-20d4e7c9cc7c" not found Oct 30 20:16:47.595116 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:47.594756 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn: secret "agc-cred-20d4e7c9cc7c" not found Oct 30 20:16:47.595116 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:47.594794 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-provider-credentials podName:f2eb7ef3-adf6-4710-9760-f952a4b003ba nodeName:}" failed. No retries permitted until 2026-10-30 20:16:48.094781566 +0000 UTC m=+2727.789989532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" (UID: "f2eb7ef3-adf6-4710-9760-f952a4b003ba") : secret "agc-cred-20d4e7c9cc7c" not found Oct 30 20:16:47.595497 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.595456 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:47.696229 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.693494 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr"] Oct 30 20:16:47.715936 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.707615 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" Oct 30 20:16:47.719900 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.719868 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr"] Oct 30 20:16:47.741190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.720923 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19ec9497-2d39-4468-8165-921ef5727351-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:16:47.741458 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.731162 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:16:47.745171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.741077 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19ec9497-2d39-4468-8165-921ef5727351-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:16:47.745528 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.745507 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ec9497-2d39-4468-8165-921ef5727351-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:16:47.804437 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.804387 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19ec9497-2d39-4468-8165-921ef5727351-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:16:47.806495 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.806437 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsxql\" (UniqueName: \"kubernetes.io/projected/19ec9497-2d39-4468-8165-921ef5727351-kube-api-access-rsxql\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:16:47.808555 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.746437 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ec9497-2d39-4468-8165-921ef5727351-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:16:47.836606 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.835616 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19ec9497-2d39-4468-8165-921ef5727351-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:16:47.836606 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.835851 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:16:47.893994 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.893961 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn"] Oct 30 20:16:47.906874 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.903101 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2"] Oct 30 20:16:47.908892 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.908848 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4917e43-nwsvr\" (UID: \"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" Oct 30 20:16:47.909245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.909218 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4917e43-nwsvr\" (UID: \"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" Oct 30 20:16:47.909462 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.909442 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2sk\" (UniqueName: \"kubernetes.io/projected/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-kube-api-access-wp2sk\") pod \"maas-api-revoke-sub-e4917e43-nwsvr\" (UID: \"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" Oct 30 20:16:47.921390 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.921332 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsxql\" (UniqueName: \"kubernetes.io/projected/19ec9497-2d39-4468-8165-921ef5727351-kube-api-access-rsxql\") pod \"payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:16:47.928226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.926808 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:16:47.928394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:47.926711 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2"] Oct 30 20:16:48.010776 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.010744 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4917e43-nwsvr\" (UID: \"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" Oct 30 20:16:48.010950 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.010798 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af043c89-cf40-4065-b7ea-36f46412aa95-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:16:48.010950 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.010867 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2sk\" (UniqueName: \"kubernetes.io/projected/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-kube-api-access-wp2sk\") pod \"maas-api-revoke-sub-e4917e43-nwsvr\" (UID: \"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" Oct 30 20:16:48.011040 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.010994 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5bhc\" (UniqueName: \"kubernetes.io/projected/af043c89-cf40-4065-b7ea-36f46412aa95-kube-api-access-p5bhc\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:16:48.011040 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.011029 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af043c89-cf40-4065-b7ea-36f46412aa95-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:16:48.011219 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.011199 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af043c89-cf40-4065-b7ea-36f46412aa95-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:16:48.011290 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.011273 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4917e43-nwsvr\" (UID: \"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" Oct 30 20:16:48.011592 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.011566 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4917e43-nwsvr\" (UID: \"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" Oct 30 20:16:48.013729 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.013702 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4917e43-nwsvr\" (UID: \"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" Oct 30 20:16:48.025000 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.024977 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2sk\" (UniqueName: \"kubernetes.io/projected/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-kube-api-access-wp2sk\") pod \"maas-api-revoke-sub-e4917e43-nwsvr\" (UID: \"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" Oct 30 20:16:48.049610 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.049586 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:16:48.112489 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.112452 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af043c89-cf40-4065-b7ea-36f46412aa95-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:16:48.112867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.112686 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af043c89-cf40-4065-b7ea-36f46412aa95-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:16:48.112867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.112800 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5bhc\" (UniqueName: \"kubernetes.io/projected/af043c89-cf40-4065-b7ea-36f46412aa95-kube-api-access-p5bhc\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:16:48.112867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.112820 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af043c89-cf40-4065-b7ea-36f46412aa95-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:16:48.113861 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.113165 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af043c89-cf40-4065-b7ea-36f46412aa95-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:16:48.113861 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:48.113269 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4d5758/agc-cred-20d4e7c9cc7c: secret "agc-cred-20d4e7c9cc7c" not found Oct 30 20:16:48.113861 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:48.113297 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn: secret "agc-cred-20d4e7c9cc7c" not found Oct 30 20:16:48.113861 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:48.113351 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-provider-credentials podName:f2eb7ef3-adf6-4710-9760-f952a4b003ba nodeName:}" failed. No retries permitted until 2026-10-30 20:16:49.113337434 +0000 UTC m=+2728.808545391 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" (UID: "f2eb7ef3-adf6-4710-9760-f952a4b003ba") : secret "agc-cred-20d4e7c9cc7c" not found Oct 30 20:16:48.113861 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:48.113651 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-222480/agc-cred-bd77348f7e09: secret "agc-cred-bd77348f7e09" not found Oct 30 20:16:48.113861 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:48.113665 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks: secret "agc-cred-bd77348f7e09" not found Oct 30 20:16:48.113861 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:48.113697 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-provider-credentials podName:5688c73b-8d9e-49d2-88b3-15efc5942caa nodeName:}" failed. No retries permitted until 2026-10-30 20:16:49.113686989 +0000 UTC m=+2728.808894946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" (UID: "5688c73b-8d9e-49d2-88b3-15efc5942caa") : secret "agc-cred-bd77348f7e09" not found Oct 30 20:16:48.113861 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.113700 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af043c89-cf40-4065-b7ea-36f46412aa95-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:16:48.115257 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.115238 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af043c89-cf40-4065-b7ea-36f46412aa95-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:16:48.140266 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.138108 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5bhc\" (UniqueName: \"kubernetes.io/projected/af043c89-cf40-4065-b7ea-36f46412aa95-kube-api-access-p5bhc\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:16:48.203787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.203748 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" Oct 30 20:16:48.259999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.259962 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:16:48.354813 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:16:48.354779 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ec9497_2d39_4468_8165_921ef5727351.slice/crio-d352abcc2f6e7d638a151625571a5d60779684b98f8edb46227f4561521259cc WatchSource:0}: Error finding container d352abcc2f6e7d638a151625571a5d60779684b98f8edb46227f4561521259cc: Status 404 returned error can't find the container with id d352abcc2f6e7d638a151625571a5d60779684b98f8edb46227f4561521259cc Oct 30 20:16:48.355998 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.355970 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v"] Oct 30 20:16:48.431421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.431373 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" event={"ID":"19ec9497-2d39-4468-8165-921ef5727351","Type":"ContainerStarted","Data":"d352abcc2f6e7d638a151625571a5d60779684b98f8edb46227f4561521259cc"} Oct 30 20:16:48.507374 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:16:48.507343 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bcbdc53_3a4f_4f3f_8145_f0f5e02f41bd.slice/crio-9ddb8ed2ed013d9776b1882054deeb3afc6a66ec0442910ce3bd95b49f908698 WatchSource:0}: Error finding container 9ddb8ed2ed013d9776b1882054deeb3afc6a66ec0442910ce3bd95b49f908698: Status 404 returned error can't find the container with id 9ddb8ed2ed013d9776b1882054deeb3afc6a66ec0442910ce3bd95b49f908698 Oct 30 20:16:48.507718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.507697 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr"] Oct 30 20:16:48.561468 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.561434 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2"] Oct 30 20:16:48.562604 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:16:48.562581 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf043c89_cf40_4065_b7ea_36f46412aa95.slice/crio-160f7fb923402f4a5d5757f0d58d58e714e1d4425ea7d3d16d007dfe51ff3ce3 WatchSource:0}: Error finding container 160f7fb923402f4a5d5757f0d58d58e714e1d4425ea7d3d16d007dfe51ff3ce3: Status 404 returned error can't find the container with id 160f7fb923402f4a5d5757f0d58d58e714e1d4425ea7d3d16d007dfe51ff3ce3 Oct 30 20:16:48.674269 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:48.671743 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:49.128887 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:49.128793 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-222480/agc-cred-bd77348f7e09: secret "agc-cred-bd77348f7e09" not found Oct 30 20:16:49.128887 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:49.128820 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks: secret "agc-cred-bd77348f7e09" not found Oct 30 20:16:49.128887 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:49.128834 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4d5758/agc-cred-20d4e7c9cc7c: secret "agc-cred-20d4e7c9cc7c" not found Oct 30 20:16:49.128887 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:49.128859 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn: secret "agc-cred-20d4e7c9cc7c" not found Oct 30 20:16:49.128887 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:49.128876 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-provider-credentials podName:5688c73b-8d9e-49d2-88b3-15efc5942caa nodeName:}" failed. No retries permitted until 2026-10-30 20:16:51.128857364 +0000 UTC m=+2730.824065321 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" (UID: "5688c73b-8d9e-49d2-88b3-15efc5942caa") : secret "agc-cred-bd77348f7e09" not found Oct 30 20:16:49.129451 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:49.128895 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-provider-credentials podName:f2eb7ef3-adf6-4710-9760-f952a4b003ba nodeName:}" failed. No retries permitted until 2026-10-30 20:16:51.12888423 +0000 UTC m=+2730.824092166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" (UID: "f2eb7ef3-adf6-4710-9760-f952a4b003ba") : secret "agc-cred-20d4e7c9cc7c" not found Oct 30 20:16:49.520898 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:49.508346 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" event={"ID":"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd","Type":"ContainerStarted","Data":"fca466f8097ad7f34aab865f0a84b55e36713f376ac5034e5cb5d1c18ddc60ba"} Oct 30 20:16:49.520898 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:49.508419 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" event={"ID":"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd","Type":"ContainerStarted","Data":"9ddb8ed2ed013d9776b1882054deeb3afc6a66ec0442910ce3bd95b49f908698"} Oct 30 20:16:49.530009 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:49.529975 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" event={"ID":"af043c89-cf40-4065-b7ea-36f46412aa95","Type":"ContainerStarted","Data":"60f42fa419beccbb3685206e2c03555e560891f3b0e22f590345d5e6c4eac8b2"} Oct 30 20:16:49.536246 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:49.536211 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" event={"ID":"af043c89-cf40-4065-b7ea-36f46412aa95","Type":"ContainerStarted","Data":"160f7fb923402f4a5d5757f0d58d58e714e1d4425ea7d3d16d007dfe51ff3ce3"} Oct 30 20:16:49.536431 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:49.536419 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:16:49.536537 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:49.536522 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" event={"ID":"19ec9497-2d39-4468-8165-921ef5727351","Type":"ContainerStarted","Data":"607f387f376a22926f473451311929ea7d404297abc74dc3d9989ac9dd87486a"} Oct 30 20:16:49.550301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:49.549821 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:16:49.559804 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:49.559731 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" podStartSLOduration=2.5597139220000003 podStartE2EDuration="2.559713922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:49.546413873 +0000 UTC m=+2729.241621832" watchObservedRunningTime="2026-10-30 20:16:49.559713922 +0000 UTC m=+2729.254921881" Oct 30 20:16:49.594224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:49.591296 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" podStartSLOduration=2.5912324509999998 podStartE2EDuration="2.591232451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:49.576680907 +0000 UTC m=+2729.271888867" watchObservedRunningTime="2026-10-30 20:16:49.591232451 +0000 UTC m=+2729.286440410" Oct 30 20:16:49.612172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:49.610260 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" podStartSLOduration=2.610237845 podStartE2EDuration="2.610237845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:49.601206169 +0000 UTC m=+2729.296414129" watchObservedRunningTime="2026-10-30 20:16:49.610237845 +0000 UTC m=+2729.305445831" Oct 30 20:16:49.628886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:49.628851 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:50.093739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:50.093695 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" podUID="d973dcd0-1e61-4929-a1a0-648efb94577c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:16:50.539099 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:50.539064 5091 generic.go:356] "Generic (PLEG): container finished" podID="1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd" containerID="fca466f8097ad7f34aab865f0a84b55e36713f376ac5034e5cb5d1c18ddc60ba" exitCode=0 Oct 30 20:16:50.539538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:50.539181 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" event={"ID":"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd","Type":"ContainerDied","Data":"fca466f8097ad7f34aab865f0a84b55e36713f376ac5034e5cb5d1c18ddc60ba"} Oct 30 20:16:50.640042 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:50.635343 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:51.114969 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:51.114935 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:51.187172 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:51.180548 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-4d5758/agc-cred-20d4e7c9cc7c: secret "agc-cred-20d4e7c9cc7c" not found Oct 30 20:16:51.187172 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:51.180593 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn: secret "agc-cred-20d4e7c9cc7c" not found Oct 30 20:16:51.187172 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:51.180662 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-provider-credentials podName:f2eb7ef3-adf6-4710-9760-f952a4b003ba nodeName:}" failed. No retries permitted until 2026-10-30 20:16:55.180642863 +0000 UTC m=+2734.875850813 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" (UID: "f2eb7ef3-adf6-4710-9760-f952a4b003ba") : secret "agc-cred-20d4e7c9cc7c" not found Oct 30 20:16:51.187172 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:51.181068 5091 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-222480/agc-cred-bd77348f7e09: secret "agc-cred-bd77348f7e09" not found Oct 30 20:16:51.187172 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:51.181087 5091 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks: secret "agc-cred-bd77348f7e09" not found Oct 30 20:16:51.187172 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:16:51.181132 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-provider-credentials podName:5688c73b-8d9e-49d2-88b3-15efc5942caa nodeName:}" failed. No retries permitted until 2026-10-30 20:16:55.18111594 +0000 UTC m=+2734.876323893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" (UID: "5688c73b-8d9e-49d2-88b3-15efc5942caa") : secret "agc-cred-bd77348f7e09" not found Oct 30 20:16:51.365216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:51.364512 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:51.426964 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:51.426925 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph"] Oct 30 20:16:51.427225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:51.427195 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerName="istio-proxy" containerID="cri-o://e9fbd8ce7aaa5f7b1f3ce5a3ad477d68b5c35005d69c74a8e978c5198cd79ed2" gracePeriod=30 Oct 30 20:16:51.427869 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:51.427843 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" podUID="d7db6874-2d87-4709-9545-0f29a1532a6c" containerName="payload-processing" containerID="cri-o://501ee98893a5b2731381939287fc18f41199f33e95ea797bf31bf0b6ddbdf772" gracePeriod=25 Oct 30 20:16:51.451719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:51.451666 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" podUID="002df7b8-201d-45a9-be26-076d2b887a7b" containerName="payload-pre-processing" containerID="cri-o://fed957e828c7039bedfa8650c9d549e3003beac340e0e6966e1faadb5e7850db" gracePeriod=25 Oct 30 20:16:51.551992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:51.551954 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:51.737236 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:51.736774 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:51.903963 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:51.903923 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn"] Oct 30 20:16:51.909930 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:51.909905 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" Oct 30 20:16:51.922819 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:51.922786 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn"] Oct 30 20:16:52.001728 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.001703 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" Oct 30 20:16:52.023790 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.023759 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/992deac8-a20c-4142-a66c-8f7ca3b3b34d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn\" (UID: \"992deac8-a20c-4142-a66c-8f7ca3b3b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" Oct 30 20:16:52.024002 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.023979 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/992deac8-a20c-4142-a66c-8f7ca3b3b34d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn\" (UID: \"992deac8-a20c-4142-a66c-8f7ca3b3b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" Oct 30 20:16:52.024124 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.024106 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckfn2\" (UniqueName: \"kubernetes.io/projected/992deac8-a20c-4142-a66c-8f7ca3b3b34d-kube-api-access-ckfn2\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn\" (UID: \"992deac8-a20c-4142-a66c-8f7ca3b3b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" Oct 30 20:16:52.124686 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.124650 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-maas-api-openshift-ca\") pod \"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd\" (UID: \"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd\") " Oct 30 20:16:52.124858 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.124821 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-maas-api-xks-ca\") pod \"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd\" (UID: \"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd\") " Oct 30 20:16:52.124921 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.124892 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-kube-api-access-wp2sk\" (UniqueName: \"kubernetes.io/projected/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-kube-api-access-wp2sk\") pod \"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd\" (UID: \"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd\") " Oct 30 20:16:52.125036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.125008 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-maas-api-openshift-ca" pod "1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd" (UID: "1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:52.125454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.125431 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/992deac8-a20c-4142-a66c-8f7ca3b3b34d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn\" (UID: \"992deac8-a20c-4142-a66c-8f7ca3b3b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" Oct 30 20:16:52.125594 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.125577 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckfn2\" (UniqueName: \"kubernetes.io/projected/992deac8-a20c-4142-a66c-8f7ca3b3b34d-kube-api-access-ckfn2\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn\" (UID: \"992deac8-a20c-4142-a66c-8f7ca3b3b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" Oct 30 20:16:52.125815 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.125797 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/992deac8-a20c-4142-a66c-8f7ca3b3b34d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn\" (UID: \"992deac8-a20c-4142-a66c-8f7ca3b3b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" Oct 30 20:16:52.125957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.125940 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:52.126197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.126135 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/992deac8-a20c-4142-a66c-8f7ca3b3b34d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn\" (UID: \"992deac8-a20c-4142-a66c-8f7ca3b3b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" Oct 30 20:16:52.127000 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.126977 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-maas-api-xks-ca" pod "1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd" (UID: "1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:16:52.127236 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.127214 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-kube-api-access-wp2sk" pod "1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd" (UID: "1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd"). InnerVolumeSpecName "kube-api-access-wp2sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:52.128492 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.128474 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/992deac8-a20c-4142-a66c-8f7ca3b3b34d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn\" (UID: \"992deac8-a20c-4142-a66c-8f7ca3b3b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" Oct 30 20:16:52.139778 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.139755 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckfn2\" (UniqueName: \"kubernetes.io/projected/992deac8-a20c-4142-a66c-8f7ca3b3b34d-kube-api-access-ckfn2\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn\" (UID: \"992deac8-a20c-4142-a66c-8f7ca3b3b34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" Oct 30 20:16:52.227302 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.227265 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:52.227302 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.227298 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp2sk\" (UniqueName: \"kubernetes.io/projected/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd-kube-api-access-wp2sk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:52.236477 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.236449 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" Oct 30 20:16:52.424901 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.424865 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:52.529503 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:16:52.529469 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992deac8_a20c_4142_a66c_8f7ca3b3b34d.slice/crio-8102429eda178de867c82bfa03183a0e27e02944ffa10a6d61c0ee5de4649bf8 WatchSource:0}: Error finding container 8102429eda178de867c82bfa03183a0e27e02944ffa10a6d61c0ee5de4649bf8: Status 404 returned error can't find the container with id 8102429eda178de867c82bfa03183a0e27e02944ffa10a6d61c0ee5de4649bf8 Oct 30 20:16:52.529894 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.529869 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn"] Oct 30 20:16:52.561747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.561716 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" event={"ID":"992deac8-a20c-4142-a66c-8f7ca3b3b34d","Type":"ContainerStarted","Data":"8102429eda178de867c82bfa03183a0e27e02944ffa10a6d61c0ee5de4649bf8"} Oct 30 20:16:52.563761 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.563734 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" Oct 30 20:16:52.563761 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.563751 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr" event={"ID":"1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd","Type":"ContainerDied","Data":"9ddb8ed2ed013d9776b1882054deeb3afc6a66ec0442910ce3bd95b49f908698"} Oct 30 20:16:52.563901 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.563798 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ddb8ed2ed013d9776b1882054deeb3afc6a66ec0442910ce3bd95b49f908698" Oct 30 20:16:52.721167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.718559 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:52.861367 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.857445 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml"] Oct 30 20:16:52.873834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.873793 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d97cc8-2ztml"] Oct 30 20:16:52.888597 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:52.888074 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:53.028362 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.028327 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:53.249059 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.249003 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:53.432502 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.432453 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" podUID="f2eb7ef3-adf6-4710-9760-f952a4b003ba" containerName="payload-processing" containerID="cri-o://bec50fb0545d9ba6394926d201138d8769d85499870016561cfb89c403230d5a" gracePeriod=25 Oct 30 20:16:53.432686 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.432464 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" podUID="5688c73b-8d9e-49d2-88b3-15efc5942caa" containerName="payload-processing" containerID="cri-o://f3933c7fc908700a7eaf5f24c652bf1ac7b16226dcb4ae40f87d78a7d751dc3d" gracePeriod=25 Oct 30 20:16:53.509193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.509096 5091 patch_prober.go:29] interesting pod/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:16:53.509193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.509167 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:16:53.579923 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.579886 5091 generic.go:356] "Generic (PLEG): container finished" podID="f2eb7ef3-adf6-4710-9760-f952a4b003ba" containerID="bec50fb0545d9ba6394926d201138d8769d85499870016561cfb89c403230d5a" exitCode=0 Oct 30 20:16:53.580628 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.579930 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" event={"ID":"f2eb7ef3-adf6-4710-9760-f952a4b003ba","Type":"ContainerDied","Data":"bec50fb0545d9ba6394926d201138d8769d85499870016561cfb89c403230d5a"} Oct 30 20:16:53.589176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.588416 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" event={"ID":"992deac8-a20c-4142-a66c-8f7ca3b3b34d","Type":"ContainerStarted","Data":"b4af77de3804181316ac2573b0da9f04d590547ddb6bd51db98a33f0f68be308"} Oct 30 20:16:53.591661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.591632 5091 generic.go:356] "Generic (PLEG): container finished" podID="5688c73b-8d9e-49d2-88b3-15efc5942caa" containerID="f3933c7fc908700a7eaf5f24c652bf1ac7b16226dcb4ae40f87d78a7d751dc3d" exitCode=0 Oct 30 20:16:53.591799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.591740 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" event={"ID":"5688c73b-8d9e-49d2-88b3-15efc5942caa","Type":"ContainerDied","Data":"f3933c7fc908700a7eaf5f24c652bf1ac7b16226dcb4ae40f87d78a7d751dc3d"} Oct 30 20:16:53.614536 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.614475 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" podStartSLOduration=2.614458147 podStartE2EDuration="2.614458147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:53.612192591 +0000 UTC m=+2733.307400549" watchObservedRunningTime="2026-10-30 20:16:53.614458147 +0000 UTC m=+2733.309666105" Oct 30 20:16:53.781997 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.780676 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:53.790846 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.790794 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:53.797472 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.797443 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:53.903055 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.903024 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-kube-api-access-6pzl6\" (UniqueName: \"kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-kube-api-access-6pzl6\") pod \"5688c73b-8d9e-49d2-88b3-15efc5942caa\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " Oct 30 20:16:53.903248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.903197 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2eb7ef3-adf6-4710-9760-f952a4b003ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2eb7ef3-adf6-4710-9760-f952a4b003ba-plugins-config-volume\") pod \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " Oct 30 20:16:53.903292 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.903246 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-provider-credentials\") pod \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " Oct 30 20:16:53.903292 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.903277 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5688c73b-8d9e-49d2-88b3-15efc5942caa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5688c73b-8d9e-49d2-88b3-15efc5942caa-routing-overlay-volume\") pod \"5688c73b-8d9e-49d2-88b3-15efc5942caa\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " Oct 30 20:16:53.903371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.903347 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2eb7ef3-adf6-4710-9760-f952a4b003ba-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2eb7ef3-adf6-4710-9760-f952a4b003ba-routing-overlay-volume\") pod \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " Oct 30 20:16:53.903434 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.903382 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-kube-api-access-lh9vn\" (UniqueName: \"kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-kube-api-access-lh9vn\") pod \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\" (UID: \"f2eb7ef3-adf6-4710-9760-f952a4b003ba\") " Oct 30 20:16:53.903506 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.903478 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5688c73b-8d9e-49d2-88b3-15efc5942caa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5688c73b-8d9e-49d2-88b3-15efc5942caa-plugins-config-volume\") pod \"5688c73b-8d9e-49d2-88b3-15efc5942caa\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " Oct 30 20:16:53.903603 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.903584 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-provider-credentials\") pod \"5688c73b-8d9e-49d2-88b3-15efc5942caa\" (UID: \"5688c73b-8d9e-49d2-88b3-15efc5942caa\") " Oct 30 20:16:53.903702 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.903674 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2eb7ef3-adf6-4710-9760-f952a4b003ba-plugins-config-volume" pod "f2eb7ef3-adf6-4710-9760-f952a4b003ba" (UID: "f2eb7ef3-adf6-4710-9760-f952a4b003ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:53.903763 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.903723 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5688c73b-8d9e-49d2-88b3-15efc5942caa-routing-overlay-volume" pod "5688c73b-8d9e-49d2-88b3-15efc5942caa" (UID: "5688c73b-8d9e-49d2-88b3-15efc5942caa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:53.903840 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.903818 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2eb7ef3-adf6-4710-9760-f952a4b003ba-routing-overlay-volume" pod "f2eb7ef3-adf6-4710-9760-f952a4b003ba" (UID: "f2eb7ef3-adf6-4710-9760-f952a4b003ba"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:53.903970 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.903947 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5688c73b-8d9e-49d2-88b3-15efc5942caa-plugins-config-volume" pod "5688c73b-8d9e-49d2-88b3-15efc5942caa" (UID: "5688c73b-8d9e-49d2-88b3-15efc5942caa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:53.904424 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.904399 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f2eb7ef3-adf6-4710-9760-f952a4b003ba-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:53.904424 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.904422 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5688c73b-8d9e-49d2-88b3-15efc5942caa-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:53.904623 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.904433 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2eb7ef3-adf6-4710-9760-f952a4b003ba-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:53.904623 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.904444 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5688c73b-8d9e-49d2-88b3-15efc5942caa-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:53.905390 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.905365 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-kube-api-access-6pzl6" pod "5688c73b-8d9e-49d2-88b3-15efc5942caa" (UID: "5688c73b-8d9e-49d2-88b3-15efc5942caa"). InnerVolumeSpecName "kube-api-access-6pzl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:53.905843 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.905825 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-provider-credentials" pod "5688c73b-8d9e-49d2-88b3-15efc5942caa" (UID: "5688c73b-8d9e-49d2-88b3-15efc5942caa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:53.906063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.906046 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-provider-credentials" pod "f2eb7ef3-adf6-4710-9760-f952a4b003ba" (UID: "f2eb7ef3-adf6-4710-9760-f952a4b003ba"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:53.906309 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:53.906292 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-kube-api-access-lh9vn" pod "f2eb7ef3-adf6-4710-9760-f952a4b003ba" (UID: "f2eb7ef3-adf6-4710-9760-f952a4b003ba"). InnerVolumeSpecName "kube-api-access-lh9vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:54.005965 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.005932 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:54.005965 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.005963 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lh9vn\" (UniqueName: \"kubernetes.io/projected/f2eb7ef3-adf6-4710-9760-f952a4b003ba-kube-api-access-lh9vn\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:54.006139 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.005974 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:54.006139 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.005983 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzl6\" (UniqueName: \"kubernetes.io/projected/5688c73b-8d9e-49d2-88b3-15efc5942caa-kube-api-access-6pzl6\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:54.077236 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.056941 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:54.443210 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.443121 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6192cc65-0103-4b75-b51c-3969262634eb" path="/var/lib/kubelet/pods/6192cc65-0103-4b75-b51c-3969262634eb/volumes" Oct 30 20:16:54.601705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.601672 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" Oct 30 20:16:54.602213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.601702 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn" event={"ID":"f2eb7ef3-adf6-4710-9760-f952a4b003ba","Type":"ContainerDied","Data":"a30bce97e7f4cc95272374392e8c0730f6c43caac1736c1d990648099bb8f223"} Oct 30 20:16:54.602213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.601772 5091 scope.go:122] "RemoveContainer" containerID="bec50fb0545d9ba6394926d201138d8769d85499870016561cfb89c403230d5a" Oct 30 20:16:54.603564 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.603540 5091 generic.go:356] "Generic (PLEG): container finished" podID="992deac8-a20c-4142-a66c-8f7ca3b3b34d" containerID="b4af77de3804181316ac2573b0da9f04d590547ddb6bd51db98a33f0f68be308" exitCode=0 Oct 30 20:16:54.603664 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.603625 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" event={"ID":"992deac8-a20c-4142-a66c-8f7ca3b3b34d","Type":"ContainerDied","Data":"b4af77de3804181316ac2573b0da9f04d590547ddb6bd51db98a33f0f68be308"} Oct 30 20:16:54.605433 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.605412 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" event={"ID":"5688c73b-8d9e-49d2-88b3-15efc5942caa","Type":"ContainerDied","Data":"6bb7ed55ea77c50a4c3726a18f0c9920dcf9f6b496755a21b7cde734dd7be772"} Oct 30 20:16:54.605532 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.605433 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks" Oct 30 20:16:54.614434 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.614371 5091 scope.go:122] "RemoveContainer" containerID="f3933c7fc908700a7eaf5f24c652bf1ac7b16226dcb4ae40f87d78a7d751dc3d" Oct 30 20:16:54.630495 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.630473 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn"] Oct 30 20:16:54.637853 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.637833 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-5cd6fdvhn"] Oct 30 20:16:54.682556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.682522 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks"] Oct 30 20:16:54.691348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:54.691323 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-68f8lskks"] Oct 30 20:16:55.197165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:55.196740 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:55.397027 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:55.396990 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:55.793466 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:55.793438 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" Oct 30 20:16:55.951266 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:55.951183 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/992deac8-a20c-4142-a66c-8f7ca3b3b34d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/992deac8-a20c-4142-a66c-8f7ca3b3b34d-maas-api-openshift-ca\") pod \"992deac8-a20c-4142-a66c-8f7ca3b3b34d\" (UID: \"992deac8-a20c-4142-a66c-8f7ca3b3b34d\") " Oct 30 20:16:55.951448 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:55.951364 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992deac8-a20c-4142-a66c-8f7ca3b3b34d-kube-api-access-ckfn2\" (UniqueName: \"kubernetes.io/projected/992deac8-a20c-4142-a66c-8f7ca3b3b34d-kube-api-access-ckfn2\") pod \"992deac8-a20c-4142-a66c-8f7ca3b3b34d\" (UID: \"992deac8-a20c-4142-a66c-8f7ca3b3b34d\") " Oct 30 20:16:55.951547 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:55.951522 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992deac8-a20c-4142-a66c-8f7ca3b3b34d-maas-api-openshift-ca" pod "992deac8-a20c-4142-a66c-8f7ca3b3b34d" (UID: "992deac8-a20c-4142-a66c-8f7ca3b3b34d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:55.951625 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:55.951612 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/992deac8-a20c-4142-a66c-8f7ca3b3b34d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/992deac8-a20c-4142-a66c-8f7ca3b3b34d-maas-api-xks-ca\") pod \"992deac8-a20c-4142-a66c-8f7ca3b3b34d\" (UID: \"992deac8-a20c-4142-a66c-8f7ca3b3b34d\") " Oct 30 20:16:55.952386 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:55.952356 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/992deac8-a20c-4142-a66c-8f7ca3b3b34d-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:55.953723 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:55.953696 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992deac8-a20c-4142-a66c-8f7ca3b3b34d-kube-api-access-ckfn2" pod "992deac8-a20c-4142-a66c-8f7ca3b3b34d" (UID: "992deac8-a20c-4142-a66c-8f7ca3b3b34d"). InnerVolumeSpecName "kube-api-access-ckfn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:55.953846 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:55.953822 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/992deac8-a20c-4142-a66c-8f7ca3b3b34d-maas-api-xks-ca" pod "992deac8-a20c-4142-a66c-8f7ca3b3b34d" (UID: "992deac8-a20c-4142-a66c-8f7ca3b3b34d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:16:56.054811 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.054774 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckfn2\" (UniqueName: \"kubernetes.io/projected/992deac8-a20c-4142-a66c-8f7ca3b3b34d-kube-api-access-ckfn2\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:56.054811 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.054816 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/992deac8-a20c-4142-a66c-8f7ca3b3b34d-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:56.121974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.121939 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" podUID="002df7b8-201d-45a9-be26-076d2b887a7b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:16:56.122185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.121944 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" podUID="d7db6874-2d87-4709-9545-0f29a1532a6c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:16:56.276096 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.270558 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:56.448228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.447137 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5688c73b-8d9e-49d2-88b3-15efc5942caa" path="/var/lib/kubelet/pods/5688c73b-8d9e-49d2-88b3-15efc5942caa/volumes" Oct 30 20:16:56.448975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.448949 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2eb7ef3-adf6-4710-9760-f952a4b003ba" path="/var/lib/kubelet/pods/f2eb7ef3-adf6-4710-9760-f952a4b003ba/volumes" Oct 30 20:16:56.633358 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.633322 5091 generic.go:356] "Generic (PLEG): container finished" podID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerID="e9fbd8ce7aaa5f7b1f3ce5a3ad477d68b5c35005d69c74a8e978c5198cd79ed2" exitCode=0 Oct 30 20:16:56.633534 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.633388 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" event={"ID":"f717fa94-89fa-438f-8a36-8d1a985d7454","Type":"ContainerDied","Data":"e9fbd8ce7aaa5f7b1f3ce5a3ad477d68b5c35005d69c74a8e978c5198cd79ed2"} Oct 30 20:16:56.633534 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.633453 5091 scope.go:122] "RemoveContainer" containerID="e3f42e072853f68097efe15c2d0fafb0d3431177a806b5fa2d3740ef1e5201ea" Oct 30 20:16:56.661031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.660988 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" event={"ID":"992deac8-a20c-4142-a66c-8f7ca3b3b34d","Type":"ContainerDied","Data":"8102429eda178de867c82bfa03183a0e27e02944ffa10a6d61c0ee5de4649bf8"} Oct 30 20:16:56.661203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.661085 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8102429eda178de867c82bfa03183a0e27e02944ffa10a6d61c0ee5de4649bf8" Oct 30 20:16:56.661203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.661180 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn" Oct 30 20:16:56.717776 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.717751 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:16:56.872224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.872131 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-workload-certs\") pod \"f717fa94-89fa-438f-8a36-8d1a985d7454\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " Oct 30 20:16:56.872644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.872240 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-podinfo\") pod \"f717fa94-89fa-438f-8a36-8d1a985d7454\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " Oct 30 20:16:56.872644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.872286 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-envoy\") pod \"f717fa94-89fa-438f-8a36-8d1a985d7454\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " Oct 30 20:16:56.872644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.872332 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-data\") pod \"f717fa94-89fa-438f-8a36-8d1a985d7454\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " Oct 30 20:16:56.872644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.872419 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-credential-socket\") pod \"f717fa94-89fa-438f-8a36-8d1a985d7454\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " Oct 30 20:16:56.872644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.872499 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-token\" (UniqueName: \"kubernetes.io/projected/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-token\") pod \"f717fa94-89fa-438f-8a36-8d1a985d7454\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " Oct 30 20:16:56.872644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.872519 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-workload-certs" pod "f717fa94-89fa-438f-8a36-8d1a985d7454" (UID: "f717fa94-89fa-438f-8a36-8d1a985d7454"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:56.872644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.872618 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f717fa94-89fa-438f-8a36-8d1a985d7454-kube-api-access-4twgv\" (UniqueName: \"kubernetes.io/projected/f717fa94-89fa-438f-8a36-8d1a985d7454-kube-api-access-4twgv\") pod \"f717fa94-89fa-438f-8a36-8d1a985d7454\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " Oct 30 20:16:56.872979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.872669 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f717fa94-89fa-438f-8a36-8d1a985d7454-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f717fa94-89fa-438f-8a36-8d1a985d7454-istiod-ca-cert\") pod \"f717fa94-89fa-438f-8a36-8d1a985d7454\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " Oct 30 20:16:56.872979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.872906 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-workload-socket\") pod \"f717fa94-89fa-438f-8a36-8d1a985d7454\" (UID: \"f717fa94-89fa-438f-8a36-8d1a985d7454\") " Oct 30 20:16:56.873747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.873712 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-data" pod "f717fa94-89fa-438f-8a36-8d1a985d7454" (UID: "f717fa94-89fa-438f-8a36-8d1a985d7454"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:56.874212 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.874120 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-credential-socket" pod "f717fa94-89fa-438f-8a36-8d1a985d7454" (UID: "f717fa94-89fa-438f-8a36-8d1a985d7454"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:56.874385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.874220 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-workload-socket" pod "f717fa94-89fa-438f-8a36-8d1a985d7454" (UID: "f717fa94-89fa-438f-8a36-8d1a985d7454"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:56.874576 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.874388 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f717fa94-89fa-438f-8a36-8d1a985d7454-istiod-ca-cert" pod "f717fa94-89fa-438f-8a36-8d1a985d7454" (UID: "f717fa94-89fa-438f-8a36-8d1a985d7454"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:16:56.874891 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.874870 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:56.875075 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.875057 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f717fa94-89fa-438f-8a36-8d1a985d7454-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:56.875282 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.875266 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:56.875435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.875421 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:56.875583 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.875569 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:56.876801 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.876737 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-envoy" pod "f717fa94-89fa-438f-8a36-8d1a985d7454" (UID: "f717fa94-89fa-438f-8a36-8d1a985d7454"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:16:56.877449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.877385 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f717fa94-89fa-438f-8a36-8d1a985d7454-kube-api-access-4twgv" pod "f717fa94-89fa-438f-8a36-8d1a985d7454" (UID: "f717fa94-89fa-438f-8a36-8d1a985d7454"). InnerVolumeSpecName "kube-api-access-4twgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:56.877922 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.877882 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-token" pod "f717fa94-89fa-438f-8a36-8d1a985d7454" (UID: "f717fa94-89fa-438f-8a36-8d1a985d7454"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:16:56.878355 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.878324 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-podinfo" pod "f717fa94-89fa-438f-8a36-8d1a985d7454" (UID: "f717fa94-89fa-438f-8a36-8d1a985d7454"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:16:56.976995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.976954 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:56.976995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.976990 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:56.976995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.977003 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4twgv\" (UniqueName: \"kubernetes.io/projected/f717fa94-89fa-438f-8a36-8d1a985d7454-kube-api-access-4twgv\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:56.977258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:56.977014 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f717fa94-89fa-438f-8a36-8d1a985d7454-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:16:57.243821 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:57.243738 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:57.677187 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:57.673570 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" Oct 30 20:16:57.677187 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:57.673603 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph" event={"ID":"f717fa94-89fa-438f-8a36-8d1a985d7454","Type":"ContainerDied","Data":"35bf66b69697e1c2ce65bbb989f08a2633e74bfa8a6d493db38a98c662f33eb2"} Oct 30 20:16:57.677187 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:57.673668 5091 scope.go:122] "RemoveContainer" containerID="e9fbd8ce7aaa5f7b1f3ce5a3ad477d68b5c35005d69c74a8e978c5198cd79ed2" Oct 30 20:16:57.707637 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:57.707608 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph"] Oct 30 20:16:57.711854 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:57.711830 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-969878-openshift-default-74f689c897-bzsph"] Oct 30 20:16:58.275171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.271648 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:58.334468 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.334433 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf"] Oct 30 20:16:58.335495 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335468 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f2eb7ef3-adf6-4710-9760-f952a4b003ba" containerName="payload-processing" Oct 30 20:16:58.335495 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335494 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2eb7ef3-adf6-4710-9760-f952a4b003ba" containerName="payload-processing" Oct 30 20:16:58.335714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335542 5091 cpu_manager.go:414] "containerMap: removing container" podUID="992deac8-a20c-4142-a66c-8f7ca3b3b34d" containerName="revoke-keys" Oct 30 20:16:58.335714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335551 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992deac8-a20c-4142-a66c-8f7ca3b3b34d" containerName="revoke-keys" Oct 30 20:16:58.335714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335566 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerName="istio-proxy" Oct 30 20:16:58.335714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335574 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerName="istio-proxy" Oct 30 20:16:58.335714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335591 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerName="istio-proxy" Oct 30 20:16:58.335714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335599 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerName="istio-proxy" Oct 30 20:16:58.335714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335630 5091 cpu_manager.go:414] "containerMap: removing container" podUID="1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd" containerName="revoke-keys" Oct 30 20:16:58.335714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335641 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd" containerName="revoke-keys" Oct 30 20:16:58.335714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335682 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5688c73b-8d9e-49d2-88b3-15efc5942caa" containerName="payload-processing" Oct 30 20:16:58.335714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335691 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5688c73b-8d9e-49d2-88b3-15efc5942caa" containerName="payload-processing" Oct 30 20:16:58.336287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335874 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2eb7ef3-adf6-4710-9760-f952a4b003ba" containerName="payload-processing" Oct 30 20:16:58.336287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335892 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="992deac8-a20c-4142-a66c-8f7ca3b3b34d" containerName="revoke-keys" Oct 30 20:16:58.336287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335903 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerName="istio-proxy" Oct 30 20:16:58.336287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335914 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" containerName="istio-proxy" Oct 30 20:16:58.336287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335928 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd" containerName="revoke-keys" Oct 30 20:16:58.336287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.335943 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5688c73b-8d9e-49d2-88b3-15efc5942caa" containerName="payload-processing" Oct 30 20:16:58.342697 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.342674 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf"] Oct 30 20:16:58.342825 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.342792 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" Oct 30 20:16:58.357305 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.357278 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:16:58.357917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.357894 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:16:58.403109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.403077 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865fe459-6cb9-40c2-9afd-835e90619526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2235aa4a-d2zsf\" (UID: \"865fe459-6cb9-40c2-9afd-835e90619526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" Oct 30 20:16:58.403260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.403193 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnf7d\" (UniqueName: \"kubernetes.io/projected/865fe459-6cb9-40c2-9afd-835e90619526-kube-api-access-lnf7d\") pod \"maas-api-revoke-sub-2235aa4a-d2zsf\" (UID: \"865fe459-6cb9-40c2-9afd-835e90619526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" Oct 30 20:16:58.403314 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.403266 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865fe459-6cb9-40c2-9afd-835e90619526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2235aa4a-d2zsf\" (UID: \"865fe459-6cb9-40c2-9afd-835e90619526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" Oct 30 20:16:58.444784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.444757 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f717fa94-89fa-438f-8a36-8d1a985d7454" path="/var/lib/kubelet/pods/f717fa94-89fa-438f-8a36-8d1a985d7454/volumes" Oct 30 20:16:58.504799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.504770 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865fe459-6cb9-40c2-9afd-835e90619526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2235aa4a-d2zsf\" (UID: \"865fe459-6cb9-40c2-9afd-835e90619526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" Oct 30 20:16:58.504955 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.504896 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnf7d\" (UniqueName: \"kubernetes.io/projected/865fe459-6cb9-40c2-9afd-835e90619526-kube-api-access-lnf7d\") pod \"maas-api-revoke-sub-2235aa4a-d2zsf\" (UID: \"865fe459-6cb9-40c2-9afd-835e90619526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" Oct 30 20:16:58.505050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.505034 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865fe459-6cb9-40c2-9afd-835e90619526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2235aa4a-d2zsf\" (UID: \"865fe459-6cb9-40c2-9afd-835e90619526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" Oct 30 20:16:58.505704 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.505680 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865fe459-6cb9-40c2-9afd-835e90619526-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2235aa4a-d2zsf\" (UID: \"865fe459-6cb9-40c2-9afd-835e90619526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" Oct 30 20:16:58.507301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.507281 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865fe459-6cb9-40c2-9afd-835e90619526-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2235aa4a-d2zsf\" (UID: \"865fe459-6cb9-40c2-9afd-835e90619526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" Oct 30 20:16:58.519509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.519481 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnf7d\" (UniqueName: \"kubernetes.io/projected/865fe459-6cb9-40c2-9afd-835e90619526-kube-api-access-lnf7d\") pod \"maas-api-revoke-sub-2235aa4a-d2zsf\" (UID: \"865fe459-6cb9-40c2-9afd-835e90619526\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" Oct 30 20:16:58.679232 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.679130 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" Oct 30 20:16:58.978196 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:16:58.977289 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865fe459_6cb9_40c2_9afd_835e90619526.slice/crio-296edc5c264f6f7bdce314a02a2d89e3f7e34a029bb2a130874c2b4b11939999 WatchSource:0}: Error finding container 296edc5c264f6f7bdce314a02a2d89e3f7e34a029bb2a130874c2b4b11939999: Status 404 returned error can't find the container with id 296edc5c264f6f7bdce314a02a2d89e3f7e34a029bb2a130874c2b4b11939999 Oct 30 20:16:58.978196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:58.977831 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf"] Oct 30 20:16:59.272526 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:59.272490 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:16:59.711670 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:59.711567 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" event={"ID":"865fe459-6cb9-40c2-9afd-835e90619526","Type":"ContainerStarted","Data":"9096330f968af83c7bf98816c28cca87083a87f15470a377fb048578aae5969e"} Oct 30 20:16:59.711670 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:59.711633 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" event={"ID":"865fe459-6cb9-40c2-9afd-835e90619526","Type":"ContainerStarted","Data":"296edc5c264f6f7bdce314a02a2d89e3f7e34a029bb2a130874c2b4b11939999"} Oct 30 20:16:59.736048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:16:59.735997 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" podStartSLOduration=1.73598524 podStartE2EDuration="1.73598524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:16:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:16:59.734602037 +0000 UTC m=+2739.429809996" watchObservedRunningTime="2026-10-30 20:16:59.73598524 +0000 UTC m=+2739.431193197" Oct 30 20:17:00.091586 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:00.091550 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" podUID="d973dcd0-1e61-4929-a1a0-648efb94577c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:17:00.361019 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:00.360916 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:00.602287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:00.602248 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:17:00.627093 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:00.626965 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:00.676576 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:00.676540 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk"] Oct 30 20:17:00.727314 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:00.727274 5091 generic.go:356] "Generic (PLEG): container finished" podID="865fe459-6cb9-40c2-9afd-835e90619526" containerID="9096330f968af83c7bf98816c28cca87083a87f15470a377fb048578aae5969e" exitCode=0 Oct 30 20:17:00.727700 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:00.727382 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" event={"ID":"865fe459-6cb9-40c2-9afd-835e90619526","Type":"ContainerDied","Data":"9096330f968af83c7bf98816c28cca87083a87f15470a377fb048578aae5969e"} Oct 30 20:17:00.785546 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:00.785505 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:00.883189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:00.877180 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8"] Oct 30 20:17:00.883189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:00.877438 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" podUID="071ef2ea-f7f5-4e12-9287-73c4f832f361" containerName="istio-proxy" containerID="cri-o://20f0990421ab107f3462fb007c9b5566560bf8a94baf2f6b82c2fc88e9d97838" gracePeriod=30 Oct 30 20:17:00.947648 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:00.947588 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:01.114169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.113821 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls"] Oct 30 20:17:01.161221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.159459 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59ce6a42-7jjls"] Oct 30 20:17:01.435963 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.435885 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk"] Oct 30 20:17:01.440710 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.440674 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" Oct 30 20:17:01.448408 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.448379 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk"] Oct 30 20:17:01.521609 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.521550 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk\" (UID: \"54ae8ff0-8ea1-4133-99a1-908bbfa06df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" Oct 30 20:17:01.521833 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.521809 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt7js\" (UniqueName: \"kubernetes.io/projected/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-kube-api-access-dt7js\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk\" (UID: \"54ae8ff0-8ea1-4133-99a1-908bbfa06df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" Oct 30 20:17:01.521951 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.521924 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk\" (UID: \"54ae8ff0-8ea1-4133-99a1-908bbfa06df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" Oct 30 20:17:01.577248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.573750 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:01.581898 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.580363 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:17:01.623640 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.623602 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk\" (UID: \"54ae8ff0-8ea1-4133-99a1-908bbfa06df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" Oct 30 20:17:01.623832 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.623802 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk\" (UID: \"54ae8ff0-8ea1-4133-99a1-908bbfa06df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" Oct 30 20:17:01.624030 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.624007 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt7js\" (UniqueName: \"kubernetes.io/projected/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-kube-api-access-dt7js\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk\" (UID: \"54ae8ff0-8ea1-4133-99a1-908bbfa06df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" Oct 30 20:17:01.624517 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.624495 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk\" (UID: \"54ae8ff0-8ea1-4133-99a1-908bbfa06df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" Oct 30 20:17:01.626346 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.626323 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk\" (UID: \"54ae8ff0-8ea1-4133-99a1-908bbfa06df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" Oct 30 20:17:01.640681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.640655 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt7js\" (UniqueName: \"kubernetes.io/projected/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-kube-api-access-dt7js\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk\" (UID: \"54ae8ff0-8ea1-4133-99a1-908bbfa06df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" Oct 30 20:17:01.679089 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.679050 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2"] Oct 30 20:17:01.768582 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.768542 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" Oct 30 20:17:01.993654 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:01.993586 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" Oct 30 20:17:02.092482 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:17:02.092437 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ae8ff0_8ea1_4133_99a1_908bbfa06df3.slice/crio-758c0d2fd739d9b8f832fbb985340728478332739128a6132a433f7db04b4e6d WatchSource:0}: Error finding container 758c0d2fd739d9b8f832fbb985340728478332739128a6132a433f7db04b4e6d: Status 404 returned error can't find the container with id 758c0d2fd739d9b8f832fbb985340728478332739128a6132a433f7db04b4e6d Oct 30 20:17:02.093849 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.093820 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk"] Oct 30 20:17:02.132315 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.132283 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/865fe459-6cb9-40c2-9afd-835e90619526-kube-api-access-lnf7d\" (UniqueName: \"kubernetes.io/projected/865fe459-6cb9-40c2-9afd-835e90619526-kube-api-access-lnf7d\") pod \"865fe459-6cb9-40c2-9afd-835e90619526\" (UID: \"865fe459-6cb9-40c2-9afd-835e90619526\") " Oct 30 20:17:02.132502 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.132464 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/865fe459-6cb9-40c2-9afd-835e90619526-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865fe459-6cb9-40c2-9afd-835e90619526-maas-api-xks-ca\") pod \"865fe459-6cb9-40c2-9afd-835e90619526\" (UID: \"865fe459-6cb9-40c2-9afd-835e90619526\") " Oct 30 20:17:02.132620 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.132590 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/865fe459-6cb9-40c2-9afd-835e90619526-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865fe459-6cb9-40c2-9afd-835e90619526-maas-api-openshift-ca\") pod \"865fe459-6cb9-40c2-9afd-835e90619526\" (UID: \"865fe459-6cb9-40c2-9afd-835e90619526\") " Oct 30 20:17:02.132972 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.132947 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/865fe459-6cb9-40c2-9afd-835e90619526-maas-api-openshift-ca" pod "865fe459-6cb9-40c2-9afd-835e90619526" (UID: "865fe459-6cb9-40c2-9afd-835e90619526"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:02.133477 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.133453 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/865fe459-6cb9-40c2-9afd-835e90619526-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:02.134563 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.134536 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/865fe459-6cb9-40c2-9afd-835e90619526-kube-api-access-lnf7d" pod "865fe459-6cb9-40c2-9afd-835e90619526" (UID: "865fe459-6cb9-40c2-9afd-835e90619526"). InnerVolumeSpecName "kube-api-access-lnf7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:02.134653 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.134608 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/865fe459-6cb9-40c2-9afd-835e90619526-maas-api-xks-ca" pod "865fe459-6cb9-40c2-9afd-835e90619526" (UID: "865fe459-6cb9-40c2-9afd-835e90619526"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:02.235286 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.235245 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnf7d\" (UniqueName: \"kubernetes.io/projected/865fe459-6cb9-40c2-9afd-835e90619526-kube-api-access-lnf7d\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:02.235286 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.235288 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/865fe459-6cb9-40c2-9afd-835e90619526-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:02.264636 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.264595 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:02.410577 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.410531 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:02.554169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.539308 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e42c2d6-2cfc-4012-9c93-1a612809a5fb" path="/var/lib/kubelet/pods/0e42c2d6-2cfc-4012-9c93-1a612809a5fb/volumes" Oct 30 20:17:02.762667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.758803 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" event={"ID":"865fe459-6cb9-40c2-9afd-835e90619526","Type":"ContainerDied","Data":"296edc5c264f6f7bdce314a02a2d89e3f7e34a029bb2a130874c2b4b11939999"} Oct 30 20:17:02.762667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.758867 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="296edc5c264f6f7bdce314a02a2d89e3f7e34a029bb2a130874c2b4b11939999" Oct 30 20:17:02.762667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.758935 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf" Oct 30 20:17:02.762667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.762361 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" event={"ID":"54ae8ff0-8ea1-4133-99a1-908bbfa06df3","Type":"ContainerStarted","Data":"1b19fac93a0417620496709cf5b39bf2270766db5089abbe9e82e37a8382a404"} Oct 30 20:17:02.762667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.762402 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" event={"ID":"54ae8ff0-8ea1-4133-99a1-908bbfa06df3","Type":"ContainerStarted","Data":"758c0d2fd739d9b8f832fbb985340728478332739128a6132a433f7db04b4e6d"} Oct 30 20:17:02.769464 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.768783 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:02.793389 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:02.793298 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" podStartSLOduration=1.793285076 podStartE2EDuration="1.793285076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:02.792646142 +0000 UTC m=+2742.487854101" watchObservedRunningTime="2026-10-30 20:17:02.793285076 +0000 UTC m=+2742.488493033" Oct 30 20:17:03.597169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:03.595959 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:03.772244 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:03.772182 5091 generic.go:356] "Generic (PLEG): container finished" podID="54ae8ff0-8ea1-4133-99a1-908bbfa06df3" containerID="1b19fac93a0417620496709cf5b39bf2270766db5089abbe9e82e37a8382a404" exitCode=0 Oct 30 20:17:03.772685 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:03.772271 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" event={"ID":"54ae8ff0-8ea1-4133-99a1-908bbfa06df3","Type":"ContainerDied","Data":"1b19fac93a0417620496709cf5b39bf2270766db5089abbe9e82e37a8382a404"} Oct 30 20:17:04.477171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:04.476961 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:04.965998 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:04.965970 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" Oct 30 20:17:05.120200 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.120104 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-kube-api-access-dt7js\" (UniqueName: \"kubernetes.io/projected/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-kube-api-access-dt7js\") pod \"54ae8ff0-8ea1-4133-99a1-908bbfa06df3\" (UID: \"54ae8ff0-8ea1-4133-99a1-908bbfa06df3\") " Oct 30 20:17:05.120372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.120212 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-maas-api-xks-ca\") pod \"54ae8ff0-8ea1-4133-99a1-908bbfa06df3\" (UID: \"54ae8ff0-8ea1-4133-99a1-908bbfa06df3\") " Oct 30 20:17:05.120372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.120344 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-maas-api-openshift-ca\") pod \"54ae8ff0-8ea1-4133-99a1-908bbfa06df3\" (UID: \"54ae8ff0-8ea1-4133-99a1-908bbfa06df3\") " Oct 30 20:17:05.120695 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.120666 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-maas-api-openshift-ca" pod "54ae8ff0-8ea1-4133-99a1-908bbfa06df3" (UID: "54ae8ff0-8ea1-4133-99a1-908bbfa06df3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:05.121051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.121032 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:05.122464 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.122437 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-maas-api-xks-ca" pod "54ae8ff0-8ea1-4133-99a1-908bbfa06df3" (UID: "54ae8ff0-8ea1-4133-99a1-908bbfa06df3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:05.122464 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.122446 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-kube-api-access-dt7js" pod "54ae8ff0-8ea1-4133-99a1-908bbfa06df3" (UID: "54ae8ff0-8ea1-4133-99a1-908bbfa06df3"). InnerVolumeSpecName "kube-api-access-dt7js". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:05.222569 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.222535 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt7js\" (UniqueName: \"kubernetes.io/projected/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-kube-api-access-dt7js\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:05.222569 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.222567 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54ae8ff0-8ea1-4133-99a1-908bbfa06df3-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:05.382228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.379261 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:05.677882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.677794 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" podUID="0ec79660-754a-446d-846c-9632c0b6b2e6" containerName="payload-processing" containerID="cri-o://defaf3f23cd5a02404442104f3e76a6067f2d74dadbdb9d4bb256f6005ca8542" gracePeriod=25 Oct 30 20:17:05.798675 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.798639 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" Oct 30 20:17:05.798817 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.798675 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk" event={"ID":"54ae8ff0-8ea1-4133-99a1-908bbfa06df3","Type":"ContainerDied","Data":"758c0d2fd739d9b8f832fbb985340728478332739128a6132a433f7db04b4e6d"} Oct 30 20:17:05.798817 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.798731 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="758c0d2fd739d9b8f832fbb985340728478332739128a6132a433f7db04b4e6d" Oct 30 20:17:05.800684 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.800659 5091 generic.go:356] "Generic (PLEG): container finished" podID="0ec79660-754a-446d-846c-9632c0b6b2e6" containerID="defaf3f23cd5a02404442104f3e76a6067f2d74dadbdb9d4bb256f6005ca8542" exitCode=0 Oct 30 20:17:05.800802 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:05.800731 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" event={"ID":"0ec79660-754a-446d-846c-9632c0b6b2e6","Type":"ContainerDied","Data":"defaf3f23cd5a02404442104f3e76a6067f2d74dadbdb9d4bb256f6005ca8542"} Oct 30 20:17:06.037504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.037470 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:17:06.122141 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.122108 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" podUID="d7db6874-2d87-4709-9545-0f29a1532a6c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:17:06.123754 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.123719 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" podUID="002df7b8-201d-45a9-be26-076d2b887a7b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:17:06.138253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.138228 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec79660-754a-446d-846c-9632c0b6b2e6-kube-api-access-6z7rs\" (UniqueName: \"kubernetes.io/projected/0ec79660-754a-446d-846c-9632c0b6b2e6-kube-api-access-6z7rs\") pod \"0ec79660-754a-446d-846c-9632c0b6b2e6\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " Oct 30 20:17:06.138496 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.138473 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec79660-754a-446d-846c-9632c0b6b2e6-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ec79660-754a-446d-846c-9632c0b6b2e6-routing-overlay-volume\") pod \"0ec79660-754a-446d-846c-9632c0b6b2e6\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " Oct 30 20:17:06.138722 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.138700 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ec79660-754a-446d-846c-9632c0b6b2e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec79660-754a-446d-846c-9632c0b6b2e6-plugins-config-volume\") pod \"0ec79660-754a-446d-846c-9632c0b6b2e6\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " Oct 30 20:17:06.138834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.138756 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ec79660-754a-446d-846c-9632c0b6b2e6-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ec79660-754a-446d-846c-9632c0b6b2e6-provider-credentials\") pod \"0ec79660-754a-446d-846c-9632c0b6b2e6\" (UID: \"0ec79660-754a-446d-846c-9632c0b6b2e6\") " Oct 30 20:17:06.139906 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.139875 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec79660-754a-446d-846c-9632c0b6b2e6-routing-overlay-volume" pod "0ec79660-754a-446d-846c-9632c0b6b2e6" (UID: "0ec79660-754a-446d-846c-9632c0b6b2e6"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:06.140050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.139880 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ec79660-754a-446d-846c-9632c0b6b2e6-plugins-config-volume" pod "0ec79660-754a-446d-846c-9632c0b6b2e6" (UID: "0ec79660-754a-446d-846c-9632c0b6b2e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:06.142792 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.142753 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec79660-754a-446d-846c-9632c0b6b2e6-kube-api-access-6z7rs" pod "0ec79660-754a-446d-846c-9632c0b6b2e6" (UID: "0ec79660-754a-446d-846c-9632c0b6b2e6"). InnerVolumeSpecName "kube-api-access-6z7rs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:06.143039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.143014 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ec79660-754a-446d-846c-9632c0b6b2e6-provider-credentials" pod "0ec79660-754a-446d-846c-9632c0b6b2e6" (UID: "0ec79660-754a-446d-846c-9632c0b6b2e6"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:06.186951 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.186911 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:06.220752 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.220361 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:17:06.241382 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.241283 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-token\" (UniqueName: \"kubernetes.io/projected/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-token\") pod \"071ef2ea-f7f5-4e12-9287-73c4f832f361\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " Oct 30 20:17:06.241382 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.241345 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-data\") pod \"071ef2ea-f7f5-4e12-9287-73c4f832f361\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " Oct 30 20:17:06.241631 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.241384 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-podinfo\") pod \"071ef2ea-f7f5-4e12-9287-73c4f832f361\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " Oct 30 20:17:06.241631 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.241416 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/071ef2ea-f7f5-4e12-9287-73c4f832f361-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/071ef2ea-f7f5-4e12-9287-73c4f832f361-istiod-ca-cert\") pod \"071ef2ea-f7f5-4e12-9287-73c4f832f361\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " Oct 30 20:17:06.241631 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.241447 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/071ef2ea-f7f5-4e12-9287-73c4f832f361-kube-api-access-78cdc\" (UniqueName: \"kubernetes.io/projected/071ef2ea-f7f5-4e12-9287-73c4f832f361-kube-api-access-78cdc\") pod \"071ef2ea-f7f5-4e12-9287-73c4f832f361\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " Oct 30 20:17:06.241631 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.241604 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-workload-certs\") pod \"071ef2ea-f7f5-4e12-9287-73c4f832f361\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " Oct 30 20:17:06.241878 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.241720 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-workload-socket\") pod \"071ef2ea-f7f5-4e12-9287-73c4f832f361\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " Oct 30 20:17:06.241878 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.241760 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-envoy\") pod \"071ef2ea-f7f5-4e12-9287-73c4f832f361\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " Oct 30 20:17:06.242032 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.242012 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-credential-socket\") pod \"071ef2ea-f7f5-4e12-9287-73c4f832f361\" (UID: \"071ef2ea-f7f5-4e12-9287-73c4f832f361\") " Oct 30 20:17:06.242778 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.242750 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z7rs\" (UniqueName: \"kubernetes.io/projected/0ec79660-754a-446d-846c-9632c0b6b2e6-kube-api-access-6z7rs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:06.242893 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.242786 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0ec79660-754a-446d-846c-9632c0b6b2e6-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:06.242893 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.242802 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ec79660-754a-446d-846c-9632c0b6b2e6-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:06.242893 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.242816 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0ec79660-754a-446d-846c-9632c0b6b2e6-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:06.243553 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.243424 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-workload-socket" pod "071ef2ea-f7f5-4e12-9287-73c4f832f361" (UID: "071ef2ea-f7f5-4e12-9287-73c4f832f361"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:06.243553 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.243444 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-workload-certs" pod "071ef2ea-f7f5-4e12-9287-73c4f832f361" (UID: "071ef2ea-f7f5-4e12-9287-73c4f832f361"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:06.243553 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.243498 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/071ef2ea-f7f5-4e12-9287-73c4f832f361-istiod-ca-cert" pod "071ef2ea-f7f5-4e12-9287-73c4f832f361" (UID: "071ef2ea-f7f5-4e12-9287-73c4f832f361"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:06.243795 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.243603 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-data" pod "071ef2ea-f7f5-4e12-9287-73c4f832f361" (UID: "071ef2ea-f7f5-4e12-9287-73c4f832f361"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:06.244113 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.243869 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-credential-socket" pod "071ef2ea-f7f5-4e12-9287-73c4f832f361" (UID: "071ef2ea-f7f5-4e12-9287-73c4f832f361"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:06.281175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.281090 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-token" pod "071ef2ea-f7f5-4e12-9287-73c4f832f361" (UID: "071ef2ea-f7f5-4e12-9287-73c4f832f361"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:06.294739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.294570 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/071ef2ea-f7f5-4e12-9287-73c4f832f361-kube-api-access-78cdc" pod "071ef2ea-f7f5-4e12-9287-73c4f832f361" (UID: "071ef2ea-f7f5-4e12-9287-73c4f832f361"). InnerVolumeSpecName "kube-api-access-78cdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:06.294739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.294691 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-podinfo" pod "071ef2ea-f7f5-4e12-9287-73c4f832f361" (UID: "071ef2ea-f7f5-4e12-9287-73c4f832f361"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:17:06.297196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.296696 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-envoy" pod "071ef2ea-f7f5-4e12-9287-73c4f832f361" (UID: "071ef2ea-f7f5-4e12-9287-73c4f832f361"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:06.412417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.410757 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:06.412417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.412242 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:06.412417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.412272 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:06.412417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.412288 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:06.412417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.412302 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:06.412417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.412315 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/071ef2ea-f7f5-4e12-9287-73c4f832f361-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:06.412417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.412331 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78cdc\" (UniqueName: \"kubernetes.io/projected/071ef2ea-f7f5-4e12-9287-73c4f832f361-kube-api-access-78cdc\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:06.412417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.412346 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:06.412417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.412361 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:06.412417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.412375 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/071ef2ea-f7f5-4e12-9287-73c4f832f361-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:06.602056 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.602005 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:06.680661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.680490 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" podUID="04dd38d4-f223-4b01-8a58-ac3ef092a862" containerName="payload-processing" containerID="cri-o://2dc3f7c6ca81099eaa48c5e29a800a597e0562dd9518ccb0234062ee0a2d8e3f" gracePeriod=25 Oct 30 20:17:06.701692 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.701661 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz"] Oct 30 20:17:06.701885 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.701871 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-b7431c-89wmz"] Oct 30 20:17:06.724437 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.724409 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" Oct 30 20:17:06.813330 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.813294 5091 generic.go:356] "Generic (PLEG): container finished" podID="071ef2ea-f7f5-4e12-9287-73c4f832f361" containerID="20f0990421ab107f3462fb007c9b5566560bf8a94baf2f6b82c2fc88e9d97838" exitCode=0 Oct 30 20:17:06.813507 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.813361 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" Oct 30 20:17:06.813507 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.813369 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" event={"ID":"071ef2ea-f7f5-4e12-9287-73c4f832f361","Type":"ContainerDied","Data":"20f0990421ab107f3462fb007c9b5566560bf8a94baf2f6b82c2fc88e9d97838"} Oct 30 20:17:06.813507 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.813437 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8" event={"ID":"071ef2ea-f7f5-4e12-9287-73c4f832f361","Type":"ContainerDied","Data":"12869bdfd0413f14d66bde01a4e1554a77e5ca326620dc033ef7e066a3b7e417"} Oct 30 20:17:06.813507 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.813461 5091 scope.go:122] "RemoveContainer" containerID="20f0990421ab107f3462fb007c9b5566560bf8a94baf2f6b82c2fc88e9d97838" Oct 30 20:17:06.815454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.815346 5091 generic.go:356] "Generic (PLEG): container finished" podID="d973dcd0-1e61-4929-a1a0-648efb94577c" containerID="d62a88067ef970155a179d2d94cbde6d036b01a624d783e8292398fe3a802857" exitCode=0 Oct 30 20:17:06.815454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.815388 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" event={"ID":"d973dcd0-1e61-4929-a1a0-648efb94577c","Type":"ContainerDied","Data":"d62a88067ef970155a179d2d94cbde6d036b01a624d783e8292398fe3a802857"} Oct 30 20:17:06.815454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.815429 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" event={"ID":"d973dcd0-1e61-4929-a1a0-648efb94577c","Type":"ContainerDied","Data":"f8d01ad2283370c8ab418687c37138a5ffdc6939609d912524434979a7ccc728"} Oct 30 20:17:06.815702 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.815479 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4" Oct 30 20:17:06.818342 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.818317 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" event={"ID":"0ec79660-754a-446d-846c-9632c0b6b2e6","Type":"ContainerDied","Data":"88f840b273409095031d996cd1c1756f7478de0f0bd98bfbc70955b6581e349f"} Oct 30 20:17:06.818342 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.818335 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk" Oct 30 20:17:06.826233 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.826214 5091 scope.go:122] "RemoveContainer" containerID="f2f279b992166019e79e7a0bfa6990bdbd4044a1aa3acfcf808ae8cbb196ec17" Oct 30 20:17:06.828817 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.828792 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d973dcd0-1e61-4929-a1a0-648efb94577c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d973dcd0-1e61-4929-a1a0-648efb94577c-plugins-config-volume\") pod \"d973dcd0-1e61-4929-a1a0-648efb94577c\" (UID: \"d973dcd0-1e61-4929-a1a0-648efb94577c\") " Oct 30 20:17:06.829038 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.829021 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d973dcd0-1e61-4929-a1a0-648efb94577c-kube-api-access-4hw2x\" (UniqueName: \"kubernetes.io/projected/d973dcd0-1e61-4929-a1a0-648efb94577c-kube-api-access-4hw2x\") pod \"d973dcd0-1e61-4929-a1a0-648efb94577c\" (UID: \"d973dcd0-1e61-4929-a1a0-648efb94577c\") " Oct 30 20:17:06.829240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.829215 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d973dcd0-1e61-4929-a1a0-648efb94577c-plugins-config-volume" pod "d973dcd0-1e61-4929-a1a0-648efb94577c" (UID: "d973dcd0-1e61-4929-a1a0-648efb94577c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:06.829676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.829651 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d973dcd0-1e61-4929-a1a0-648efb94577c-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:06.831277 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.831247 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d973dcd0-1e61-4929-a1a0-648efb94577c-kube-api-access-4hw2x" pod "d973dcd0-1e61-4929-a1a0-648efb94577c" (UID: "d973dcd0-1e61-4929-a1a0-648efb94577c"). InnerVolumeSpecName "kube-api-access-4hw2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:06.841911 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.841887 5091 scope.go:122] "RemoveContainer" containerID="20f0990421ab107f3462fb007c9b5566560bf8a94baf2f6b82c2fc88e9d97838" Oct 30 20:17:06.843741 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:06.843709 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f0990421ab107f3462fb007c9b5566560bf8a94baf2f6b82c2fc88e9d97838\": container with ID starting with 20f0990421ab107f3462fb007c9b5566560bf8a94baf2f6b82c2fc88e9d97838 not found: ID does not exist" containerID="20f0990421ab107f3462fb007c9b5566560bf8a94baf2f6b82c2fc88e9d97838" Oct 30 20:17:06.843851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.843752 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f0990421ab107f3462fb007c9b5566560bf8a94baf2f6b82c2fc88e9d97838"} err="failed to get container status \"20f0990421ab107f3462fb007c9b5566560bf8a94baf2f6b82c2fc88e9d97838\": rpc error: code = NotFound desc = could not find container \"20f0990421ab107f3462fb007c9b5566560bf8a94baf2f6b82c2fc88e9d97838\": container with ID starting with 20f0990421ab107f3462fb007c9b5566560bf8a94baf2f6b82c2fc88e9d97838 not found: ID does not exist" Oct 30 20:17:06.843851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.843779 5091 scope.go:122] "RemoveContainer" containerID="f2f279b992166019e79e7a0bfa6990bdbd4044a1aa3acfcf808ae8cbb196ec17" Oct 30 20:17:06.844112 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:06.844088 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f279b992166019e79e7a0bfa6990bdbd4044a1aa3acfcf808ae8cbb196ec17\": container with ID starting with f2f279b992166019e79e7a0bfa6990bdbd4044a1aa3acfcf808ae8cbb196ec17 not found: ID does not exist" containerID="f2f279b992166019e79e7a0bfa6990bdbd4044a1aa3acfcf808ae8cbb196ec17" Oct 30 20:17:06.844214 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.844122 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f279b992166019e79e7a0bfa6990bdbd4044a1aa3acfcf808ae8cbb196ec17"} err="failed to get container status \"f2f279b992166019e79e7a0bfa6990bdbd4044a1aa3acfcf808ae8cbb196ec17\": rpc error: code = NotFound desc = could not find container \"f2f279b992166019e79e7a0bfa6990bdbd4044a1aa3acfcf808ae8cbb196ec17\": container with ID starting with f2f279b992166019e79e7a0bfa6990bdbd4044a1aa3acfcf808ae8cbb196ec17 not found: ID does not exist" Oct 30 20:17:06.844214 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.844181 5091 scope.go:122] "RemoveContainer" containerID="d62a88067ef970155a179d2d94cbde6d036b01a624d783e8292398fe3a802857" Oct 30 20:17:06.848021 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.848000 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8"] Oct 30 20:17:06.853117 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.853066 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d97cc8-openshift-default-665557959c-796d8"] Oct 30 20:17:06.859955 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.859934 5091 scope.go:122] "RemoveContainer" containerID="d62a88067ef970155a179d2d94cbde6d036b01a624d783e8292398fe3a802857" Oct 30 20:17:06.860308 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:06.860285 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d62a88067ef970155a179d2d94cbde6d036b01a624d783e8292398fe3a802857\": container with ID starting with d62a88067ef970155a179d2d94cbde6d036b01a624d783e8292398fe3a802857 not found: ID does not exist" containerID="d62a88067ef970155a179d2d94cbde6d036b01a624d783e8292398fe3a802857" Oct 30 20:17:06.860409 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.860318 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d62a88067ef970155a179d2d94cbde6d036b01a624d783e8292398fe3a802857"} err="failed to get container status \"d62a88067ef970155a179d2d94cbde6d036b01a624d783e8292398fe3a802857\": rpc error: code = NotFound desc = could not find container \"d62a88067ef970155a179d2d94cbde6d036b01a624d783e8292398fe3a802857\": container with ID starting with d62a88067ef970155a179d2d94cbde6d036b01a624d783e8292398fe3a802857 not found: ID does not exist" Oct 30 20:17:06.860409 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.860341 5091 scope.go:122] "RemoveContainer" containerID="defaf3f23cd5a02404442104f3e76a6067f2d74dadbdb9d4bb256f6005ca8542" Oct 30 20:17:06.880343 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.880302 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk"] Oct 30 20:17:06.898520 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.898495 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-9589btknk"] Oct 30 20:17:06.931764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.931727 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hw2x\" (UniqueName: \"kubernetes.io/projected/d973dcd0-1e61-4929-a1a0-648efb94577c-kube-api-access-4hw2x\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:06.968589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:06.968561 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:17:07.032586 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.032551 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04dd38d4-f223-4b01-8a58-ac3ef092a862-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04dd38d4-f223-4b01-8a58-ac3ef092a862-plugins-config-volume\") pod \"04dd38d4-f223-4b01-8a58-ac3ef092a862\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " Oct 30 20:17:07.032749 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.032671 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04dd38d4-f223-4b01-8a58-ac3ef092a862-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04dd38d4-f223-4b01-8a58-ac3ef092a862-routing-overlay-volume\") pod \"04dd38d4-f223-4b01-8a58-ac3ef092a862\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " Oct 30 20:17:07.032821 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.032808 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04dd38d4-f223-4b01-8a58-ac3ef092a862-kube-api-access-nj5xk\" (UniqueName: \"kubernetes.io/projected/04dd38d4-f223-4b01-8a58-ac3ef092a862-kube-api-access-nj5xk\") pod \"04dd38d4-f223-4b01-8a58-ac3ef092a862\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " Oct 30 20:17:07.032948 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.032932 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04dd38d4-f223-4b01-8a58-ac3ef092a862-provider-credentials\" (UniqueName: \"kubernetes.io/projected/04dd38d4-f223-4b01-8a58-ac3ef092a862-provider-credentials\") pod \"04dd38d4-f223-4b01-8a58-ac3ef092a862\" (UID: \"04dd38d4-f223-4b01-8a58-ac3ef092a862\") " Oct 30 20:17:07.033068 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.033039 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04dd38d4-f223-4b01-8a58-ac3ef092a862-plugins-config-volume" pod "04dd38d4-f223-4b01-8a58-ac3ef092a862" (UID: "04dd38d4-f223-4b01-8a58-ac3ef092a862"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:07.033182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.033047 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04dd38d4-f223-4b01-8a58-ac3ef092a862-routing-overlay-volume" pod "04dd38d4-f223-4b01-8a58-ac3ef092a862" (UID: "04dd38d4-f223-4b01-8a58-ac3ef092a862"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:07.033588 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.033559 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/04dd38d4-f223-4b01-8a58-ac3ef092a862-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:07.033704 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.033592 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/04dd38d4-f223-4b01-8a58-ac3ef092a862-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:07.035084 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.034987 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:07.035458 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.035435 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04dd38d4-f223-4b01-8a58-ac3ef092a862-provider-credentials" pod "04dd38d4-f223-4b01-8a58-ac3ef092a862" (UID: "04dd38d4-f223-4b01-8a58-ac3ef092a862"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:07.035538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.035486 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04dd38d4-f223-4b01-8a58-ac3ef092a862-kube-api-access-nj5xk" pod "04dd38d4-f223-4b01-8a58-ac3ef092a862" (UID: "04dd38d4-f223-4b01-8a58-ac3ef092a862"). InnerVolumeSpecName "kube-api-access-nj5xk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:07.136313 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.136282 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nj5xk\" (UniqueName: \"kubernetes.io/projected/04dd38d4-f223-4b01-8a58-ac3ef092a862-kube-api-access-nj5xk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:07.136666 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.136315 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/04dd38d4-f223-4b01-8a58-ac3ef092a862-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:07.153324 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.153291 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4"] Oct 30 20:17:07.158722 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.158696 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d97cc8-7dbb5df47-6d8p4"] Oct 30 20:17:07.577664 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.576816 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:07.828560 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.828478 5091 generic.go:356] "Generic (PLEG): container finished" podID="04dd38d4-f223-4b01-8a58-ac3ef092a862" containerID="2dc3f7c6ca81099eaa48c5e29a800a597e0562dd9518ccb0234062ee0a2d8e3f" exitCode=0 Oct 30 20:17:07.828560 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.828531 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" Oct 30 20:17:07.828790 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.828563 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" event={"ID":"04dd38d4-f223-4b01-8a58-ac3ef092a862","Type":"ContainerDied","Data":"2dc3f7c6ca81099eaa48c5e29a800a597e0562dd9518ccb0234062ee0a2d8e3f"} Oct 30 20:17:07.828790 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.828629 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2" event={"ID":"04dd38d4-f223-4b01-8a58-ac3ef092a862","Type":"ContainerDied","Data":"26e0b463b2e6578b5d88c0ab0f19dae8d810dccfa8f8a4856eeadda1c80653a5"} Oct 30 20:17:07.828790 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.828650 5091 scope.go:122] "RemoveContainer" containerID="2dc3f7c6ca81099eaa48c5e29a800a597e0562dd9518ccb0234062ee0a2d8e3f" Oct 30 20:17:07.846255 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.846231 5091 scope.go:122] "RemoveContainer" containerID="2dc3f7c6ca81099eaa48c5e29a800a597e0562dd9518ccb0234062ee0a2d8e3f" Oct 30 20:17:07.846515 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:07.846494 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dc3f7c6ca81099eaa48c5e29a800a597e0562dd9518ccb0234062ee0a2d8e3f\": container with ID starting with 2dc3f7c6ca81099eaa48c5e29a800a597e0562dd9518ccb0234062ee0a2d8e3f not found: ID does not exist" containerID="2dc3f7c6ca81099eaa48c5e29a800a597e0562dd9518ccb0234062ee0a2d8e3f" Oct 30 20:17:07.846590 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.846524 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dc3f7c6ca81099eaa48c5e29a800a597e0562dd9518ccb0234062ee0a2d8e3f"} err="failed to get container status \"2dc3f7c6ca81099eaa48c5e29a800a597e0562dd9518ccb0234062ee0a2d8e3f\": rpc error: code = NotFound desc = could not find container \"2dc3f7c6ca81099eaa48c5e29a800a597e0562dd9518ccb0234062ee0a2d8e3f\": container with ID starting with 2dc3f7c6ca81099eaa48c5e29a800a597e0562dd9518ccb0234062ee0a2d8e3f not found: ID does not exist" Oct 30 20:17:07.863341 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.863303 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2"] Oct 30 20:17:07.867348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:07.867323 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-6c68rlzh2"] Oct 30 20:17:08.274692 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.274659 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph"] Oct 30 20:17:08.275386 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275367 5091 cpu_manager.go:414] "containerMap: removing container" podUID="071ef2ea-f7f5-4e12-9287-73c4f832f361" containerName="istio-proxy" Oct 30 20:17:08.275456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275388 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="071ef2ea-f7f5-4e12-9287-73c4f832f361" containerName="istio-proxy" Oct 30 20:17:08.275456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275410 5091 cpu_manager.go:414] "containerMap: removing container" podUID="071ef2ea-f7f5-4e12-9287-73c4f832f361" containerName="istio-proxy" Oct 30 20:17:08.275456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275415 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="071ef2ea-f7f5-4e12-9287-73c4f832f361" containerName="istio-proxy" Oct 30 20:17:08.275456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275429 5091 cpu_manager.go:414] "containerMap: removing container" podUID="865fe459-6cb9-40c2-9afd-835e90619526" containerName="revoke-keys" Oct 30 20:17:08.275456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275434 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="865fe459-6cb9-40c2-9afd-835e90619526" containerName="revoke-keys" Oct 30 20:17:08.275456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275450 5091 cpu_manager.go:414] "containerMap: removing container" podUID="54ae8ff0-8ea1-4133-99a1-908bbfa06df3" containerName="revoke-keys" Oct 30 20:17:08.275456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275455 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54ae8ff0-8ea1-4133-99a1-908bbfa06df3" containerName="revoke-keys" Oct 30 20:17:08.275709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275473 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0ec79660-754a-446d-846c-9632c0b6b2e6" containerName="payload-processing" Oct 30 20:17:08.275709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275478 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ec79660-754a-446d-846c-9632c0b6b2e6" containerName="payload-processing" Oct 30 20:17:08.275709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275491 5091 cpu_manager.go:414] "containerMap: removing container" podUID="04dd38d4-f223-4b01-8a58-ac3ef092a862" containerName="payload-processing" Oct 30 20:17:08.275709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275496 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04dd38d4-f223-4b01-8a58-ac3ef092a862" containerName="payload-processing" Oct 30 20:17:08.275709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275511 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d973dcd0-1e61-4929-a1a0-648efb94577c" containerName="payload-pre-processing" Oct 30 20:17:08.275709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275515 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d973dcd0-1e61-4929-a1a0-648efb94577c" containerName="payload-pre-processing" Oct 30 20:17:08.275709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275613 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="071ef2ea-f7f5-4e12-9287-73c4f832f361" containerName="istio-proxy" Oct 30 20:17:08.275709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275621 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="865fe459-6cb9-40c2-9afd-835e90619526" containerName="revoke-keys" Oct 30 20:17:08.275709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275629 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="54ae8ff0-8ea1-4133-99a1-908bbfa06df3" containerName="revoke-keys" Oct 30 20:17:08.275709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275638 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ec79660-754a-446d-846c-9632c0b6b2e6" containerName="payload-processing" Oct 30 20:17:08.275709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275646 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="04dd38d4-f223-4b01-8a58-ac3ef092a862" containerName="payload-processing" Oct 30 20:17:08.275709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.275654 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d973dcd0-1e61-4929-a1a0-648efb94577c" containerName="payload-pre-processing" Oct 30 20:17:08.280389 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.280365 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.287314 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.287289 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c257e7\"/\"openshift-service-ca.crt\"" Oct 30 20:17:08.287446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.287316 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c257e7\"/\"kube-root-ca.crt\"" Oct 30 20:17:08.287446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.287375 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c257e7\"/\"default-dockercfg-28md6\"" Oct 30 20:17:08.287589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.287552 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c257e7\"/\"test-model-c257e7-kserve-self-signed-certs\"" Oct 30 20:17:08.292593 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.292562 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph"] Oct 30 20:17:08.313433 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.313403 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:08.388726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.388690 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-tmp-dir\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.388878 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.388820 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-model-cache\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.388990 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.388971 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-dshm\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.389075 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.389055 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-home\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.389111 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.389089 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6b4k\" (UniqueName: \"kubernetes.io/projected/176a900b-0376-4b44-b443-09f050c96daf-kube-api-access-t6b4k\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.389267 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.389252 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176a900b-0376-4b44-b443-09f050c96daf-tls-certs\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.473785 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.473745 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:08.500225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.491222 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176a900b-0376-4b44-b443-09f050c96daf-tls-certs\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.500225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.491381 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-tmp-dir\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.500225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.491471 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-model-cache\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.500225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.491595 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-dshm\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.500225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.491649 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-home\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.500225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.491675 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6b4k\" (UniqueName: \"kubernetes.io/projected/176a900b-0376-4b44-b443-09f050c96daf-kube-api-access-t6b4k\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.500225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.496555 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-model-cache\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.500225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.496824 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-tmp-dir\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.501135 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.501110 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-home\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.510766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.510739 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-dshm\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.556221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.542352 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6b4k\" (UniqueName: \"kubernetes.io/projected/176a900b-0376-4b44-b443-09f050c96daf-kube-api-access-t6b4k\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.558671 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.558599 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04dd38d4-f223-4b01-8a58-ac3ef092a862" path="/var/lib/kubelet/pods/04dd38d4-f223-4b01-8a58-ac3ef092a862/volumes" Oct 30 20:17:08.588416 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.571099 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176a900b-0376-4b44-b443-09f050c96daf-tls-certs\") pod \"test-model-c257e7-kserve-6c88cc97f7-h9wph\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.588416 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.586787 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="071ef2ea-f7f5-4e12-9287-73c4f832f361" path="/var/lib/kubelet/pods/071ef2ea-f7f5-4e12-9287-73c4f832f361/volumes" Oct 30 20:17:08.588416 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.587359 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ec79660-754a-446d-846c-9632c0b6b2e6" path="/var/lib/kubelet/pods/0ec79660-754a-446d-846c-9632c0b6b2e6/volumes" Oct 30 20:17:08.588416 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.587837 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12da7406-84d4-4eb6-83ce-49b324d67d71" path="/var/lib/kubelet/pods/12da7406-84d4-4eb6-83ce-49b324d67d71/volumes" Oct 30 20:17:08.588853 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.588649 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d973dcd0-1e61-4929-a1a0-648efb94577c" path="/var/lib/kubelet/pods/d973dcd0-1e61-4929-a1a0-648efb94577c/volumes" Oct 30 20:17:08.601834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.601809 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:08.896664 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:17:08.896634 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod176a900b_0376_4b44_b443_09f050c96daf.slice/crio-0a765f1dbe2cd9c2d88ed222223ef4b5d8ed7551074e4f78db9a6d666d1b14fa WatchSource:0}: Error finding container 0a765f1dbe2cd9c2d88ed222223ef4b5d8ed7551074e4f78db9a6d666d1b14fa: Status 404 returned error can't find the container with id 0a765f1dbe2cd9c2d88ed222223ef4b5d8ed7551074e4f78db9a6d666d1b14fa Oct 30 20:17:08.900230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:08.900204 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph"] Oct 30 20:17:09.313969 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:09.308867 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:09.558238 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:09.558204 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:09.791780 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:09.791743 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:09.862592 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:09.862553 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" event={"ID":"176a900b-0376-4b44-b443-09f050c96daf","Type":"ContainerStarted","Data":"02c5678127773ae6d48174b611fad090ce97b6aca773ae541ef0e36de18bde9d"} Oct 30 20:17:09.862783 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:09.862605 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" event={"ID":"176a900b-0376-4b44-b443-09f050c96daf","Type":"ContainerStarted","Data":"0a765f1dbe2cd9c2d88ed222223ef4b5d8ed7551074e4f78db9a6d666d1b14fa"} Oct 30 20:17:09.889833 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:09.889771 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" podStartSLOduration=1.889752241 podStartE2EDuration="1.889752241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:09.88664944 +0000 UTC m=+2749.581857399" watchObservedRunningTime="2026-10-30 20:17:09.889752241 +0000 UTC m=+2749.584960200" Oct 30 20:17:10.135933 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.135853 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822"] Oct 30 20:17:10.137217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.137196 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="071ef2ea-f7f5-4e12-9287-73c4f832f361" containerName="istio-proxy" Oct 30 20:17:10.141693 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.141668 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:10.148966 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.147852 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:10.149840 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.149810 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822"] Oct 30 20:17:10.313172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.296471 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:10.366171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.346416 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bc8bf29-7275-49f7-a260-9abbda340f8e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f92t822\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:10.374585 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.374547 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqj99\" (UniqueName: \"kubernetes.io/projected/7bc8bf29-7275-49f7-a260-9abbda340f8e-kube-api-access-jqj99\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f92t822\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:10.377185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.374926 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc8bf29-7275-49f7-a260-9abbda340f8e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f92t822\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:10.377546 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.377520 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bc8bf29-7275-49f7-a260-9abbda340f8e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f92t822\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:10.478627 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.478539 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqj99\" (UniqueName: \"kubernetes.io/projected/7bc8bf29-7275-49f7-a260-9abbda340f8e-kube-api-access-jqj99\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f92t822\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:10.478827 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.478636 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc8bf29-7275-49f7-a260-9abbda340f8e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f92t822\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:10.478827 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.478744 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bc8bf29-7275-49f7-a260-9abbda340f8e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f92t822\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:10.478983 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.478855 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bc8bf29-7275-49f7-a260-9abbda340f8e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f92t822\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:10.479535 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.479506 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bc8bf29-7275-49f7-a260-9abbda340f8e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f92t822\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:10.479667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.479640 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc8bf29-7275-49f7-a260-9abbda340f8e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f92t822\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:10.481545 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.481524 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bc8bf29-7275-49f7-a260-9abbda340f8e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f92t822\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:10.496057 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.496035 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqj99\" (UniqueName: \"kubernetes.io/projected/7bc8bf29-7275-49f7-a260-9abbda340f8e-kube-api-access-jqj99\") pod \"payload-processing-external-model-e2e-extmod-a-222480-59f92t822\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:10.539170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.538452 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:10.563171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.557387 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g"] Oct 30 20:17:10.620167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.599817 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:10.620167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.602914 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:10.675170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.674292 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g"] Oct 30 20:17:10.694168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.684309 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:10.694168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.684368 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:10.694168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.684434 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:10.694168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.684468 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5b7v\" (UniqueName: \"kubernetes.io/projected/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-kube-api-access-q5b7v\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:10.755589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.755550 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:10.829175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.805008 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:10.849860 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.829839 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:10.850357 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.850325 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:10.866170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.850562 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5b7v\" (UniqueName: \"kubernetes.io/projected/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-kube-api-access-q5b7v\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:10.900683 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.806844 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:10.971587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.848348 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:10.972181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.944450 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb"] Oct 30 20:17:10.983290 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.983207 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5b7v\" (UniqueName: \"kubernetes.io/projected/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-kube-api-access-q5b7v\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:10.984846 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.984819 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb"] Oct 30 20:17:10.985113 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.985096 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" Oct 30 20:17:10.991356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.991293 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:10.992519 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.992492 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:17:10.992652 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:10.992573 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:17:11.070371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.070331 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceea20c5-24d4-4269-8f0a-a022f2a34e76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e02073a0-stsbb\" (UID: \"ceea20c5-24d4-4269-8f0a-a022f2a34e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" Oct 30 20:17:11.070546 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.070454 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceea20c5-24d4-4269-8f0a-a022f2a34e76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e02073a0-stsbb\" (UID: \"ceea20c5-24d4-4269-8f0a-a022f2a34e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" Oct 30 20:17:11.070613 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.070575 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78f46\" (UniqueName: \"kubernetes.io/projected/ceea20c5-24d4-4269-8f0a-a022f2a34e76-kube-api-access-78f46\") pod \"maas-api-revoke-sub-e02073a0-stsbb\" (UID: \"ceea20c5-24d4-4269-8f0a-a022f2a34e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" Oct 30 20:17:11.172563 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.172527 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceea20c5-24d4-4269-8f0a-a022f2a34e76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e02073a0-stsbb\" (UID: \"ceea20c5-24d4-4269-8f0a-a022f2a34e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" Oct 30 20:17:11.172774 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.172590 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceea20c5-24d4-4269-8f0a-a022f2a34e76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e02073a0-stsbb\" (UID: \"ceea20c5-24d4-4269-8f0a-a022f2a34e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" Oct 30 20:17:11.172774 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.172655 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78f46\" (UniqueName: \"kubernetes.io/projected/ceea20c5-24d4-4269-8f0a-a022f2a34e76-kube-api-access-78f46\") pod \"maas-api-revoke-sub-e02073a0-stsbb\" (UID: \"ceea20c5-24d4-4269-8f0a-a022f2a34e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" Oct 30 20:17:11.173216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.173193 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceea20c5-24d4-4269-8f0a-a022f2a34e76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e02073a0-stsbb\" (UID: \"ceea20c5-24d4-4269-8f0a-a022f2a34e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" Oct 30 20:17:11.175050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.175025 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceea20c5-24d4-4269-8f0a-a022f2a34e76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e02073a0-stsbb\" (UID: \"ceea20c5-24d4-4269-8f0a-a022f2a34e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" Oct 30 20:17:11.186252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.186228 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78f46\" (UniqueName: \"kubernetes.io/projected/ceea20c5-24d4-4269-8f0a-a022f2a34e76-kube-api-access-78f46\") pod \"maas-api-revoke-sub-e02073a0-stsbb\" (UID: \"ceea20c5-24d4-4269-8f0a-a022f2a34e76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" Oct 30 20:17:11.267484 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:17:11.267450 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc8bf29_7275_49f7_a260_9abbda340f8e.slice/crio-369d12914e5e3a69ef7a7643706ef9d8e4364f898a4420e0d41ae2578659fcb5 WatchSource:0}: Error finding container 369d12914e5e3a69ef7a7643706ef9d8e4364f898a4420e0d41ae2578659fcb5: Status 404 returned error can't find the container with id 369d12914e5e3a69ef7a7643706ef9d8e4364f898a4420e0d41ae2578659fcb5 Oct 30 20:17:11.271579 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.271548 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822"] Oct 30 20:17:11.281753 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.281699 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:11.332426 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.332382 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" Oct 30 20:17:11.580410 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.580376 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g"] Oct 30 20:17:11.642187 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.642105 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:11.658727 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.658061 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb"] Oct 30 20:17:11.860161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.860124 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" Oct 30 20:17:11.925000 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.924971 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" Oct 30 20:17:11.992566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.992535 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7db6874-2d87-4709-9545-0f29a1532a6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7db6874-2d87-4709-9545-0f29a1532a6c-plugins-config-volume\") pod \"d7db6874-2d87-4709-9545-0f29a1532a6c\" (UID: \"d7db6874-2d87-4709-9545-0f29a1532a6c\") " Oct 30 20:17:11.992807 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.992787 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/002df7b8-201d-45a9-be26-076d2b887a7b-kube-api-access-2jf89\" (UniqueName: \"kubernetes.io/projected/002df7b8-201d-45a9-be26-076d2b887a7b-kube-api-access-2jf89\") pod \"002df7b8-201d-45a9-be26-076d2b887a7b\" (UID: \"002df7b8-201d-45a9-be26-076d2b887a7b\") " Oct 30 20:17:11.993009 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.992994 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/002df7b8-201d-45a9-be26-076d2b887a7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/002df7b8-201d-45a9-be26-076d2b887a7b-plugins-config-volume\") pod \"002df7b8-201d-45a9-be26-076d2b887a7b\" (UID: \"002df7b8-201d-45a9-be26-076d2b887a7b\") " Oct 30 20:17:11.993173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.993138 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7db6874-2d87-4709-9545-0f29a1532a6c-kube-api-access-n2pfk\" (UniqueName: \"kubernetes.io/projected/d7db6874-2d87-4709-9545-0f29a1532a6c-kube-api-access-n2pfk\") pod \"d7db6874-2d87-4709-9545-0f29a1532a6c\" (UID: \"d7db6874-2d87-4709-9545-0f29a1532a6c\") " Oct 30 20:17:11.994807 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.994779 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7db6874-2d87-4709-9545-0f29a1532a6c-plugins-config-volume" pod "d7db6874-2d87-4709-9545-0f29a1532a6c" (UID: "d7db6874-2d87-4709-9545-0f29a1532a6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:11.995077 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.995001 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/002df7b8-201d-45a9-be26-076d2b887a7b-plugins-config-volume" pod "002df7b8-201d-45a9-be26-076d2b887a7b" (UID: "002df7b8-201d-45a9-be26-076d2b887a7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:11.996486 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:11.996451 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/002df7b8-201d-45a9-be26-076d2b887a7b-kube-api-access-2jf89" pod "002df7b8-201d-45a9-be26-076d2b887a7b" (UID: "002df7b8-201d-45a9-be26-076d2b887a7b"). InnerVolumeSpecName "kube-api-access-2jf89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:12.002156 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.000499 5091 generic.go:356] "Generic (PLEG): container finished" podID="002df7b8-201d-45a9-be26-076d2b887a7b" containerID="fed957e828c7039bedfa8650c9d549e3003beac340e0e6966e1faadb5e7850db" exitCode=0 Oct 30 20:17:12.002156 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.000785 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" event={"ID":"002df7b8-201d-45a9-be26-076d2b887a7b","Type":"ContainerDied","Data":"fed957e828c7039bedfa8650c9d549e3003beac340e0e6966e1faadb5e7850db"} Oct 30 20:17:12.002156 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.000867 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" event={"ID":"002df7b8-201d-45a9-be26-076d2b887a7b","Type":"ContainerDied","Data":"f273e950b8fcf4bd70520a7be21b209fcf085afc975547d4543ba766aa1e4cb5"} Oct 30 20:17:12.002156 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.000898 5091 scope.go:122] "RemoveContainer" containerID="fed957e828c7039bedfa8650c9d549e3003beac340e0e6966e1faadb5e7850db" Oct 30 20:17:12.002156 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.001043 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5" Oct 30 20:17:12.007046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.006257 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" event={"ID":"f6fd0eda-db26-4ed8-b3fc-e972d1db813f","Type":"ContainerStarted","Data":"f08aee8d42171f0c9db5637e25245b837fc863b4fbbb26acf8889e8a335b4f61"} Oct 30 20:17:12.007046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.006313 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" event={"ID":"f6fd0eda-db26-4ed8-b3fc-e972d1db813f","Type":"ContainerStarted","Data":"fbaf9d69c85672d1e75054ebe685384f6156f9a6e15d9eb3a8dfcf866fe5073f"} Oct 30 20:17:12.007046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.007002 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:12.009257 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.009236 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7db6874-2d87-4709-9545-0f29a1532a6c-kube-api-access-n2pfk" pod "d7db6874-2d87-4709-9545-0f29a1532a6c" (UID: "d7db6874-2d87-4709-9545-0f29a1532a6c"). InnerVolumeSpecName "kube-api-access-n2pfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:12.010921 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.010548 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" event={"ID":"ceea20c5-24d4-4269-8f0a-a022f2a34e76","Type":"ContainerStarted","Data":"10d9b4896c0fb0a9adbf7332fad14f1d913858f1c3232b4ba55fb9617a1c271f"} Oct 30 20:17:12.015643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.014848 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" event={"ID":"7bc8bf29-7275-49f7-a260-9abbda340f8e","Type":"ContainerStarted","Data":"6e8be5392ec98945bc5d8619de65ae85e0f42f098b29a8d9608049b18b2539f3"} Oct 30 20:17:12.015643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.014892 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" event={"ID":"7bc8bf29-7275-49f7-a260-9abbda340f8e","Type":"ContainerStarted","Data":"369d12914e5e3a69ef7a7643706ef9d8e4364f898a4420e0d41ae2578659fcb5"} Oct 30 20:17:12.015643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.015589 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:12.018114 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.018053 5091 generic.go:356] "Generic (PLEG): container finished" podID="d7db6874-2d87-4709-9545-0f29a1532a6c" containerID="501ee98893a5b2731381939287fc18f41199f33e95ea797bf31bf0b6ddbdf772" exitCode=0 Oct 30 20:17:12.021195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.019286 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" event={"ID":"d7db6874-2d87-4709-9545-0f29a1532a6c","Type":"ContainerDied","Data":"501ee98893a5b2731381939287fc18f41199f33e95ea797bf31bf0b6ddbdf772"} Oct 30 20:17:12.021195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.019331 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" event={"ID":"d7db6874-2d87-4709-9545-0f29a1532a6c","Type":"ContainerDied","Data":"c9cb1001d11da4406b76a16ea16366fe7be34c92a6a1cd249041035355944801"} Oct 30 20:17:12.021195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.019517 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx" Oct 30 20:17:12.028574 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.028495 5091 scope.go:122] "RemoveContainer" containerID="fed957e828c7039bedfa8650c9d549e3003beac340e0e6966e1faadb5e7850db" Oct 30 20:17:12.030085 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:12.030027 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fed957e828c7039bedfa8650c9d549e3003beac340e0e6966e1faadb5e7850db\": container with ID starting with fed957e828c7039bedfa8650c9d549e3003beac340e0e6966e1faadb5e7850db not found: ID does not exist" containerID="fed957e828c7039bedfa8650c9d549e3003beac340e0e6966e1faadb5e7850db" Oct 30 20:17:12.030262 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.030097 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fed957e828c7039bedfa8650c9d549e3003beac340e0e6966e1faadb5e7850db"} err="failed to get container status \"fed957e828c7039bedfa8650c9d549e3003beac340e0e6966e1faadb5e7850db\": rpc error: code = NotFound desc = could not find container \"fed957e828c7039bedfa8650c9d549e3003beac340e0e6966e1faadb5e7850db\": container with ID starting with fed957e828c7039bedfa8650c9d549e3003beac340e0e6966e1faadb5e7850db not found: ID does not exist" Oct 30 20:17:12.030262 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.030121 5091 scope.go:122] "RemoveContainer" containerID="501ee98893a5b2731381939287fc18f41199f33e95ea797bf31bf0b6ddbdf772" Oct 30 20:17:12.044546 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.044450 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:12.072591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.072564 5091 scope.go:122] "RemoveContainer" containerID="501ee98893a5b2731381939287fc18f41199f33e95ea797bf31bf0b6ddbdf772" Oct 30 20:17:12.072960 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:12.072933 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"501ee98893a5b2731381939287fc18f41199f33e95ea797bf31bf0b6ddbdf772\": container with ID starting with 501ee98893a5b2731381939287fc18f41199f33e95ea797bf31bf0b6ddbdf772 not found: ID does not exist" containerID="501ee98893a5b2731381939287fc18f41199f33e95ea797bf31bf0b6ddbdf772" Oct 30 20:17:12.073091 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.072970 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"501ee98893a5b2731381939287fc18f41199f33e95ea797bf31bf0b6ddbdf772"} err="failed to get container status \"501ee98893a5b2731381939287fc18f41199f33e95ea797bf31bf0b6ddbdf772\": rpc error: code = NotFound desc = could not find container \"501ee98893a5b2731381939287fc18f41199f33e95ea797bf31bf0b6ddbdf772\": container with ID starting with 501ee98893a5b2731381939287fc18f41199f33e95ea797bf31bf0b6ddbdf772 not found: ID does not exist" Oct 30 20:17:12.073311 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.073261 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" podStartSLOduration=2.073243847 podStartE2EDuration="2.073243847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:12.038964892 +0000 UTC m=+2751.734172852" watchObservedRunningTime="2026-10-30 20:17:12.073243847 +0000 UTC m=+2751.768451806" Oct 30 20:17:12.080020 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.079973 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" podStartSLOduration=2.079956612 podStartE2EDuration="2.079956612s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:12.072800337 +0000 UTC m=+2751.768008299" watchObservedRunningTime="2026-10-30 20:17:12.079956612 +0000 UTC m=+2751.775164574" Oct 30 20:17:12.096979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.096950 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/002df7b8-201d-45a9-be26-076d2b887a7b-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:12.096979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.096975 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2pfk\" (UniqueName: \"kubernetes.io/projected/d7db6874-2d87-4709-9545-0f29a1532a6c-kube-api-access-n2pfk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:12.097128 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.096987 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7db6874-2d87-4709-9545-0f29a1532a6c-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:12.097128 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.096999 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jf89\" (UniqueName: \"kubernetes.io/projected/002df7b8-201d-45a9-be26-076d2b887a7b-kube-api-access-2jf89\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:12.101586 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.101561 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5"] Oct 30 20:17:12.107666 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.107643 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-b7431c-67ff4df979-t5hh5"] Oct 30 20:17:12.126094 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.126068 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx"] Oct 30 20:17:12.134477 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.134452 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-b7431c-78d6f57bcb-qnsqx"] Oct 30 20:17:12.187073 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.187042 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:12.453791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.453709 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="002df7b8-201d-45a9-be26-076d2b887a7b" path="/var/lib/kubelet/pods/002df7b8-201d-45a9-be26-076d2b887a7b/volumes" Oct 30 20:17:12.454505 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.454482 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7db6874-2d87-4709-9545-0f29a1532a6c" path="/var/lib/kubelet/pods/d7db6874-2d87-4709-9545-0f29a1532a6c/volumes" Oct 30 20:17:12.594167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:12.593977 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:13.029773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:13.029723 5091 generic.go:356] "Generic (PLEG): container finished" podID="ceea20c5-24d4-4269-8f0a-a022f2a34e76" containerID="56747554a424103d36cedcea199914e38ed7a3fc661b958d07360438881d8ad0" exitCode=0 Oct 30 20:17:13.030211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:13.030137 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" event={"ID":"ceea20c5-24d4-4269-8f0a-a022f2a34e76","Type":"ContainerDied","Data":"56747554a424103d36cedcea199914e38ed7a3fc661b958d07360438881d8ad0"} Oct 30 20:17:13.030414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:13.030393 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg"] Oct 30 20:17:13.035103 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:13.035083 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f19b333-hggmg"] Oct 30 20:17:13.381389 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:13.367516 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:14.103195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.102760 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:14.252606 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.252575 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" Oct 30 20:17:14.254073 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.254015 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:14.341234 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.341114 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceea20c5-24d4-4269-8f0a-a022f2a34e76-kube-api-access-78f46\" (UniqueName: \"kubernetes.io/projected/ceea20c5-24d4-4269-8f0a-a022f2a34e76-kube-api-access-78f46\") pod \"ceea20c5-24d4-4269-8f0a-a022f2a34e76\" (UID: \"ceea20c5-24d4-4269-8f0a-a022f2a34e76\") " Oct 30 20:17:14.341404 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.341359 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceea20c5-24d4-4269-8f0a-a022f2a34e76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceea20c5-24d4-4269-8f0a-a022f2a34e76-maas-api-xks-ca\") pod \"ceea20c5-24d4-4269-8f0a-a022f2a34e76\" (UID: \"ceea20c5-24d4-4269-8f0a-a022f2a34e76\") " Oct 30 20:17:14.341484 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.341435 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceea20c5-24d4-4269-8f0a-a022f2a34e76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceea20c5-24d4-4269-8f0a-a022f2a34e76-maas-api-openshift-ca\") pod \"ceea20c5-24d4-4269-8f0a-a022f2a34e76\" (UID: \"ceea20c5-24d4-4269-8f0a-a022f2a34e76\") " Oct 30 20:17:14.341810 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.341784 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceea20c5-24d4-4269-8f0a-a022f2a34e76-maas-api-openshift-ca" pod "ceea20c5-24d4-4269-8f0a-a022f2a34e76" (UID: "ceea20c5-24d4-4269-8f0a-a022f2a34e76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:14.342130 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.342113 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceea20c5-24d4-4269-8f0a-a022f2a34e76-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:14.343497 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.343471 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceea20c5-24d4-4269-8f0a-a022f2a34e76-maas-api-xks-ca" pod "ceea20c5-24d4-4269-8f0a-a022f2a34e76" (UID: "ceea20c5-24d4-4269-8f0a-a022f2a34e76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:14.343497 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.343488 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceea20c5-24d4-4269-8f0a-a022f2a34e76-kube-api-access-78f46" pod "ceea20c5-24d4-4269-8f0a-a022f2a34e76" (UID: "ceea20c5-24d4-4269-8f0a-a022f2a34e76"). InnerVolumeSpecName "kube-api-access-78f46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:14.407827 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.407789 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:14.417902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.417873 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b"] Oct 30 20:17:14.418098 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.418070 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" podUID="0fbaef91-57be-4511-a038-a607def818b5" containerName="istio-proxy" containerID="cri-o://3d02e820397d0c5e6c9b8d41196e997da72b575f0aa1c3b5168d254ab55841ec" gracePeriod=30 Oct 30 20:17:14.443239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.443211 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceea20c5-24d4-4269-8f0a-a022f2a34e76-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:14.443366 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.443240 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78f46\" (UniqueName: \"kubernetes.io/projected/ceea20c5-24d4-4269-8f0a-a022f2a34e76-kube-api-access-78f46\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:14.443787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.443762 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a49a5707-dc8c-4371-835f-8c9f99a4cc94" path="/var/lib/kubelet/pods/a49a5707-dc8c-4371-835f-8c9f99a4cc94/volumes" Oct 30 20:17:14.589168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.587119 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:14.879962 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.879906 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5"] Oct 30 20:17:14.881339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.881304 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ceea20c5-24d4-4269-8f0a-a022f2a34e76" containerName="revoke-keys" Oct 30 20:17:14.881339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.881329 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceea20c5-24d4-4269-8f0a-a022f2a34e76" containerName="revoke-keys" Oct 30 20:17:14.881539 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.881364 5091 cpu_manager.go:414] "containerMap: removing container" podUID="002df7b8-201d-45a9-be26-076d2b887a7b" containerName="payload-pre-processing" Oct 30 20:17:14.881539 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.881372 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="002df7b8-201d-45a9-be26-076d2b887a7b" containerName="payload-pre-processing" Oct 30 20:17:14.881539 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.881394 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d7db6874-2d87-4709-9545-0f29a1532a6c" containerName="payload-processing" Oct 30 20:17:14.881539 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.881401 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7db6874-2d87-4709-9545-0f29a1532a6c" containerName="payload-processing" Oct 30 20:17:14.881764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.881581 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceea20c5-24d4-4269-8f0a-a022f2a34e76" containerName="revoke-keys" Oct 30 20:17:14.881764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.881598 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="002df7b8-201d-45a9-be26-076d2b887a7b" containerName="payload-pre-processing" Oct 30 20:17:14.881764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.881612 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7db6874-2d87-4709-9545-0f29a1532a6c" containerName="payload-processing" Oct 30 20:17:14.887212 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.887179 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" Oct 30 20:17:14.897699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.897676 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5"] Oct 30 20:17:14.919139 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.919102 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:14.968460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.968423 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c932c433-9932-4f9b-9176-c4b070a058f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5\" (UID: \"c932c433-9932-4f9b-9176-c4b070a058f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" Oct 30 20:17:14.968680 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.968653 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c932c433-9932-4f9b-9176-c4b070a058f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5\" (UID: \"c932c433-9932-4f9b-9176-c4b070a058f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" Oct 30 20:17:14.968942 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:14.968915 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56fpn\" (UniqueName: \"kubernetes.io/projected/c932c433-9932-4f9b-9176-c4b070a058f9-kube-api-access-56fpn\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5\" (UID: \"c932c433-9932-4f9b-9176-c4b070a058f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" Oct 30 20:17:15.051887 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.051856 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" Oct 30 20:17:15.052093 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.051883 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb" event={"ID":"ceea20c5-24d4-4269-8f0a-a022f2a34e76","Type":"ContainerDied","Data":"10d9b4896c0fb0a9adbf7332fad14f1d913858f1c3232b4ba55fb9617a1c271f"} Oct 30 20:17:15.052093 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.051940 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10d9b4896c0fb0a9adbf7332fad14f1d913858f1c3232b4ba55fb9617a1c271f" Oct 30 20:17:15.081227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.070616 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c932c433-9932-4f9b-9176-c4b070a058f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5\" (UID: \"c932c433-9932-4f9b-9176-c4b070a058f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" Oct 30 20:17:15.081227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.070768 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c932c433-9932-4f9b-9176-c4b070a058f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5\" (UID: \"c932c433-9932-4f9b-9176-c4b070a058f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" Oct 30 20:17:15.081227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.071005 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56fpn\" (UniqueName: \"kubernetes.io/projected/c932c433-9932-4f9b-9176-c4b070a058f9-kube-api-access-56fpn\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5\" (UID: \"c932c433-9932-4f9b-9176-c4b070a058f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" Oct 30 20:17:15.081227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.071669 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c932c433-9932-4f9b-9176-c4b070a058f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5\" (UID: \"c932c433-9932-4f9b-9176-c4b070a058f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" Oct 30 20:17:15.091303 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.089986 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c932c433-9932-4f9b-9176-c4b070a058f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5\" (UID: \"c932c433-9932-4f9b-9176-c4b070a058f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" Oct 30 20:17:15.103981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.103952 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56fpn\" (UniqueName: \"kubernetes.io/projected/c932c433-9932-4f9b-9176-c4b070a058f9-kube-api-access-56fpn\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5\" (UID: \"c932c433-9932-4f9b-9176-c4b070a058f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" Oct 30 20:17:15.193474 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.193216 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:15.208634 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.208586 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" Oct 30 20:17:15.373165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.371383 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:15.458611 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.458530 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn"] Oct 30 20:17:15.465109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.465061 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-b24a1a-h62fn"] Oct 30 20:17:15.508028 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:17:15.508000 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc932c433_9932_4f9b_9176_c4b070a058f9.slice/crio-45c4347bc771c1cf8984b060439772cf63bcb3423240ad3e51f2ef91b2838f9f WatchSource:0}: Error finding container 45c4347bc771c1cf8984b060439772cf63bcb3423240ad3e51f2ef91b2838f9f: Status 404 returned error can't find the container with id 45c4347bc771c1cf8984b060439772cf63bcb3423240ad3e51f2ef91b2838f9f Oct 30 20:17:15.508877 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.508852 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5"] Oct 30 20:17:15.513861 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.513834 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:15.892504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.892470 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:15.896638 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.896612 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk"] Oct 30 20:17:15.900544 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:15.900520 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-0eeba5-vq6rk"] Oct 30 20:17:16.040204 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:16.040136 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:17:16.058384 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:16.058335 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:16.065207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:16.065177 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" event={"ID":"c932c433-9932-4f9b-9176-c4b070a058f9","Type":"ContainerStarted","Data":"fac137b2e79462925158430ab8df78e6f5cdb8c2ba565a1287a8593b2a97be4d"} Oct 30 20:17:16.065363 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:16.065237 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" event={"ID":"c932c433-9932-4f9b-9176-c4b070a058f9","Type":"ContainerStarted","Data":"45c4347bc771c1cf8984b060439772cf63bcb3423240ad3e51f2ef91b2838f9f"} Oct 30 20:17:16.101998 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:16.101946 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" podStartSLOduration=2.101933186 podStartE2EDuration="2.101933186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:16.100361123 +0000 UTC m=+2755.795569083" watchObservedRunningTime="2026-10-30 20:17:16.101933186 +0000 UTC m=+2755.797141144" Oct 30 20:17:16.442713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:16.442678 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54ae8ff0-8ea1-4133-99a1-908bbfa06df3" path="/var/lib/kubelet/pods/54ae8ff0-8ea1-4133-99a1-908bbfa06df3/volumes" Oct 30 20:17:16.443333 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:16.443167 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="992deac8-a20c-4142-a66c-8f7ca3b3b34d" path="/var/lib/kubelet/pods/992deac8-a20c-4142-a66c-8f7ca3b3b34d/volumes" Oct 30 20:17:16.494242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:16.494204 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:17.000169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:16.998170 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:17.078387 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:17.078351 5091 generic.go:356] "Generic (PLEG): container finished" podID="c932c433-9932-4f9b-9176-c4b070a058f9" containerID="fac137b2e79462925158430ab8df78e6f5cdb8c2ba565a1287a8593b2a97be4d" exitCode=0 Oct 30 20:17:17.078579 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:17.078463 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" event={"ID":"c932c433-9932-4f9b-9176-c4b070a058f9","Type":"ContainerDied","Data":"fac137b2e79462925158430ab8df78e6f5cdb8c2ba565a1287a8593b2a97be4d"} Oct 30 20:17:17.882287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:17.878492 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:18.271919 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:18.271887 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" Oct 30 20:17:18.353936 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:18.353905 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c932c433-9932-4f9b-9176-c4b070a058f9-kube-api-access-56fpn\" (UniqueName: \"kubernetes.io/projected/c932c433-9932-4f9b-9176-c4b070a058f9-kube-api-access-56fpn\") pod \"c932c433-9932-4f9b-9176-c4b070a058f9\" (UID: \"c932c433-9932-4f9b-9176-c4b070a058f9\") " Oct 30 20:17:18.354112 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:18.353966 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c932c433-9932-4f9b-9176-c4b070a058f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c932c433-9932-4f9b-9176-c4b070a058f9-maas-api-openshift-ca\") pod \"c932c433-9932-4f9b-9176-c4b070a058f9\" (UID: \"c932c433-9932-4f9b-9176-c4b070a058f9\") " Oct 30 20:17:18.354180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:18.354116 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c932c433-9932-4f9b-9176-c4b070a058f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c932c433-9932-4f9b-9176-c4b070a058f9-maas-api-xks-ca\") pod \"c932c433-9932-4f9b-9176-c4b070a058f9\" (UID: \"c932c433-9932-4f9b-9176-c4b070a058f9\") " Oct 30 20:17:18.354490 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:18.354455 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c932c433-9932-4f9b-9176-c4b070a058f9-maas-api-openshift-ca" pod "c932c433-9932-4f9b-9176-c4b070a058f9" (UID: "c932c433-9932-4f9b-9176-c4b070a058f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:18.355205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:18.355180 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c932c433-9932-4f9b-9176-c4b070a058f9-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:18.356897 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:18.356868 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c932c433-9932-4f9b-9176-c4b070a058f9-maas-api-xks-ca" pod "c932c433-9932-4f9b-9176-c4b070a058f9" (UID: "c932c433-9932-4f9b-9176-c4b070a058f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:18.357186 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:18.357164 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c932c433-9932-4f9b-9176-c4b070a058f9-kube-api-access-56fpn" pod "c932c433-9932-4f9b-9176-c4b070a058f9" (UID: "c932c433-9932-4f9b-9176-c4b070a058f9"). InnerVolumeSpecName "kube-api-access-56fpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:18.457124 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:18.457095 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56fpn\" (UniqueName: \"kubernetes.io/projected/c932c433-9932-4f9b-9176-c4b070a058f9-kube-api-access-56fpn\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:18.457124 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:18.457124 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c932c433-9932-4f9b-9176-c4b070a058f9-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:18.602705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:18.602676 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:18.618435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:18.618405 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:18.709224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:18.688218 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:19.098943 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.098911 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" Oct 30 20:17:19.098943 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.098936 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5" event={"ID":"c932c433-9932-4f9b-9176-c4b070a058f9","Type":"ContainerDied","Data":"45c4347bc771c1cf8984b060439772cf63bcb3423240ad3e51f2ef91b2838f9f"} Oct 30 20:17:19.099472 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.098998 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45c4347bc771c1cf8984b060439772cf63bcb3423240ad3e51f2ef91b2838f9f" Oct 30 20:17:19.110715 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.110691 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:17:19.463640 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:19.463602 5091 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fbaef91_57be_4511_a038_a607def818b5.slice/crio-3d02e820397d0c5e6c9b8d41196e997da72b575f0aa1c3b5168d254ab55841ec.scope\": RecentStats: unable to find data in memory cache]" Oct 30 20:17:19.581175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.569973 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:19.789572 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.789544 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 20:17:19.881167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.881113 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fbaef91-57be-4511-a038-a607def818b5-kube-api-access-89xwg\" (UniqueName: \"kubernetes.io/projected/0fbaef91-57be-4511-a038-a607def818b5-kube-api-access-89xwg\") pod \"0fbaef91-57be-4511-a038-a607def818b5\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " Oct 30 20:17:19.881371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.881338 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fbaef91-57be-4511-a038-a607def818b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbaef91-57be-4511-a038-a607def818b5-istiod-ca-cert\") pod \"0fbaef91-57be-4511-a038-a607def818b5\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " Oct 30 20:17:19.881649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.881549 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-credential-socket\") pod \"0fbaef91-57be-4511-a038-a607def818b5\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " Oct 30 20:17:19.881649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.881629 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-istio-envoy\") pod \"0fbaef91-57be-4511-a038-a607def818b5\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " Oct 30 20:17:19.881865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.881678 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0fbaef91-57be-4511-a038-a607def818b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbaef91-57be-4511-a038-a607def818b5-istio-podinfo\") pod \"0fbaef91-57be-4511-a038-a607def818b5\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " Oct 30 20:17:19.881865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.881759 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fbaef91-57be-4511-a038-a607def818b5-istio-token\" (UniqueName: \"kubernetes.io/projected/0fbaef91-57be-4511-a038-a607def818b5-istio-token\") pod \"0fbaef91-57be-4511-a038-a607def818b5\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " Oct 30 20:17:19.882030 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.881863 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-istio-data\") pod \"0fbaef91-57be-4511-a038-a607def818b5\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " Oct 30 20:17:19.882030 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.881947 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-workload-socket\") pod \"0fbaef91-57be-4511-a038-a607def818b5\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " Oct 30 20:17:19.882030 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.882003 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-workload-certs\") pod \"0fbaef91-57be-4511-a038-a607def818b5\" (UID: \"0fbaef91-57be-4511-a038-a607def818b5\") " Oct 30 20:17:19.882904 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.882582 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fbaef91-57be-4511-a038-a607def818b5-istiod-ca-cert" pod "0fbaef91-57be-4511-a038-a607def818b5" (UID: "0fbaef91-57be-4511-a038-a607def818b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:19.882904 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.882832 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0fbaef91-57be-4511-a038-a607def818b5-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:19.883224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.883085 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-istio-data" pod "0fbaef91-57be-4511-a038-a607def818b5" (UID: "0fbaef91-57be-4511-a038-a607def818b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:19.883340 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.883308 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-workload-socket" pod "0fbaef91-57be-4511-a038-a607def818b5" (UID: "0fbaef91-57be-4511-a038-a607def818b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:19.883516 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.883493 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-workload-certs" pod "0fbaef91-57be-4511-a038-a607def818b5" (UID: "0fbaef91-57be-4511-a038-a607def818b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:19.884335 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.884312 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-credential-socket" pod "0fbaef91-57be-4511-a038-a607def818b5" (UID: "0fbaef91-57be-4511-a038-a607def818b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:19.886676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.886651 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-istio-envoy" pod "0fbaef91-57be-4511-a038-a607def818b5" (UID: "0fbaef91-57be-4511-a038-a607def818b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:19.887217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.887182 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbaef91-57be-4511-a038-a607def818b5-istio-token" pod "0fbaef91-57be-4511-a038-a607def818b5" (UID: "0fbaef91-57be-4511-a038-a607def818b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:19.888509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.888483 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0fbaef91-57be-4511-a038-a607def818b5-istio-podinfo" pod "0fbaef91-57be-4511-a038-a607def818b5" (UID: "0fbaef91-57be-4511-a038-a607def818b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:17:19.888631 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.888485 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fbaef91-57be-4511-a038-a607def818b5-kube-api-access-89xwg" pod "0fbaef91-57be-4511-a038-a607def818b5" (UID: "0fbaef91-57be-4511-a038-a607def818b5"). InnerVolumeSpecName "kube-api-access-89xwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:19.984747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.984709 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:19.984747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.984747 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:19.984932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.984763 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0fbaef91-57be-4511-a038-a607def818b5-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:19.984932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.984777 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fbaef91-57be-4511-a038-a607def818b5-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:19.984932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.984791 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:19.984932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.984804 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:19.984932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.984817 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fbaef91-57be-4511-a038-a607def818b5-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:19.984932 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:19.984832 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89xwg\" (UniqueName: \"kubernetes.io/projected/0fbaef91-57be-4511-a038-a607def818b5-kube-api-access-89xwg\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:20.109566 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:20.109533 5091 generic.go:356] "Generic (PLEG): container finished" podID="0fbaef91-57be-4511-a038-a607def818b5" containerID="3d02e820397d0c5e6c9b8d41196e997da72b575f0aa1c3b5168d254ab55841ec" exitCode=0 Oct 30 20:17:20.110028 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:20.109586 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" Oct 30 20:17:20.110028 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:20.109625 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" event={"ID":"0fbaef91-57be-4511-a038-a607def818b5","Type":"ContainerDied","Data":"3d02e820397d0c5e6c9b8d41196e997da72b575f0aa1c3b5168d254ab55841ec"} Oct 30 20:17:20.110028 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:20.109718 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b" event={"ID":"0fbaef91-57be-4511-a038-a607def818b5","Type":"ContainerDied","Data":"6b77a0a757cd2d0ef396afcc8de56acdc6694904750a640d498ee79e1d828dea"} Oct 30 20:17:20.110028 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:20.109751 5091 scope.go:122] "RemoveContainer" containerID="3d02e820397d0c5e6c9b8d41196e997da72b575f0aa1c3b5168d254ab55841ec" Oct 30 20:17:20.122710 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:20.122689 5091 scope.go:122] "RemoveContainer" containerID="928ff57d39d92e480d79933914c6f7521efe79ac550a7792249ed935e9647e38" Oct 30 20:17:20.132915 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:20.132894 5091 scope.go:122] "RemoveContainer" containerID="3d02e820397d0c5e6c9b8d41196e997da72b575f0aa1c3b5168d254ab55841ec" Oct 30 20:17:20.133204 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:20.133183 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d02e820397d0c5e6c9b8d41196e997da72b575f0aa1c3b5168d254ab55841ec\": container with ID starting with 3d02e820397d0c5e6c9b8d41196e997da72b575f0aa1c3b5168d254ab55841ec not found: ID does not exist" containerID="3d02e820397d0c5e6c9b8d41196e997da72b575f0aa1c3b5168d254ab55841ec" Oct 30 20:17:20.133252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:20.133214 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d02e820397d0c5e6c9b8d41196e997da72b575f0aa1c3b5168d254ab55841ec"} err="failed to get container status \"3d02e820397d0c5e6c9b8d41196e997da72b575f0aa1c3b5168d254ab55841ec\": rpc error: code = NotFound desc = could not find container \"3d02e820397d0c5e6c9b8d41196e997da72b575f0aa1c3b5168d254ab55841ec\": container with ID starting with 3d02e820397d0c5e6c9b8d41196e997da72b575f0aa1c3b5168d254ab55841ec not found: ID does not exist" Oct 30 20:17:20.133252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:20.133233 5091 scope.go:122] "RemoveContainer" containerID="928ff57d39d92e480d79933914c6f7521efe79ac550a7792249ed935e9647e38" Oct 30 20:17:20.133483 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:20.133464 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"928ff57d39d92e480d79933914c6f7521efe79ac550a7792249ed935e9647e38\": container with ID starting with 928ff57d39d92e480d79933914c6f7521efe79ac550a7792249ed935e9647e38 not found: ID does not exist" containerID="928ff57d39d92e480d79933914c6f7521efe79ac550a7792249ed935e9647e38" Oct 30 20:17:20.133526 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:20.133494 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"928ff57d39d92e480d79933914c6f7521efe79ac550a7792249ed935e9647e38"} err="failed to get container status \"928ff57d39d92e480d79933914c6f7521efe79ac550a7792249ed935e9647e38\": rpc error: code = NotFound desc = could not find container \"928ff57d39d92e480d79933914c6f7521efe79ac550a7792249ed935e9647e38\": container with ID starting with 928ff57d39d92e480d79933914c6f7521efe79ac550a7792249ed935e9647e38 not found: ID does not exist" Oct 30 20:17:20.143676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:20.143651 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b"] Oct 30 20:17:20.150476 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:20.150450 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-b7431c-openshift-default-5cccbf5b4c-qrd5b"] Oct 30 20:17:20.444548 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:20.444466 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fbaef91-57be-4511-a038-a607def818b5" path="/var/lib/kubelet/pods/0fbaef91-57be-4511-a038-a607def818b5/volumes" Oct 30 20:17:20.454421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:20.454387 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:21.361460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.361424 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:21.457026 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.456992 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp"] Oct 30 20:17:21.457696 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.457680 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c932c433-9932-4f9b-9176-c4b070a058f9" containerName="revoke-keys" Oct 30 20:17:21.457761 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.457696 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c932c433-9932-4f9b-9176-c4b070a058f9" containerName="revoke-keys" Oct 30 20:17:21.457761 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.457722 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0fbaef91-57be-4511-a038-a607def818b5" containerName="istio-proxy" Oct 30 20:17:21.457761 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.457728 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fbaef91-57be-4511-a038-a607def818b5" containerName="istio-proxy" Oct 30 20:17:21.457860 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.457764 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0fbaef91-57be-4511-a038-a607def818b5" containerName="istio-proxy" Oct 30 20:17:21.457860 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.457770 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fbaef91-57be-4511-a038-a607def818b5" containerName="istio-proxy" Oct 30 20:17:21.457926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.457863 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fbaef91-57be-4511-a038-a607def818b5" containerName="istio-proxy" Oct 30 20:17:21.457926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.457877 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fbaef91-57be-4511-a038-a607def818b5" containerName="istio-proxy" Oct 30 20:17:21.457926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.457885 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c932c433-9932-4f9b-9176-c4b070a058f9" containerName="revoke-keys" Oct 30 20:17:21.463054 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.463032 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" Oct 30 20:17:21.470628 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.470606 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:17:21.470910 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.470889 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:17:21.471890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.471864 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp"] Oct 30 20:17:21.611581 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.611503 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbwgb\" (UniqueName: \"kubernetes.io/projected/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-kube-api-access-jbwgb\") pod \"maas-api-revoke-sub-50ff8fa5-tgzqp\" (UID: \"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" Oct 30 20:17:21.611759 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.611639 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50ff8fa5-tgzqp\" (UID: \"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" Oct 30 20:17:21.611918 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.611887 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50ff8fa5-tgzqp\" (UID: \"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" Oct 30 20:17:21.713540 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.713506 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbwgb\" (UniqueName: \"kubernetes.io/projected/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-kube-api-access-jbwgb\") pod \"maas-api-revoke-sub-50ff8fa5-tgzqp\" (UID: \"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" Oct 30 20:17:21.713679 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.713547 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50ff8fa5-tgzqp\" (UID: \"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" Oct 30 20:17:21.713765 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.713742 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50ff8fa5-tgzqp\" (UID: \"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" Oct 30 20:17:21.714379 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.714354 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50ff8fa5-tgzqp\" (UID: \"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" Oct 30 20:17:21.715903 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.715881 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50ff8fa5-tgzqp\" (UID: \"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" Oct 30 20:17:21.742310 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.742280 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbwgb\" (UniqueName: \"kubernetes.io/projected/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-kube-api-access-jbwgb\") pod \"maas-api-revoke-sub-50ff8fa5-tgzqp\" (UID: \"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" Oct 30 20:17:21.789320 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:21.789281 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" Oct 30 20:17:22.083483 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:17:22.083451 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode66a26f3_5b02_4bb5_b3b2_f33cffb93bec.slice/crio-f572bc0a52343522191ce1956ecd876c12ef13e6d8c61886feea162cd5ee21f8 WatchSource:0}: Error finding container f572bc0a52343522191ce1956ecd876c12ef13e6d8c61886feea162cd5ee21f8: Status 404 returned error can't find the container with id f572bc0a52343522191ce1956ecd876c12ef13e6d8c61886feea162cd5ee21f8 Oct 30 20:17:22.085938 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:22.085638 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp"] Oct 30 20:17:22.129883 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:22.129851 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" event={"ID":"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec","Type":"ContainerStarted","Data":"f572bc0a52343522191ce1956ecd876c12ef13e6d8c61886feea162cd5ee21f8"} Oct 30 20:17:22.248213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:22.245731 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:23.140882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:23.140848 5091 generic.go:356] "Generic (PLEG): container finished" podID="e66a26f3-5b02-4bb5-b3b2-f33cffb93bec" containerID="20bc969515e87e36c85fbba75c399cb963835874925bb0eeaec1c682386803b3" exitCode=0 Oct 30 20:17:23.141284 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:23.140932 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" event={"ID":"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec","Type":"ContainerDied","Data":"20bc969515e87e36c85fbba75c399cb963835874925bb0eeaec1c682386803b3"} Oct 30 20:17:23.174602 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:23.174562 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:24.044373 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.044344 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:24.044555 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.044423 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:24.106866 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.106815 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2"] Oct 30 20:17:24.126739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.126709 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v"] Oct 30 20:17:24.386181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.385813 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:24.398535 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.398506 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" Oct 30 20:17:24.504954 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.504922 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-maas-api-openshift-ca\") pod \"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec\" (UID: \"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec\") " Oct 30 20:17:24.505279 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.505254 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-maas-api-xks-ca\") pod \"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec\" (UID: \"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec\") " Oct 30 20:17:24.505349 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.505330 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-maas-api-openshift-ca" pod "e66a26f3-5b02-4bb5-b3b2-f33cffb93bec" (UID: "e66a26f3-5b02-4bb5-b3b2-f33cffb93bec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:24.505349 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.505341 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-kube-api-access-jbwgb\" (UniqueName: \"kubernetes.io/projected/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-kube-api-access-jbwgb\") pod \"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec\" (UID: \"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec\") " Oct 30 20:17:24.506173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.506131 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:24.507490 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.507462 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-maas-api-xks-ca" pod "e66a26f3-5b02-4bb5-b3b2-f33cffb93bec" (UID: "e66a26f3-5b02-4bb5-b3b2-f33cffb93bec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:24.507614 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.507595 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-kube-api-access-jbwgb" pod "e66a26f3-5b02-4bb5-b3b2-f33cffb93bec" (UID: "e66a26f3-5b02-4bb5-b3b2-f33cffb93bec"). InnerVolumeSpecName "kube-api-access-jbwgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:24.607370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.607307 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:24.607370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.607333 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbwgb\" (UniqueName: \"kubernetes.io/projected/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec-kube-api-access-jbwgb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:24.663290 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.656045 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:24.814407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.814365 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:24.920912 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.920829 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h"] Oct 30 20:17:24.921063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:24.921037 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" containerName="istio-proxy" containerID="cri-o://dca0e8d59dc560059e21e189ccf25b63fd4c4b307a1843209d4f15ca8cb47e60" gracePeriod=30 Oct 30 20:17:25.007583 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.007542 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:25.168416 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.168384 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" Oct 30 20:17:25.168416 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.168408 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp" event={"ID":"e66a26f3-5b02-4bb5-b3b2-f33cffb93bec","Type":"ContainerDied","Data":"f572bc0a52343522191ce1956ecd876c12ef13e6d8c61886feea162cd5ee21f8"} Oct 30 20:17:25.168646 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.168466 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f572bc0a52343522191ce1956ecd876c12ef13e6d8c61886feea162cd5ee21f8" Oct 30 20:17:25.171869 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.171798 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:25.340352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.340313 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:25.447576 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.447485 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9"] Oct 30 20:17:25.448737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.448682 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e66a26f3-5b02-4bb5-b3b2-f33cffb93bec" containerName="revoke-keys" Oct 30 20:17:25.448737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.448708 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e66a26f3-5b02-4bb5-b3b2-f33cffb93bec" containerName="revoke-keys" Oct 30 20:17:25.448945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.448931 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e66a26f3-5b02-4bb5-b3b2-f33cffb93bec" containerName="revoke-keys" Oct 30 20:17:25.452812 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.452792 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" Oct 30 20:17:25.462870 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.462710 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9"] Oct 30 20:17:25.463242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.463221 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:17:25.463410 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.463396 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:17:25.537587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.530207 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:25.647012 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.616044 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c13df5-9523-4987-a4e5-b4a678050ba8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9\" (UID: \"a9c13df5-9523-4987-a4e5-b4a678050ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" Oct 30 20:17:25.647012 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.616281 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c13df5-9523-4987-a4e5-b4a678050ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9\" (UID: \"a9c13df5-9523-4987-a4e5-b4a678050ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" Oct 30 20:17:25.647012 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.616506 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl6ht\" (UniqueName: \"kubernetes.io/projected/a9c13df5-9523-4987-a4e5-b4a678050ba8-kube-api-access-tl6ht\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9\" (UID: \"a9c13df5-9523-4987-a4e5-b4a678050ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" Oct 30 20:17:25.682378 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.682336 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt"] Oct 30 20:17:25.688279 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.688239 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt"] Oct 30 20:17:25.688438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.688356 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" Oct 30 20:17:25.717744 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.717688 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47tvb\" (UniqueName: \"kubernetes.io/projected/ed41553a-05cb-4bd6-9824-2fb89be87afa-kube-api-access-47tvb\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt\" (UID: \"ed41553a-05cb-4bd6-9824-2fb89be87afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" Oct 30 20:17:25.717961 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.717750 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl6ht\" (UniqueName: \"kubernetes.io/projected/a9c13df5-9523-4987-a4e5-b4a678050ba8-kube-api-access-tl6ht\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9\" (UID: \"a9c13df5-9523-4987-a4e5-b4a678050ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" Oct 30 20:17:25.718105 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.718075 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed41553a-05cb-4bd6-9824-2fb89be87afa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt\" (UID: \"ed41553a-05cb-4bd6-9824-2fb89be87afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" Oct 30 20:17:25.718304 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.718282 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c13df5-9523-4987-a4e5-b4a678050ba8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9\" (UID: \"a9c13df5-9523-4987-a4e5-b4a678050ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" Oct 30 20:17:25.718399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.718358 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed41553a-05cb-4bd6-9824-2fb89be87afa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt\" (UID: \"ed41553a-05cb-4bd6-9824-2fb89be87afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" Oct 30 20:17:25.718490 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.718473 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c13df5-9523-4987-a4e5-b4a678050ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9\" (UID: \"a9c13df5-9523-4987-a4e5-b4a678050ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" Oct 30 20:17:25.720218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.720192 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c13df5-9523-4987-a4e5-b4a678050ba8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9\" (UID: \"a9c13df5-9523-4987-a4e5-b4a678050ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" Oct 30 20:17:25.722647 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.722620 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c13df5-9523-4987-a4e5-b4a678050ba8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9\" (UID: \"a9c13df5-9523-4987-a4e5-b4a678050ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" Oct 30 20:17:25.741443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.741413 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl6ht\" (UniqueName: \"kubernetes.io/projected/a9c13df5-9523-4987-a4e5-b4a678050ba8-kube-api-access-tl6ht\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9\" (UID: \"a9c13df5-9523-4987-a4e5-b4a678050ba8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" Oct 30 20:17:25.770127 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.770101 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" Oct 30 20:17:25.820541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.820499 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47tvb\" (UniqueName: \"kubernetes.io/projected/ed41553a-05cb-4bd6-9824-2fb89be87afa-kube-api-access-47tvb\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt\" (UID: \"ed41553a-05cb-4bd6-9824-2fb89be87afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" Oct 30 20:17:25.820721 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.820579 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed41553a-05cb-4bd6-9824-2fb89be87afa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt\" (UID: \"ed41553a-05cb-4bd6-9824-2fb89be87afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" Oct 30 20:17:25.820791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.820764 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed41553a-05cb-4bd6-9824-2fb89be87afa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt\" (UID: \"ed41553a-05cb-4bd6-9824-2fb89be87afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" Oct 30 20:17:25.821787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.821730 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed41553a-05cb-4bd6-9824-2fb89be87afa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt\" (UID: \"ed41553a-05cb-4bd6-9824-2fb89be87afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" Oct 30 20:17:25.825372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.825343 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed41553a-05cb-4bd6-9824-2fb89be87afa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt\" (UID: \"ed41553a-05cb-4bd6-9824-2fb89be87afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" Oct 30 20:17:25.837315 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:25.836493 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47tvb\" (UniqueName: \"kubernetes.io/projected/ed41553a-05cb-4bd6-9824-2fb89be87afa-kube-api-access-47tvb\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt\" (UID: \"ed41553a-05cb-4bd6-9824-2fb89be87afa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" Oct 30 20:17:26.011397 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:26.011362 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" Oct 30 20:17:26.059755 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:26.059723 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9"] Oct 30 20:17:26.059907 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:17:26.059765 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9c13df5_9523_4987_a4e5_b4a678050ba8.slice/crio-98513e592a0e61560b282c167eb8504205025c378a50f3e2480e2256697aff50 WatchSource:0}: Error finding container 98513e592a0e61560b282c167eb8504205025c378a50f3e2480e2256697aff50: Status 404 returned error can't find the container with id 98513e592a0e61560b282c167eb8504205025c378a50f3e2480e2256697aff50 Oct 30 20:17:26.180909 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:26.180874 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" event={"ID":"a9c13df5-9523-4987-a4e5-b4a678050ba8","Type":"ContainerStarted","Data":"98513e592a0e61560b282c167eb8504205025c378a50f3e2480e2256697aff50"} Oct 30 20:17:26.319303 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:17:26.319270 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded41553a_05cb_4bd6_9824_2fb89be87afa.slice/crio-3c8bf345035985de952d3c8b161a7cd19c5d85f8d686dbcbfc6bc99f7339cd28 WatchSource:0}: Error finding container 3c8bf345035985de952d3c8b161a7cd19c5d85f8d686dbcbfc6bc99f7339cd28: Status 404 returned error can't find the container with id 3c8bf345035985de952d3c8b161a7cd19c5d85f8d686dbcbfc6bc99f7339cd28 Oct 30 20:17:26.320973 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:26.320943 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt"] Oct 30 20:17:26.702176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:26.698711 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:27.197320 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:27.197281 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" event={"ID":"ed41553a-05cb-4bd6-9824-2fb89be87afa","Type":"ContainerStarted","Data":"04b65825c9602565f820af37c3b5986f26b28c3f4138e594b411e10889f9886a"} Oct 30 20:17:27.197467 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:27.197335 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" event={"ID":"ed41553a-05cb-4bd6-9824-2fb89be87afa","Type":"ContainerStarted","Data":"3c8bf345035985de952d3c8b161a7cd19c5d85f8d686dbcbfc6bc99f7339cd28"} Oct 30 20:17:27.199373 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:27.199349 5091 generic.go:356] "Generic (PLEG): container finished" podID="a9c13df5-9523-4987-a4e5-b4a678050ba8" containerID="3eba951691ad960dae81576dfcb31df52fd1b5656c665d4abeea4da221368b5f" exitCode=0 Oct 30 20:17:27.199489 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:27.199394 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" event={"ID":"a9c13df5-9523-4987-a4e5-b4a678050ba8","Type":"ContainerDied","Data":"3eba951691ad960dae81576dfcb31df52fd1b5656c665d4abeea4da221368b5f"} Oct 30 20:17:27.230063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:27.230013 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" podStartSLOduration=2.22999559 podStartE2EDuration="2.22999559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:27.223394241 +0000 UTC m=+2766.918602198" watchObservedRunningTime="2026-10-30 20:17:27.22999559 +0000 UTC m=+2766.925203552" Oct 30 20:17:27.542301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:27.540659 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:28.209360 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:28.209325 5091 generic.go:356] "Generic (PLEG): container finished" podID="ed41553a-05cb-4bd6-9824-2fb89be87afa" containerID="04b65825c9602565f820af37c3b5986f26b28c3f4138e594b411e10889f9886a" exitCode=0 Oct 30 20:17:28.209847 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:28.209411 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" event={"ID":"ed41553a-05cb-4bd6-9824-2fb89be87afa","Type":"ContainerDied","Data":"04b65825c9602565f820af37c3b5986f26b28c3f4138e594b411e10889f9886a"} Oct 30 20:17:28.402703 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:28.402111 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:28.442530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:28.442503 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" Oct 30 20:17:28.474992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:28.474915 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9c13df5-9523-4987-a4e5-b4a678050ba8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c13df5-9523-4987-a4e5-b4a678050ba8-maas-api-xks-ca\") pod \"a9c13df5-9523-4987-a4e5-b4a678050ba8\" (UID: \"a9c13df5-9523-4987-a4e5-b4a678050ba8\") " Oct 30 20:17:28.474992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:28.474964 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9c13df5-9523-4987-a4e5-b4a678050ba8-kube-api-access-tl6ht\" (UniqueName: \"kubernetes.io/projected/a9c13df5-9523-4987-a4e5-b4a678050ba8-kube-api-access-tl6ht\") pod \"a9c13df5-9523-4987-a4e5-b4a678050ba8\" (UID: \"a9c13df5-9523-4987-a4e5-b4a678050ba8\") " Oct 30 20:17:28.475543 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:28.475134 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9c13df5-9523-4987-a4e5-b4a678050ba8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c13df5-9523-4987-a4e5-b4a678050ba8-maas-api-openshift-ca\") pod \"a9c13df5-9523-4987-a4e5-b4a678050ba8\" (UID: \"a9c13df5-9523-4987-a4e5-b4a678050ba8\") " Oct 30 20:17:28.475655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:28.475512 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9c13df5-9523-4987-a4e5-b4a678050ba8-maas-api-openshift-ca" pod "a9c13df5-9523-4987-a4e5-b4a678050ba8" (UID: "a9c13df5-9523-4987-a4e5-b4a678050ba8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:28.476593 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:28.476566 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9c13df5-9523-4987-a4e5-b4a678050ba8-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:28.493201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:28.478211 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9c13df5-9523-4987-a4e5-b4a678050ba8-maas-api-xks-ca" pod "a9c13df5-9523-4987-a4e5-b4a678050ba8" (UID: "a9c13df5-9523-4987-a4e5-b4a678050ba8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:28.493201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:28.478675 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9c13df5-9523-4987-a4e5-b4a678050ba8-kube-api-access-tl6ht" pod "a9c13df5-9523-4987-a4e5-b4a678050ba8" (UID: "a9c13df5-9523-4987-a4e5-b4a678050ba8"). InnerVolumeSpecName "kube-api-access-tl6ht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:28.577547 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:28.577506 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9c13df5-9523-4987-a4e5-b4a678050ba8-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:28.577547 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:28.577546 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl6ht\" (UniqueName: \"kubernetes.io/projected/a9c13df5-9523-4987-a4e5-b4a678050ba8-kube-api-access-tl6ht\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:29.107347 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.107308 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" podUID="af043c89-cf40-4065-b7ea-36f46412aa95" containerName="payload-processing" containerID="cri-o://60f42fa419beccbb3685206e2c03555e560891f3b0e22f590345d5e6c4eac8b2" gracePeriod=25 Oct 30 20:17:29.127225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.127192 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" podUID="19ec9497-2d39-4468-8165-921ef5727351" containerName="payload-processing" containerID="cri-o://607f387f376a22926f473451311929ea7d404297abc74dc3d9989ac9dd87486a" gracePeriod=25 Oct 30 20:17:29.219838 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.219816 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" Oct 30 20:17:29.220176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.219852 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9" event={"ID":"a9c13df5-9523-4987-a4e5-b4a678050ba8","Type":"ContainerDied","Data":"98513e592a0e61560b282c167eb8504205025c378a50f3e2480e2256697aff50"} Oct 30 20:17:29.220176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.219907 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98513e592a0e61560b282c167eb8504205025c378a50f3e2480e2256697aff50" Oct 30 20:17:29.313857 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.313787 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:29.502818 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.502494 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" Oct 30 20:17:29.582260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.582203 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:17:29.592893 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.592864 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:17:29.623364 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.623035 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed41553a-05cb-4bd6-9824-2fb89be87afa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed41553a-05cb-4bd6-9824-2fb89be87afa-maas-api-xks-ca\") pod \"ed41553a-05cb-4bd6-9824-2fb89be87afa\" (UID: \"ed41553a-05cb-4bd6-9824-2fb89be87afa\") " Oct 30 20:17:29.623364 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.623129 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed41553a-05cb-4bd6-9824-2fb89be87afa-kube-api-access-47tvb\" (UniqueName: \"kubernetes.io/projected/ed41553a-05cb-4bd6-9824-2fb89be87afa-kube-api-access-47tvb\") pod \"ed41553a-05cb-4bd6-9824-2fb89be87afa\" (UID: \"ed41553a-05cb-4bd6-9824-2fb89be87afa\") " Oct 30 20:17:29.623604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.623418 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed41553a-05cb-4bd6-9824-2fb89be87afa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed41553a-05cb-4bd6-9824-2fb89be87afa-maas-api-openshift-ca\") pod \"ed41553a-05cb-4bd6-9824-2fb89be87afa\" (UID: \"ed41553a-05cb-4bd6-9824-2fb89be87afa\") " Oct 30 20:17:29.624748 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.624489 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed41553a-05cb-4bd6-9824-2fb89be87afa-maas-api-openshift-ca" pod "ed41553a-05cb-4bd6-9824-2fb89be87afa" (UID: "ed41553a-05cb-4bd6-9824-2fb89be87afa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:29.634420 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.634350 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed41553a-05cb-4bd6-9824-2fb89be87afa-kube-api-access-47tvb" pod "ed41553a-05cb-4bd6-9824-2fb89be87afa" (UID: "ed41553a-05cb-4bd6-9824-2fb89be87afa"). InnerVolumeSpecName "kube-api-access-47tvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:29.634548 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.634434 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed41553a-05cb-4bd6-9824-2fb89be87afa-maas-api-xks-ca" pod "ed41553a-05cb-4bd6-9824-2fb89be87afa" (UID: "ed41553a-05cb-4bd6-9824-2fb89be87afa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:29.724845 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.724812 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ec9497-2d39-4468-8165-921ef5727351-kube-api-access-rsxql\" (UniqueName: \"kubernetes.io/projected/19ec9497-2d39-4468-8165-921ef5727351-kube-api-access-rsxql\") pod \"19ec9497-2d39-4468-8165-921ef5727351\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " Oct 30 20:17:29.725016 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.724857 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af043c89-cf40-4065-b7ea-36f46412aa95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af043c89-cf40-4065-b7ea-36f46412aa95-plugins-config-volume\") pod \"af043c89-cf40-4065-b7ea-36f46412aa95\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " Oct 30 20:17:29.725016 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.724961 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ec9497-2d39-4468-8165-921ef5727351-provider-credentials\" (UniqueName: \"kubernetes.io/projected/19ec9497-2d39-4468-8165-921ef5727351-provider-credentials\") pod \"19ec9497-2d39-4468-8165-921ef5727351\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " Oct 30 20:17:29.725016 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.725000 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af043c89-cf40-4065-b7ea-36f46412aa95-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af043c89-cf40-4065-b7ea-36f46412aa95-routing-overlay-volume\") pod \"af043c89-cf40-4065-b7ea-36f46412aa95\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " Oct 30 20:17:29.725170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.725041 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ec9497-2d39-4468-8165-921ef5727351-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ec9497-2d39-4468-8165-921ef5727351-plugins-config-volume\") pod \"19ec9497-2d39-4468-8165-921ef5727351\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " Oct 30 20:17:29.725313 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.725288 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ec9497-2d39-4468-8165-921ef5727351-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19ec9497-2d39-4468-8165-921ef5727351-routing-overlay-volume\") pod \"19ec9497-2d39-4468-8165-921ef5727351\" (UID: \"19ec9497-2d39-4468-8165-921ef5727351\") " Oct 30 20:17:29.725443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.725408 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af043c89-cf40-4065-b7ea-36f46412aa95-routing-overlay-volume" pod "af043c89-cf40-4065-b7ea-36f46412aa95" (UID: "af043c89-cf40-4065-b7ea-36f46412aa95"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:29.725701 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.725420 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af043c89-cf40-4065-b7ea-36f46412aa95-kube-api-access-p5bhc\" (UniqueName: \"kubernetes.io/projected/af043c89-cf40-4065-b7ea-36f46412aa95-kube-api-access-p5bhc\") pod \"af043c89-cf40-4065-b7ea-36f46412aa95\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " Oct 30 20:17:29.725806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.725447 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af043c89-cf40-4065-b7ea-36f46412aa95-plugins-config-volume" pod "af043c89-cf40-4065-b7ea-36f46412aa95" (UID: "af043c89-cf40-4065-b7ea-36f46412aa95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:29.725806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.725746 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af043c89-cf40-4065-b7ea-36f46412aa95-provider-credentials\" (UniqueName: \"kubernetes.io/projected/af043c89-cf40-4065-b7ea-36f46412aa95-provider-credentials\") pod \"af043c89-cf40-4065-b7ea-36f46412aa95\" (UID: \"af043c89-cf40-4065-b7ea-36f46412aa95\") " Oct 30 20:17:29.725806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.725603 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ec9497-2d39-4468-8165-921ef5727351-plugins-config-volume" pod "19ec9497-2d39-4468-8165-921ef5727351" (UID: "19ec9497-2d39-4468-8165-921ef5727351"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:29.725806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.725705 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ec9497-2d39-4468-8165-921ef5727351-routing-overlay-volume" pod "19ec9497-2d39-4468-8165-921ef5727351" (UID: "19ec9497-2d39-4468-8165-921ef5727351"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:29.726557 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.726529 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed41553a-05cb-4bd6-9824-2fb89be87afa-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:29.726655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.726563 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19ec9497-2d39-4468-8165-921ef5727351-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:29.726655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.726581 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47tvb\" (UniqueName: \"kubernetes.io/projected/ed41553a-05cb-4bd6-9824-2fb89be87afa-kube-api-access-47tvb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:29.726655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.726597 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/af043c89-cf40-4065-b7ea-36f46412aa95-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:29.726655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.726612 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed41553a-05cb-4bd6-9824-2fb89be87afa-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:29.726655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.726628 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/af043c89-cf40-4065-b7ea-36f46412aa95-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:29.726655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.726642 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ec9497-2d39-4468-8165-921ef5727351-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:29.727858 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.727831 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ec9497-2d39-4468-8165-921ef5727351-provider-credentials" pod "19ec9497-2d39-4468-8165-921ef5727351" (UID: "19ec9497-2d39-4468-8165-921ef5727351"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:29.727977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.727872 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af043c89-cf40-4065-b7ea-36f46412aa95-kube-api-access-p5bhc" pod "af043c89-cf40-4065-b7ea-36f46412aa95" (UID: "af043c89-cf40-4065-b7ea-36f46412aa95"). InnerVolumeSpecName "kube-api-access-p5bhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:29.727977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.727962 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af043c89-cf40-4065-b7ea-36f46412aa95-provider-credentials" pod "af043c89-cf40-4065-b7ea-36f46412aa95" (UID: "af043c89-cf40-4065-b7ea-36f46412aa95"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:29.728111 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.727991 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ec9497-2d39-4468-8165-921ef5727351-kube-api-access-rsxql" pod "19ec9497-2d39-4468-8165-921ef5727351" (UID: "19ec9497-2d39-4468-8165-921ef5727351"). InnerVolumeSpecName "kube-api-access-rsxql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:29.829129 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.829043 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5bhc\" (UniqueName: \"kubernetes.io/projected/af043c89-cf40-4065-b7ea-36f46412aa95-kube-api-access-p5bhc\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:29.829129 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.829083 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/af043c89-cf40-4065-b7ea-36f46412aa95-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:29.829129 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.829098 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsxql\" (UniqueName: \"kubernetes.io/projected/19ec9497-2d39-4468-8165-921ef5727351-kube-api-access-rsxql\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:29.829129 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:29.829113 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19ec9497-2d39-4468-8165-921ef5727351-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:30.223766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.210967 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:30.238332 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.238293 5091 generic.go:356] "Generic (PLEG): container finished" podID="af043c89-cf40-4065-b7ea-36f46412aa95" containerID="60f42fa419beccbb3685206e2c03555e560891f3b0e22f590345d5e6c4eac8b2" exitCode=0 Oct 30 20:17:30.238463 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.238442 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" Oct 30 20:17:30.238463 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.238448 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" event={"ID":"af043c89-cf40-4065-b7ea-36f46412aa95","Type":"ContainerDied","Data":"60f42fa419beccbb3685206e2c03555e560891f3b0e22f590345d5e6c4eac8b2"} Oct 30 20:17:30.238547 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.238532 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2" event={"ID":"af043c89-cf40-4065-b7ea-36f46412aa95","Type":"ContainerDied","Data":"160f7fb923402f4a5d5757f0d58d58e714e1d4425ea7d3d16d007dfe51ff3ce3"} Oct 30 20:17:30.238588 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.238565 5091 scope.go:122] "RemoveContainer" containerID="60f42fa419beccbb3685206e2c03555e560891f3b0e22f590345d5e6c4eac8b2" Oct 30 20:17:30.240668 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.240636 5091 generic.go:356] "Generic (PLEG): container finished" podID="19ec9497-2d39-4468-8165-921ef5727351" containerID="607f387f376a22926f473451311929ea7d404297abc74dc3d9989ac9dd87486a" exitCode=0 Oct 30 20:17:30.240780 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.240726 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" Oct 30 20:17:30.240868 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.240789 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" event={"ID":"19ec9497-2d39-4468-8165-921ef5727351","Type":"ContainerDied","Data":"607f387f376a22926f473451311929ea7d404297abc74dc3d9989ac9dd87486a"} Oct 30 20:17:30.240868 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.240856 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v" event={"ID":"19ec9497-2d39-4468-8165-921ef5727351","Type":"ContainerDied","Data":"d352abcc2f6e7d638a151625571a5d60779684b98f8edb46227f4561521259cc"} Oct 30 20:17:30.254897 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.254871 5091 scope.go:122] "RemoveContainer" containerID="60f42fa419beccbb3685206e2c03555e560891f3b0e22f590345d5e6c4eac8b2" Oct 30 20:17:30.255631 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:30.255602 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60f42fa419beccbb3685206e2c03555e560891f3b0e22f590345d5e6c4eac8b2\": container with ID starting with 60f42fa419beccbb3685206e2c03555e560891f3b0e22f590345d5e6c4eac8b2 not found: ID does not exist" containerID="60f42fa419beccbb3685206e2c03555e560891f3b0e22f590345d5e6c4eac8b2" Oct 30 20:17:30.255738 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.255641 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60f42fa419beccbb3685206e2c03555e560891f3b0e22f590345d5e6c4eac8b2"} err="failed to get container status \"60f42fa419beccbb3685206e2c03555e560891f3b0e22f590345d5e6c4eac8b2\": rpc error: code = NotFound desc = could not find container \"60f42fa419beccbb3685206e2c03555e560891f3b0e22f590345d5e6c4eac8b2\": container with ID starting with 60f42fa419beccbb3685206e2c03555e560891f3b0e22f590345d5e6c4eac8b2 not found: ID does not exist" Oct 30 20:17:30.255738 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.255667 5091 scope.go:122] "RemoveContainer" containerID="607f387f376a22926f473451311929ea7d404297abc74dc3d9989ac9dd87486a" Oct 30 20:17:30.256497 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.256471 5091 generic.go:356] "Generic (PLEG): container finished" podID="f206f771-c29d-41db-8768-12d2feefe4e3" containerID="dca0e8d59dc560059e21e189ccf25b63fd4c4b307a1843209d4f15ca8cb47e60" exitCode=0 Oct 30 20:17:30.256726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.256672 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" event={"ID":"f206f771-c29d-41db-8768-12d2feefe4e3","Type":"ContainerDied","Data":"dca0e8d59dc560059e21e189ccf25b63fd4c4b307a1843209d4f15ca8cb47e60"} Oct 30 20:17:30.259252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.259179 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" event={"ID":"ed41553a-05cb-4bd6-9824-2fb89be87afa","Type":"ContainerDied","Data":"3c8bf345035985de952d3c8b161a7cd19c5d85f8d686dbcbfc6bc99f7339cd28"} Oct 30 20:17:30.259411 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.259391 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt" Oct 30 20:17:30.259602 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.259581 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c8bf345035985de952d3c8b161a7cd19c5d85f8d686dbcbfc6bc99f7339cd28" Oct 30 20:17:30.267628 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.267610 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 20:17:30.271688 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.271667 5091 scope.go:122] "RemoveContainer" containerID="607f387f376a22926f473451311929ea7d404297abc74dc3d9989ac9dd87486a" Oct 30 20:17:30.271980 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:30.271962 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607f387f376a22926f473451311929ea7d404297abc74dc3d9989ac9dd87486a\": container with ID starting with 607f387f376a22926f473451311929ea7d404297abc74dc3d9989ac9dd87486a not found: ID does not exist" containerID="607f387f376a22926f473451311929ea7d404297abc74dc3d9989ac9dd87486a" Oct 30 20:17:30.272052 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.271986 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607f387f376a22926f473451311929ea7d404297abc74dc3d9989ac9dd87486a"} err="failed to get container status \"607f387f376a22926f473451311929ea7d404297abc74dc3d9989ac9dd87486a\": rpc error: code = NotFound desc = could not find container \"607f387f376a22926f473451311929ea7d404297abc74dc3d9989ac9dd87486a\": container with ID starting with 607f387f376a22926f473451311929ea7d404297abc74dc3d9989ac9dd87486a not found: ID does not exist" Oct 30 20:17:30.272052 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.272001 5091 scope.go:122] "RemoveContainer" containerID="8165d039203b13ecfef0568b2ff185b4e100ab5341ebf9306bdf750f4cb14819" Oct 30 20:17:30.302194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.302164 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2"] Oct 30 20:17:30.317671 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.317645 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-68f8x2pw2"] Oct 30 20:17:30.331761 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.331729 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v"] Oct 30 20:17:30.335435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.335415 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-596fvxq5v"] Oct 30 20:17:30.339246 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.339223 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-workload-certs\") pod \"f206f771-c29d-41db-8768-12d2feefe4e3\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " Oct 30 20:17:30.339330 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.339318 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-istio-data\") pod \"f206f771-c29d-41db-8768-12d2feefe4e3\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " Oct 30 20:17:30.339409 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.339399 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-workload-socket\") pod \"f206f771-c29d-41db-8768-12d2feefe4e3\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " Oct 30 20:17:30.339455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.339441 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-credential-socket\") pod \"f206f771-c29d-41db-8768-12d2feefe4e3\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " Oct 30 20:17:30.339542 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.339518 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-workload-certs" pod "f206f771-c29d-41db-8768-12d2feefe4e3" (UID: "f206f771-c29d-41db-8768-12d2feefe4e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:30.339594 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.339535 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f206f771-c29d-41db-8768-12d2feefe4e3-kube-api-access-8ps5g\" (UniqueName: \"kubernetes.io/projected/f206f771-c29d-41db-8768-12d2feefe4e3-kube-api-access-8ps5g\") pod \"f206f771-c29d-41db-8768-12d2feefe4e3\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " Oct 30 20:17:30.339594 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.339588 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f206f771-c29d-41db-8768-12d2feefe4e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f206f771-c29d-41db-8768-12d2feefe4e3-istio-podinfo\") pod \"f206f771-c29d-41db-8768-12d2feefe4e3\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " Oct 30 20:17:30.339683 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.339634 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-istio-data" pod "f206f771-c29d-41db-8768-12d2feefe4e3" (UID: "f206f771-c29d-41db-8768-12d2feefe4e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:30.339683 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.339665 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-workload-socket" pod "f206f771-c29d-41db-8768-12d2feefe4e3" (UID: "f206f771-c29d-41db-8768-12d2feefe4e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:30.339774 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.339670 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f206f771-c29d-41db-8768-12d2feefe4e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f206f771-c29d-41db-8768-12d2feefe4e3-istiod-ca-cert\") pod \"f206f771-c29d-41db-8768-12d2feefe4e3\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " Oct 30 20:17:30.339820 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.339767 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-credential-socket" pod "f206f771-c29d-41db-8768-12d2feefe4e3" (UID: "f206f771-c29d-41db-8768-12d2feefe4e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:30.339872 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.339813 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-istio-envoy\") pod \"f206f771-c29d-41db-8768-12d2feefe4e3\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " Oct 30 20:17:30.339872 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.339860 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f206f771-c29d-41db-8768-12d2feefe4e3-istio-token\" (UniqueName: \"kubernetes.io/projected/f206f771-c29d-41db-8768-12d2feefe4e3-istio-token\") pod \"f206f771-c29d-41db-8768-12d2feefe4e3\" (UID: \"f206f771-c29d-41db-8768-12d2feefe4e3\") " Oct 30 20:17:30.340033 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.340009 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f206f771-c29d-41db-8768-12d2feefe4e3-istiod-ca-cert" pod "f206f771-c29d-41db-8768-12d2feefe4e3" (UID: "f206f771-c29d-41db-8768-12d2feefe4e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:30.340636 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.340609 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f206f771-c29d-41db-8768-12d2feefe4e3-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:30.340735 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.340639 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:30.340735 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.340654 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:30.340735 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.340670 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:30.340735 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.340684 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:30.341955 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.341928 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-istio-envoy" pod "f206f771-c29d-41db-8768-12d2feefe4e3" (UID: "f206f771-c29d-41db-8768-12d2feefe4e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:30.342051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.341998 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f206f771-c29d-41db-8768-12d2feefe4e3-istio-podinfo" pod "f206f771-c29d-41db-8768-12d2feefe4e3" (UID: "f206f771-c29d-41db-8768-12d2feefe4e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:17:30.342051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.342020 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f206f771-c29d-41db-8768-12d2feefe4e3-kube-api-access-8ps5g" pod "f206f771-c29d-41db-8768-12d2feefe4e3" (UID: "f206f771-c29d-41db-8768-12d2feefe4e3"). InnerVolumeSpecName "kube-api-access-8ps5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:30.342300 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.342280 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f206f771-c29d-41db-8768-12d2feefe4e3-istio-token" pod "f206f771-c29d-41db-8768-12d2feefe4e3" (UID: "f206f771-c29d-41db-8768-12d2feefe4e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:30.441915 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.441880 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f206f771-c29d-41db-8768-12d2feefe4e3-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:30.441915 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.441916 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f206f771-c29d-41db-8768-12d2feefe4e3-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:30.442220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.441932 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ps5g\" (UniqueName: \"kubernetes.io/projected/f206f771-c29d-41db-8768-12d2feefe4e3-kube-api-access-8ps5g\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:30.442220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.441948 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f206f771-c29d-41db-8768-12d2feefe4e3-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:30.448395 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.448360 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ec9497-2d39-4468-8165-921ef5727351" path="/var/lib/kubelet/pods/19ec9497-2d39-4468-8165-921ef5727351/volumes" Oct 30 20:17:30.448990 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:30.448968 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af043c89-cf40-4065-b7ea-36f46412aa95" path="/var/lib/kubelet/pods/af043c89-cf40-4065-b7ea-36f46412aa95/volumes" Oct 30 20:17:31.194774 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:31.194692 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:31.271352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:31.271320 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" Oct 30 20:17:31.271863 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:31.271348 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h" event={"ID":"f206f771-c29d-41db-8768-12d2feefe4e3","Type":"ContainerDied","Data":"22291d3700b0460e42d6fec0347f2b5c30e0f572be1858b5cf9f90806c536a44"} Oct 30 20:17:31.271863 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:31.271419 5091 scope.go:122] "RemoveContainer" containerID="dca0e8d59dc560059e21e189ccf25b63fd4c4b307a1843209d4f15ca8cb47e60" Oct 30 20:17:31.325999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:31.325963 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h"] Oct 30 20:17:31.333101 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:31.333065 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-b24a1a-openshift-default-7cbd596bb9-jch5h"] Oct 30 20:17:32.210799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:32.210761 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:32.442807 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:32.442776 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" path="/var/lib/kubelet/pods/f206f771-c29d-41db-8768-12d2feefe4e3/volumes" Oct 30 20:17:33.009456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:33.009420 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822"] Oct 30 20:17:33.118660 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:33.118610 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:33.308244 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:33.306733 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:33.484239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:33.482034 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g"] Oct 30 20:17:33.551718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:33.551640 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:33.782982 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:33.782942 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:34.348170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:34.346632 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:35.202953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:35.202916 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:35.428724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:35.428690 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:35.536948 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:35.536877 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5"] Oct 30 20:17:35.548401 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:35.548369 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-7cbcbb-l5nv5"] Oct 30 20:17:35.608065 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:35.608030 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:35.781541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:35.781502 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:35.961791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:35.961695 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:36.110034 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:36.109991 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:36.333171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:36.327120 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:36.463662 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:36.463630 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c932c433-9932-4f9b-9176-c4b070a058f9" path="/var/lib/kubelet/pods/c932c433-9932-4f9b-9176-c4b070a058f9/volumes" Oct 30 20:17:36.496392 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:36.496349 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:36.664465 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:36.664381 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:37.585220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:37.581732 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:37.738713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:37.738675 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:38.009806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.009766 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" podUID="7bc8bf29-7275-49f7-a260-9abbda340f8e" containerName="payload-processing" containerID="cri-o://6e8be5392ec98945bc5d8619de65ae85e0f42f098b29a8d9608049b18b2539f3" gracePeriod=25 Oct 30 20:17:38.312063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.312038 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:38.450878 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.450844 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc8bf29-7275-49f7-a260-9abbda340f8e-kube-api-access-jqj99\" (UniqueName: \"kubernetes.io/projected/7bc8bf29-7275-49f7-a260-9abbda340f8e-kube-api-access-jqj99\") pod \"7bc8bf29-7275-49f7-a260-9abbda340f8e\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " Oct 30 20:17:38.451053 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.451006 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc8bf29-7275-49f7-a260-9abbda340f8e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bc8bf29-7275-49f7-a260-9abbda340f8e-routing-overlay-volume\") pod \"7bc8bf29-7275-49f7-a260-9abbda340f8e\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " Oct 30 20:17:38.451200 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.451180 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc8bf29-7275-49f7-a260-9abbda340f8e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bc8bf29-7275-49f7-a260-9abbda340f8e-provider-credentials\") pod \"7bc8bf29-7275-49f7-a260-9abbda340f8e\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " Oct 30 20:17:38.451302 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.451244 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc8bf29-7275-49f7-a260-9abbda340f8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc8bf29-7275-49f7-a260-9abbda340f8e-plugins-config-volume\") pod \"7bc8bf29-7275-49f7-a260-9abbda340f8e\" (UID: \"7bc8bf29-7275-49f7-a260-9abbda340f8e\") " Oct 30 20:17:38.451372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.451325 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc8bf29-7275-49f7-a260-9abbda340f8e-routing-overlay-volume" pod "7bc8bf29-7275-49f7-a260-9abbda340f8e" (UID: "7bc8bf29-7275-49f7-a260-9abbda340f8e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:38.451671 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.451648 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc8bf29-7275-49f7-a260-9abbda340f8e-plugins-config-volume" pod "7bc8bf29-7275-49f7-a260-9abbda340f8e" (UID: "7bc8bf29-7275-49f7-a260-9abbda340f8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:38.451801 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.451755 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7bc8bf29-7275-49f7-a260-9abbda340f8e-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:38.451801 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.451776 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7bc8bf29-7275-49f7-a260-9abbda340f8e-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:38.453206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.453182 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc8bf29-7275-49f7-a260-9abbda340f8e-kube-api-access-jqj99" pod "7bc8bf29-7275-49f7-a260-9abbda340f8e" (UID: "7bc8bf29-7275-49f7-a260-9abbda340f8e"). InnerVolumeSpecName "kube-api-access-jqj99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:38.454054 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.454026 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc8bf29-7275-49f7-a260-9abbda340f8e-provider-credentials" pod "7bc8bf29-7275-49f7-a260-9abbda340f8e" (UID: "7bc8bf29-7275-49f7-a260-9abbda340f8e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:38.460318 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.460293 5091 generic.go:356] "Generic (PLEG): container finished" podID="7bc8bf29-7275-49f7-a260-9abbda340f8e" containerID="6e8be5392ec98945bc5d8619de65ae85e0f42f098b29a8d9608049b18b2539f3" exitCode=0 Oct 30 20:17:38.460413 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.460340 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" Oct 30 20:17:38.460413 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.460377 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" event={"ID":"7bc8bf29-7275-49f7-a260-9abbda340f8e","Type":"ContainerDied","Data":"6e8be5392ec98945bc5d8619de65ae85e0f42f098b29a8d9608049b18b2539f3"} Oct 30 20:17:38.460513 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.460426 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822" event={"ID":"7bc8bf29-7275-49f7-a260-9abbda340f8e","Type":"ContainerDied","Data":"369d12914e5e3a69ef7a7643706ef9d8e4364f898a4420e0d41ae2578659fcb5"} Oct 30 20:17:38.460513 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.460453 5091 scope.go:122] "RemoveContainer" containerID="6e8be5392ec98945bc5d8619de65ae85e0f42f098b29a8d9608049b18b2539f3" Oct 30 20:17:38.505174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.501582 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" podUID="f6fd0eda-db26-4ed8-b3fc-e972d1db813f" containerName="payload-processing" containerID="cri-o://f08aee8d42171f0c9db5637e25245b837fc863b4fbbb26acf8889e8a335b4f61" gracePeriod=25 Oct 30 20:17:38.518669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.514759 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:38.538207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.535843 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822"] Oct 30 20:17:38.540013 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.539988 5091 scope.go:122] "RemoveContainer" containerID="6e8be5392ec98945bc5d8619de65ae85e0f42f098b29a8d9608049b18b2539f3" Oct 30 20:17:38.540484 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:38.540453 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e8be5392ec98945bc5d8619de65ae85e0f42f098b29a8d9608049b18b2539f3\": container with ID starting with 6e8be5392ec98945bc5d8619de65ae85e0f42f098b29a8d9608049b18b2539f3 not found: ID does not exist" containerID="6e8be5392ec98945bc5d8619de65ae85e0f42f098b29a8d9608049b18b2539f3" Oct 30 20:17:38.540618 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.540493 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e8be5392ec98945bc5d8619de65ae85e0f42f098b29a8d9608049b18b2539f3"} err="failed to get container status \"6e8be5392ec98945bc5d8619de65ae85e0f42f098b29a8d9608049b18b2539f3\": rpc error: code = NotFound desc = could not find container \"6e8be5392ec98945bc5d8619de65ae85e0f42f098b29a8d9608049b18b2539f3\": container with ID starting with 6e8be5392ec98945bc5d8619de65ae85e0f42f098b29a8d9608049b18b2539f3 not found: ID does not exist" Oct 30 20:17:38.542074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.542048 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-222480/payload-processing-external-model-e2e-extmod-a-222480-59f92t822"] Oct 30 20:17:38.554007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.553872 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7bc8bf29-7275-49f7-a260-9abbda340f8e-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:38.554007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.553916 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqj99\" (UniqueName: \"kubernetes.io/projected/7bc8bf29-7275-49f7-a260-9abbda340f8e-kube-api-access-jqj99\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:38.783688 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.783654 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:38.960379 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.960296 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-routing-overlay-volume\") pod \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " Oct 30 20:17:38.960557 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.960412 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-kube-api-access-q5b7v\" (UniqueName: \"kubernetes.io/projected/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-kube-api-access-q5b7v\") pod \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " Oct 30 20:17:38.960557 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.960456 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-provider-credentials\") pod \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " Oct 30 20:17:38.960557 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.960524 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-plugins-config-volume\") pod \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\" (UID: \"f6fd0eda-db26-4ed8-b3fc-e972d1db813f\") " Oct 30 20:17:38.960806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.960674 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-routing-overlay-volume" pod "f6fd0eda-db26-4ed8-b3fc-e972d1db813f" (UID: "f6fd0eda-db26-4ed8-b3fc-e972d1db813f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:38.960968 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.960942 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-plugins-config-volume" pod "f6fd0eda-db26-4ed8-b3fc-e972d1db813f" (UID: "f6fd0eda-db26-4ed8-b3fc-e972d1db813f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:38.961330 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.961310 5091 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-routing-overlay-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:38.961404 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.961334 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:38.962661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.962639 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-provider-credentials" pod "f6fd0eda-db26-4ed8-b3fc-e972d1db813f" (UID: "f6fd0eda-db26-4ed8-b3fc-e972d1db813f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:38.963039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:38.963018 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-kube-api-access-q5b7v" pod "f6fd0eda-db26-4ed8-b3fc-e972d1db813f" (UID: "f6fd0eda-db26-4ed8-b3fc-e972d1db813f"). InnerVolumeSpecName "kube-api-access-q5b7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:39.062335 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:39.062303 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5b7v\" (UniqueName: \"kubernetes.io/projected/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-kube-api-access-q5b7v\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:39.062335 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:39.062334 5091 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f6fd0eda-db26-4ed8-b3fc-e972d1db813f-provider-credentials\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:39.471922 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:39.471888 5091 generic.go:356] "Generic (PLEG): container finished" podID="f6fd0eda-db26-4ed8-b3fc-e972d1db813f" containerID="f08aee8d42171f0c9db5637e25245b837fc863b4fbbb26acf8889e8a335b4f61" exitCode=0 Oct 30 20:17:39.472184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:39.471951 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" Oct 30 20:17:39.472184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:39.471965 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" event={"ID":"f6fd0eda-db26-4ed8-b3fc-e972d1db813f","Type":"ContainerDied","Data":"f08aee8d42171f0c9db5637e25245b837fc863b4fbbb26acf8889e8a335b4f61"} Oct 30 20:17:39.472184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:39.472028 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g" event={"ID":"f6fd0eda-db26-4ed8-b3fc-e972d1db813f","Type":"ContainerDied","Data":"fbaf9d69c85672d1e75054ebe685384f6156f9a6e15d9eb3a8dfcf866fe5073f"} Oct 30 20:17:39.472184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:39.472060 5091 scope.go:122] "RemoveContainer" containerID="f08aee8d42171f0c9db5637e25245b837fc863b4fbbb26acf8889e8a335b4f61" Oct 30 20:17:39.484233 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:39.484214 5091 scope.go:122] "RemoveContainer" containerID="f08aee8d42171f0c9db5637e25245b837fc863b4fbbb26acf8889e8a335b4f61" Oct 30 20:17:39.484492 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:39.484473 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08aee8d42171f0c9db5637e25245b837fc863b4fbbb26acf8889e8a335b4f61\": container with ID starting with f08aee8d42171f0c9db5637e25245b837fc863b4fbbb26acf8889e8a335b4f61 not found: ID does not exist" containerID="f08aee8d42171f0c9db5637e25245b837fc863b4fbbb26acf8889e8a335b4f61" Oct 30 20:17:39.484563 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:39.484503 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08aee8d42171f0c9db5637e25245b837fc863b4fbbb26acf8889e8a335b4f61"} err="failed to get container status \"f08aee8d42171f0c9db5637e25245b837fc863b4fbbb26acf8889e8a335b4f61\": rpc error: code = NotFound desc = could not find container \"f08aee8d42171f0c9db5637e25245b837fc863b4fbbb26acf8889e8a335b4f61\": container with ID starting with f08aee8d42171f0c9db5637e25245b837fc863b4fbbb26acf8889e8a335b4f61 not found: ID does not exist" Oct 30 20:17:39.507919 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:39.507897 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g"] Oct 30 20:17:39.514010 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:39.513987 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-4d5758/payload-processing-external-model-e2e-extmod-b-4d5758-b6b62j25g"] Oct 30 20:17:39.596172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:39.595173 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:40.443551 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:40.443516 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc8bf29-7275-49f7-a260-9abbda340f8e" path="/var/lib/kubelet/pods/7bc8bf29-7275-49f7-a260-9abbda340f8e/volumes" Oct 30 20:17:40.445002 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:40.444972 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6fd0eda-db26-4ed8-b3fc-e972d1db813f" path="/var/lib/kubelet/pods/f6fd0eda-db26-4ed8-b3fc-e972d1db813f/volumes" Oct 30 20:17:40.459660 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:40.458598 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:40.774682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:40.774631 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:40.919385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:40.919347 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:41.296986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.289420 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:41.436370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.436338 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:41.630914 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.630832 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v"] Oct 30 20:17:41.631914 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.631890 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a9c13df5-9523-4987-a4e5-b4a678050ba8" containerName="revoke-keys" Oct 30 20:17:41.631974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.631919 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9c13df5-9523-4987-a4e5-b4a678050ba8" containerName="revoke-keys" Oct 30 20:17:41.631974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.631953 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" containerName="istio-proxy" Oct 30 20:17:41.631974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.631961 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" containerName="istio-proxy" Oct 30 20:17:41.632087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.631978 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ed41553a-05cb-4bd6-9824-2fb89be87afa" containerName="revoke-keys" Oct 30 20:17:41.632087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.631988 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed41553a-05cb-4bd6-9824-2fb89be87afa" containerName="revoke-keys" Oct 30 20:17:41.632087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632004 5091 cpu_manager.go:414] "containerMap: removing container" podUID="af043c89-cf40-4065-b7ea-36f46412aa95" containerName="payload-processing" Oct 30 20:17:41.632087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632012 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af043c89-cf40-4065-b7ea-36f46412aa95" containerName="payload-processing" Oct 30 20:17:41.632087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632036 5091 cpu_manager.go:414] "containerMap: removing container" podUID="7bc8bf29-7275-49f7-a260-9abbda340f8e" containerName="payload-processing" Oct 30 20:17:41.632087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632043 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc8bf29-7275-49f7-a260-9abbda340f8e" containerName="payload-processing" Oct 30 20:17:41.632087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632066 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f6fd0eda-db26-4ed8-b3fc-e972d1db813f" containerName="payload-processing" Oct 30 20:17:41.632087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632073 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6fd0eda-db26-4ed8-b3fc-e972d1db813f" containerName="payload-processing" Oct 30 20:17:41.632429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632119 5091 cpu_manager.go:414] "containerMap: removing container" podUID="19ec9497-2d39-4468-8165-921ef5727351" containerName="payload-processing" Oct 30 20:17:41.632429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632128 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ec9497-2d39-4468-8165-921ef5727351" containerName="payload-processing" Oct 30 20:17:41.632429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632299 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" containerName="istio-proxy" Oct 30 20:17:41.632429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632315 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ec9497-2d39-4468-8165-921ef5727351" containerName="payload-processing" Oct 30 20:17:41.632429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632328 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9c13df5-9523-4987-a4e5-b4a678050ba8" containerName="revoke-keys" Oct 30 20:17:41.632429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632343 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" containerName="istio-proxy" Oct 30 20:17:41.632429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632355 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed41553a-05cb-4bd6-9824-2fb89be87afa" containerName="revoke-keys" Oct 30 20:17:41.632429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632364 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="af043c89-cf40-4065-b7ea-36f46412aa95" containerName="payload-processing" Oct 30 20:17:41.632429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632377 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc8bf29-7275-49f7-a260-9abbda340f8e" containerName="payload-processing" Oct 30 20:17:41.632429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.632390 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6fd0eda-db26-4ed8-b3fc-e972d1db813f" containerName="payload-processing" Oct 30 20:17:41.638230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.638207 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.651571 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.650513 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v"] Oct 30 20:17:41.705180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.705131 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-89c124-openshift-default-dockercfg-s2d4p\"" Oct 30 20:17:41.757172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.750875 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-data\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.757172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.750951 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnnpb\" (UniqueName: \"kubernetes.io/projected/320dc7ae-3660-41d1-bc53-49d3427a0b6d-kube-api-access-vnnpb\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.757172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.751003 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-credential-socket\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.757172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.751063 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-podinfo\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.757172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.751094 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istiod-ca-cert\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.783172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.751140 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-workload-certs\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.783172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.774325 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-token\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.783172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.774376 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-workload-socket\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.783172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.774428 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-envoy\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.841172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.831463 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:41.904216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.886411 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-data\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.904216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.886490 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnnpb\" (UniqueName: \"kubernetes.io/projected/320dc7ae-3660-41d1-bc53-49d3427a0b6d-kube-api-access-vnnpb\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.904216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.886542 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-credential-socket\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.904216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.886599 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-podinfo\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.904216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.886631 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istiod-ca-cert\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.904216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.886682 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-workload-certs\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.904216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.886714 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-token\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.904216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.886736 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-workload-socket\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.904216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.886774 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-envoy\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.904216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.904032 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-data\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.919413 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.904330 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-credential-socket\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.919413 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.904610 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-workload-certs\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.919413 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.905241 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istiod-ca-cert\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.919413 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.906009 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-workload-socket\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.947632 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.936869 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnnpb\" (UniqueName: \"kubernetes.io/projected/320dc7ae-3660-41d1-bc53-49d3427a0b6d-kube-api-access-vnnpb\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.966895 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.950251 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-token\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.966895 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.952896 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-podinfo\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.966895 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.953176 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-envoy\") pod \"e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:41.981858 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:41.979622 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:42.017817 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:42.006749 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:42.184125 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:42.184019 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:42.324207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:42.324119 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:17:42.324321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:42.324240 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:17:42.324321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:42.324294 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:17:42.334299 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:42.334265 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v"] Oct 30 20:17:42.550514 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:42.550469 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" event={"ID":"320dc7ae-3660-41d1-bc53-49d3427a0b6d","Type":"ContainerStarted","Data":"f834c3bc0ffbb915da2643b1434f92a79d4a8a71e0eb0914ad67c8e29cfab6a7"} Oct 30 20:17:42.550728 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:42.550539 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" event={"ID":"320dc7ae-3660-41d1-bc53-49d3427a0b6d","Type":"ContainerStarted","Data":"82a4ce5c726095b8621b583e30cca42efa5dc801d63ec2d54ce4d2daad229d16"} Oct 30 20:17:42.579049 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:42.578978 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" podStartSLOduration=1.578964437 podStartE2EDuration="1.578964437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:42.577639366 +0000 UTC m=+2782.272847323" watchObservedRunningTime="2026-10-30 20:17:42.578964437 +0000 UTC m=+2782.274172395" Oct 30 20:17:42.984614 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:42.984527 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:43.179464 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:43.174064 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:43.755137 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:43.755102 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:43.913664 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:43.913622 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:43.994432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:43.994402 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:44.005387 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.005298 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv"] Oct 30 20:17:44.006171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.005556 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" podUID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" containerName="istio-proxy" containerID="cri-o://daaec18d4a1bc97f4f8a355474073ef66e2787785128308e8465aaebe009697e" gracePeriod=30 Oct 30 20:17:44.132177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.126625 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:44.331650 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.331564 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:44.567571 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.567540 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:44.569012 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.568755 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:17:44.569012 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.568897 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:44.720819 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.720743 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f"] Oct 30 20:17:44.721509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.721491 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" containerName="istio-proxy" Oct 30 20:17:44.721561 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.721511 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f206f771-c29d-41db-8768-12d2feefe4e3" containerName="istio-proxy" Oct 30 20:17:44.725514 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.725492 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" Oct 30 20:17:44.732819 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.732486 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:17:44.732819 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.732536 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:17:44.737743 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.737718 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f"] Oct 30 20:17:44.789507 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.789476 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ab86d3-0a1e-44b3-8456-be82ab10e104-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f\" (UID: \"75ab86d3-0a1e-44b3-8456-be82ab10e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" Oct 30 20:17:44.789634 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.789531 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ab86d3-0a1e-44b3-8456-be82ab10e104-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f\" (UID: \"75ab86d3-0a1e-44b3-8456-be82ab10e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" Oct 30 20:17:44.789688 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.789665 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxwcd\" (UniqueName: \"kubernetes.io/projected/75ab86d3-0a1e-44b3-8456-be82ab10e104-kube-api-access-qxwcd\") pod \"maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f\" (UID: \"75ab86d3-0a1e-44b3-8456-be82ab10e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" Oct 30 20:17:44.891013 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.890977 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxwcd\" (UniqueName: \"kubernetes.io/projected/75ab86d3-0a1e-44b3-8456-be82ab10e104-kube-api-access-qxwcd\") pod \"maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f\" (UID: \"75ab86d3-0a1e-44b3-8456-be82ab10e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" Oct 30 20:17:44.891297 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.891280 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ab86d3-0a1e-44b3-8456-be82ab10e104-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f\" (UID: \"75ab86d3-0a1e-44b3-8456-be82ab10e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" Oct 30 20:17:44.891374 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.891345 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ab86d3-0a1e-44b3-8456-be82ab10e104-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f\" (UID: \"75ab86d3-0a1e-44b3-8456-be82ab10e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" Oct 30 20:17:44.891922 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.891898 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ab86d3-0a1e-44b3-8456-be82ab10e104-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f\" (UID: \"75ab86d3-0a1e-44b3-8456-be82ab10e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" Oct 30 20:17:44.893837 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.893814 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ab86d3-0a1e-44b3-8456-be82ab10e104-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f\" (UID: \"75ab86d3-0a1e-44b3-8456-be82ab10e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" Oct 30 20:17:44.910286 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:44.910263 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxwcd\" (UniqueName: \"kubernetes.io/projected/75ab86d3-0a1e-44b3-8456-be82ab10e104-kube-api-access-qxwcd\") pod \"maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f\" (UID: \"75ab86d3-0a1e-44b3-8456-be82ab10e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" Oct 30 20:17:45.043749 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:45.043710 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" Oct 30 20:17:45.066331 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:45.066296 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:45.212799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:45.212765 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:45.473247 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:45.465713 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f"] Oct 30 20:17:45.557221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:45.553818 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:45.573860 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:45.573136 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9"] Oct 30 20:17:45.579684 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:45.579079 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-54d4c4-pfgx9"] Oct 30 20:17:45.584949 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:45.584913 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" event={"ID":"75ab86d3-0a1e-44b3-8456-be82ab10e104","Type":"ContainerStarted","Data":"0808a28c8a168c6301ff9b6bfc2d89404696f3438dd6b32339bfec48cd71c79e"} Oct 30 20:17:45.734213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:45.734108 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:45.977590 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:45.977552 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:17:46.111061 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.111022 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:46.385977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.385877 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:46.410099 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.410063 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl"] Oct 30 20:17:46.415160 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.415121 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" Oct 30 20:17:46.430355 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.428582 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-89c124\"" Oct 30 20:17:46.430355 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.428764 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-89c124-dockercfg-kclc7\"" Oct 30 20:17:46.444866 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.444801 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85571ef8-1def-4ecc-af2a-39168d653b2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl\" (UID: \"85571ef8-1def-4ecc-af2a-39168d653b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" Oct 30 20:17:46.445022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.444891 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqlft\" (UniqueName: \"kubernetes.io/projected/85571ef8-1def-4ecc-af2a-39168d653b2e-kube-api-access-pqlft\") pod \"payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl\" (UID: \"85571ef8-1def-4ecc-af2a-39168d653b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" Oct 30 20:17:46.457762 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.457719 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9c13df5-9523-4987-a4e5-b4a678050ba8" path="/var/lib/kubelet/pods/a9c13df5-9523-4987-a4e5-b4a678050ba8/volumes" Oct 30 20:17:46.458319 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.458291 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl"] Oct 30 20:17:46.472871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.472842 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw"] Oct 30 20:17:46.509483 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.509449 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" Oct 30 20:17:46.624167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.565673 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:46.624167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.605791 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqlft\" (UniqueName: \"kubernetes.io/projected/85571ef8-1def-4ecc-af2a-39168d653b2e-kube-api-access-pqlft\") pod \"payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl\" (UID: \"85571ef8-1def-4ecc-af2a-39168d653b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" Oct 30 20:17:46.624167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.606036 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v47q\" (UniqueName: \"kubernetes.io/projected/506a22bb-16c2-4cc8-bbfe-6a9586f1d434-kube-api-access-6v47q\") pod \"payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw\" (UID: \"506a22bb-16c2-4cc8-bbfe-6a9586f1d434\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" Oct 30 20:17:46.624167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.606286 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85571ef8-1def-4ecc-af2a-39168d653b2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl\" (UID: \"85571ef8-1def-4ecc-af2a-39168d653b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" Oct 30 20:17:46.624167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.606326 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/506a22bb-16c2-4cc8-bbfe-6a9586f1d434-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw\" (UID: \"506a22bb-16c2-4cc8-bbfe-6a9586f1d434\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" Oct 30 20:17:46.624167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.608700 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85571ef8-1def-4ecc-af2a-39168d653b2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl\" (UID: \"85571ef8-1def-4ecc-af2a-39168d653b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" Oct 30 20:17:46.666504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.665836 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw"] Oct 30 20:17:46.730440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.730217 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqlft\" (UniqueName: \"kubernetes.io/projected/85571ef8-1def-4ecc-af2a-39168d653b2e-kube-api-access-pqlft\") pod \"payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl\" (UID: \"85571ef8-1def-4ecc-af2a-39168d653b2e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" Oct 30 20:17:46.733126 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.733087 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/506a22bb-16c2-4cc8-bbfe-6a9586f1d434-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw\" (UID: \"506a22bb-16c2-4cc8-bbfe-6a9586f1d434\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" Oct 30 20:17:46.733420 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.733396 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v47q\" (UniqueName: \"kubernetes.io/projected/506a22bb-16c2-4cc8-bbfe-6a9586f1d434-kube-api-access-6v47q\") pod \"payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw\" (UID: \"506a22bb-16c2-4cc8-bbfe-6a9586f1d434\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" Oct 30 20:17:46.740241 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.740132 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" event={"ID":"75ab86d3-0a1e-44b3-8456-be82ab10e104","Type":"ContainerStarted","Data":"c6ac0eb923175b8937d85b657204ca3b938bb7deae5f5b2fdcc7061efbb41deb"} Oct 30 20:17:46.740458 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.740439 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" Oct 30 20:17:46.741119 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.741091 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/506a22bb-16c2-4cc8-bbfe-6a9586f1d434-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw\" (UID: \"506a22bb-16c2-4cc8-bbfe-6a9586f1d434\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" Oct 30 20:17:46.751648 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.751614 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v47q\" (UniqueName: \"kubernetes.io/projected/506a22bb-16c2-4cc8-bbfe-6a9586f1d434-kube-api-access-6v47q\") pod \"payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw\" (UID: \"506a22bb-16c2-4cc8-bbfe-6a9586f1d434\") " pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" Oct 30 20:17:46.925233 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:46.917039 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:47.036055 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:17:47.036024 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85571ef8_1def_4ecc_af2a_39168d653b2e.slice/crio-d7acd9ea1ea4b4f09f687a03674681df45fe45d5837a6f74abf82898bea7bd63 WatchSource:0}: Error finding container d7acd9ea1ea4b4f09f687a03674681df45fe45d5837a6f74abf82898bea7bd63: Status 404 returned error can't find the container with id d7acd9ea1ea4b4f09f687a03674681df45fe45d5837a6f74abf82898bea7bd63 Oct 30 20:17:47.036524 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:47.036497 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl"] Oct 30 20:17:47.039195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:47.039174 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" Oct 30 20:17:47.347320 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:17:47.347283 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506a22bb_16c2_4cc8_bbfe_6a9586f1d434.slice/crio-8ff4ba3308de656c30fe8789c51f4cfc3cf14818f51349822eae6281a1ec156b WatchSource:0}: Error finding container 8ff4ba3308de656c30fe8789c51f4cfc3cf14818f51349822eae6281a1ec156b: Status 404 returned error can't find the container with id 8ff4ba3308de656c30fe8789c51f4cfc3cf14818f51349822eae6281a1ec156b Oct 30 20:17:47.348893 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:47.348854 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw"] Oct 30 20:17:47.680920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:47.680831 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:47.748846 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:47.748812 5091 generic.go:356] "Generic (PLEG): container finished" podID="75ab86d3-0a1e-44b3-8456-be82ab10e104" containerID="c6ac0eb923175b8937d85b657204ca3b938bb7deae5f5b2fdcc7061efbb41deb" exitCode=0 Oct 30 20:17:47.749031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:47.748860 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" event={"ID":"75ab86d3-0a1e-44b3-8456-be82ab10e104","Type":"ContainerDied","Data":"c6ac0eb923175b8937d85b657204ca3b938bb7deae5f5b2fdcc7061efbb41deb"} Oct 30 20:17:47.750862 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:47.750838 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" event={"ID":"85571ef8-1def-4ecc-af2a-39168d653b2e","Type":"ContainerStarted","Data":"0239af929a749312bd3456b8c545afbe6f79f017faa29ab5a7a5fed9791d234a"} Oct 30 20:17:47.750985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:47.750879 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" event={"ID":"85571ef8-1def-4ecc-af2a-39168d653b2e","Type":"ContainerStarted","Data":"d7acd9ea1ea4b4f09f687a03674681df45fe45d5837a6f74abf82898bea7bd63"} Oct 30 20:17:47.750985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:47.750919 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" Oct 30 20:17:47.752786 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:47.752759 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" event={"ID":"506a22bb-16c2-4cc8-bbfe-6a9586f1d434","Type":"ContainerStarted","Data":"337da125b3fb23bcd6cd8744e260f800e97a9d3e59b8df408ec5a4ca2436d2d8"} Oct 30 20:17:47.752913 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:47.752804 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" event={"ID":"506a22bb-16c2-4cc8-bbfe-6a9586f1d434","Type":"ContainerStarted","Data":"8ff4ba3308de656c30fe8789c51f4cfc3cf14818f51349822eae6281a1ec156b"} Oct 30 20:17:47.752913 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:47.752858 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" Oct 30 20:17:47.773303 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:47.773249 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" podStartSLOduration=1.773230162 podStartE2EDuration="1.773230162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:47.772759075 +0000 UTC m=+2787.467967029" watchObservedRunningTime="2026-10-30 20:17:47.773230162 +0000 UTC m=+2787.468438119" Oct 30 20:17:47.798916 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:47.798860 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" podStartSLOduration=1.798845448 podStartE2EDuration="1.798845448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:47.794480034 +0000 UTC m=+2787.489687992" watchObservedRunningTime="2026-10-30 20:17:47.798845448 +0000 UTC m=+2787.494053440" Oct 30 20:17:47.952235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:47.949928 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:48.064277 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.064242 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt"] Oct 30 20:17:48.109742 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.109706 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:48.145435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.141692 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" Oct 30 20:17:48.173172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.171407 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-779fa3-4rknt"] Oct 30 20:17:48.273828 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.273782 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75ab86d3-0a1e-44b3-8456-be82ab10e104-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ab86d3-0a1e-44b3-8456-be82ab10e104-maas-api-xks-ca\") pod \"75ab86d3-0a1e-44b3-8456-be82ab10e104\" (UID: \"75ab86d3-0a1e-44b3-8456-be82ab10e104\") " Oct 30 20:17:48.274022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.273877 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ab86d3-0a1e-44b3-8456-be82ab10e104-kube-api-access-qxwcd\" (UniqueName: \"kubernetes.io/projected/75ab86d3-0a1e-44b3-8456-be82ab10e104-kube-api-access-qxwcd\") pod \"75ab86d3-0a1e-44b3-8456-be82ab10e104\" (UID: \"75ab86d3-0a1e-44b3-8456-be82ab10e104\") " Oct 30 20:17:48.274022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.273991 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ab86d3-0a1e-44b3-8456-be82ab10e104-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ab86d3-0a1e-44b3-8456-be82ab10e104-maas-api-openshift-ca\") pod \"75ab86d3-0a1e-44b3-8456-be82ab10e104\" (UID: \"75ab86d3-0a1e-44b3-8456-be82ab10e104\") " Oct 30 20:17:48.274432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.274388 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ab86d3-0a1e-44b3-8456-be82ab10e104-maas-api-openshift-ca" pod "75ab86d3-0a1e-44b3-8456-be82ab10e104" (UID: "75ab86d3-0a1e-44b3-8456-be82ab10e104"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:48.274860 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.274839 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75ab86d3-0a1e-44b3-8456-be82ab10e104-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:48.276645 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.276621 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75ab86d3-0a1e-44b3-8456-be82ab10e104-maas-api-xks-ca" pod "75ab86d3-0a1e-44b3-8456-be82ab10e104" (UID: "75ab86d3-0a1e-44b3-8456-be82ab10e104"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:48.276731 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.276664 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ab86d3-0a1e-44b3-8456-be82ab10e104-kube-api-access-qxwcd" pod "75ab86d3-0a1e-44b3-8456-be82ab10e104" (UID: "75ab86d3-0a1e-44b3-8456-be82ab10e104"). InnerVolumeSpecName "kube-api-access-qxwcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:48.376045 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.376006 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75ab86d3-0a1e-44b3-8456-be82ab10e104-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:48.376045 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.376042 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxwcd\" (UniqueName: \"kubernetes.io/projected/75ab86d3-0a1e-44b3-8456-be82ab10e104-kube-api-access-qxwcd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:48.443812 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.443776 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed41553a-05cb-4bd6-9824-2fb89be87afa" path="/var/lib/kubelet/pods/ed41553a-05cb-4bd6-9824-2fb89be87afa/volumes" Oct 30 20:17:48.505655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.505619 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:48.764188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.764128 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" event={"ID":"75ab86d3-0a1e-44b3-8456-be82ab10e104","Type":"ContainerDied","Data":"0808a28c8a168c6301ff9b6bfc2d89404696f3438dd6b32339bfec48cd71c79e"} Oct 30 20:17:48.764388 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.764200 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0808a28c8a168c6301ff9b6bfc2d89404696f3438dd6b32339bfec48cd71c79e" Oct 30 20:17:48.764388 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:48.764141 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f" Oct 30 20:17:49.208647 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.208616 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:49.307789 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.307764 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 20:17:49.395184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.395132 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-token\" (UniqueName: \"kubernetes.io/projected/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-token\") pod \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " Oct 30 20:17:49.395678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.395213 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-workload-socket\") pod \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " Oct 30 20:17:49.395678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.395327 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-workload-certs\") pod \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " Oct 30 20:17:49.395678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.395403 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-credential-socket\") pod \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " Oct 30 20:17:49.395678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.395461 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d750d29-4fd8-4bd5-b879-ff33ba319afc-kube-api-access-9wbmr\" (UniqueName: \"kubernetes.io/projected/9d750d29-4fd8-4bd5-b879-ff33ba319afc-kube-api-access-9wbmr\") pod \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " Oct 30 20:17:49.395678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.395519 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-envoy\") pod \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " Oct 30 20:17:49.395678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.395559 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istiod-ca-cert\") pod \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " Oct 30 20:17:49.395678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.395577 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-podinfo\") pod \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " Oct 30 20:17:49.396093 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.395665 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-credential-socket" pod "9d750d29-4fd8-4bd5-b879-ff33ba319afc" (UID: "9d750d29-4fd8-4bd5-b879-ff33ba319afc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:49.396093 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.395725 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-data\") pod \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\" (UID: \"9d750d29-4fd8-4bd5-b879-ff33ba319afc\") " Oct 30 20:17:49.396093 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.395743 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-workload-certs" pod "9d750d29-4fd8-4bd5-b879-ff33ba319afc" (UID: "9d750d29-4fd8-4bd5-b879-ff33ba319afc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:49.396093 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.396025 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istiod-ca-cert" pod "9d750d29-4fd8-4bd5-b879-ff33ba319afc" (UID: "9d750d29-4fd8-4bd5-b879-ff33ba319afc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:49.396385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.396103 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-workload-socket" pod "9d750d29-4fd8-4bd5-b879-ff33ba319afc" (UID: "9d750d29-4fd8-4bd5-b879-ff33ba319afc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:49.396385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.396283 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-data" pod "9d750d29-4fd8-4bd5-b879-ff33ba319afc" (UID: "9d750d29-4fd8-4bd5-b879-ff33ba319afc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:49.396623 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.396574 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:49.396623 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.396609 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:49.396839 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.396627 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:49.396839 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.396642 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:49.396839 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.396657 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:49.398355 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.398325 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-podinfo" pod "9d750d29-4fd8-4bd5-b879-ff33ba319afc" (UID: "9d750d29-4fd8-4bd5-b879-ff33ba319afc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:17:49.398463 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.398413 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-token" pod "9d750d29-4fd8-4bd5-b879-ff33ba319afc" (UID: "9d750d29-4fd8-4bd5-b879-ff33ba319afc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:49.398463 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.398420 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d750d29-4fd8-4bd5-b879-ff33ba319afc-kube-api-access-9wbmr" pod "9d750d29-4fd8-4bd5-b879-ff33ba319afc" (UID: "9d750d29-4fd8-4bd5-b879-ff33ba319afc"). InnerVolumeSpecName "kube-api-access-9wbmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:49.398562 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.398549 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-envoy" pod "9d750d29-4fd8-4bd5-b879-ff33ba319afc" (UID: "9d750d29-4fd8-4bd5-b879-ff33ba319afc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:49.498303 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.498274 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wbmr\" (UniqueName: \"kubernetes.io/projected/9d750d29-4fd8-4bd5-b879-ff33ba319afc-kube-api-access-9wbmr\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:49.498303 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.498302 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:49.498493 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.498313 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:49.498493 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.498322 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d750d29-4fd8-4bd5-b879-ff33ba319afc-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:49.613064 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.612977 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld"] Oct 30 20:17:49.736174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.735289 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:49.775256 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.775220 5091 generic.go:356] "Generic (PLEG): container finished" podID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" containerID="daaec18d4a1bc97f4f8a355474073ef66e2787785128308e8465aaebe009697e" exitCode=0 Oct 30 20:17:49.775458 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.775280 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" Oct 30 20:17:49.775458 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.775299 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" event={"ID":"9d750d29-4fd8-4bd5-b879-ff33ba319afc","Type":"ContainerDied","Data":"daaec18d4a1bc97f4f8a355474073ef66e2787785128308e8465aaebe009697e"} Oct 30 20:17:49.775458 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.775390 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv" event={"ID":"9d750d29-4fd8-4bd5-b879-ff33ba319afc","Type":"ContainerDied","Data":"3aa850ecf43c5af6c4530114576fb562aa96bb53627dd78a2b012385ac112e65"} Oct 30 20:17:49.775458 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.775421 5091 scope.go:122] "RemoveContainer" containerID="daaec18d4a1bc97f4f8a355474073ef66e2787785128308e8465aaebe009697e" Oct 30 20:17:49.787992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.787973 5091 scope.go:122] "RemoveContainer" containerID="03b46fa6422e9f1fa0bfffa7e75b2f5fb1197c945e6d84456ea2933b605dae84" Oct 30 20:17:49.798090 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.798070 5091 scope.go:122] "RemoveContainer" containerID="daaec18d4a1bc97f4f8a355474073ef66e2787785128308e8465aaebe009697e" Oct 30 20:17:49.798356 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:49.798334 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daaec18d4a1bc97f4f8a355474073ef66e2787785128308e8465aaebe009697e\": container with ID starting with daaec18d4a1bc97f4f8a355474073ef66e2787785128308e8465aaebe009697e not found: ID does not exist" containerID="daaec18d4a1bc97f4f8a355474073ef66e2787785128308e8465aaebe009697e" Oct 30 20:17:49.798430 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.798365 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daaec18d4a1bc97f4f8a355474073ef66e2787785128308e8465aaebe009697e"} err="failed to get container status \"daaec18d4a1bc97f4f8a355474073ef66e2787785128308e8465aaebe009697e\": rpc error: code = NotFound desc = could not find container \"daaec18d4a1bc97f4f8a355474073ef66e2787785128308e8465aaebe009697e\": container with ID starting with daaec18d4a1bc97f4f8a355474073ef66e2787785128308e8465aaebe009697e not found: ID does not exist" Oct 30 20:17:49.798430 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.798383 5091 scope.go:122] "RemoveContainer" containerID="03b46fa6422e9f1fa0bfffa7e75b2f5fb1197c945e6d84456ea2933b605dae84" Oct 30 20:17:49.798654 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:49.798634 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03b46fa6422e9f1fa0bfffa7e75b2f5fb1197c945e6d84456ea2933b605dae84\": container with ID starting with 03b46fa6422e9f1fa0bfffa7e75b2f5fb1197c945e6d84456ea2933b605dae84 not found: ID does not exist" containerID="03b46fa6422e9f1fa0bfffa7e75b2f5fb1197c945e6d84456ea2933b605dae84" Oct 30 20:17:49.798701 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.798663 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03b46fa6422e9f1fa0bfffa7e75b2f5fb1197c945e6d84456ea2933b605dae84"} err="failed to get container status \"03b46fa6422e9f1fa0bfffa7e75b2f5fb1197c945e6d84456ea2933b605dae84\": rpc error: code = NotFound desc = could not find container \"03b46fa6422e9f1fa0bfffa7e75b2f5fb1197c945e6d84456ea2933b605dae84\": container with ID starting with 03b46fa6422e9f1fa0bfffa7e75b2f5fb1197c945e6d84456ea2933b605dae84 not found: ID does not exist" Oct 30 20:17:49.818780 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.818755 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv"] Oct 30 20:17:49.830868 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:49.830834 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7cbcbb-openshift-default-666c99dff9-k84lv"] Oct 30 20:17:50.270180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:50.270021 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:50.446164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:50.445608 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" path="/var/lib/kubelet/pods/9d750d29-4fd8-4bd5-b879-ff33ba319afc/volumes" Oct 30 20:17:51.132204 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:51.132124 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:52.034224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:52.027814 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:52.797901 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:52.797864 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:53.316901 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:53.316866 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:53.469780 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:53.469740 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:53.599845 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:53.599762 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m"] Oct 30 20:17:53.600037 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:53.600006 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" podUID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerName="istio-proxy" containerID="cri-o://8a34b82f37a51d39c907a01d4354f8fe58d3a160a7cf56f98e696787ace7a106" gracePeriod=30 Oct 30 20:17:53.673981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:53.673943 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:53.831713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:53.828605 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:54.214863 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.214822 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:54.332176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.332122 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz"] Oct 30 20:17:54.333679 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.333645 5091 cpu_manager.go:414] "containerMap: removing container" podUID="75ab86d3-0a1e-44b3-8456-be82ab10e104" containerName="revoke-keys" Oct 30 20:17:54.333679 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.333677 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ab86d3-0a1e-44b3-8456-be82ab10e104" containerName="revoke-keys" Oct 30 20:17:54.333874 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.333750 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" containerName="istio-proxy" Oct 30 20:17:54.333874 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.333761 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" containerName="istio-proxy" Oct 30 20:17:54.333874 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.333838 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" containerName="istio-proxy" Oct 30 20:17:54.333874 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.333846 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" containerName="istio-proxy" Oct 30 20:17:54.334197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.334177 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" containerName="istio-proxy" Oct 30 20:17:54.334277 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.334214 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ab86d3-0a1e-44b3-8456-be82ab10e104" containerName="revoke-keys" Oct 30 20:17:54.341237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.341209 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" Oct 30 20:17:54.344505 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.344467 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz"] Oct 30 20:17:54.373196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.350770 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:17:54.373196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.354101 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:17:54.397047 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.397009 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d4e116-fd32-41e2-a463-b1ff1a023828-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz\" (UID: \"09d4e116-fd32-41e2-a463-b1ff1a023828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" Oct 30 20:17:54.397243 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.397167 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs4fd\" (UniqueName: \"kubernetes.io/projected/09d4e116-fd32-41e2-a463-b1ff1a023828-kube-api-access-vs4fd\") pod \"maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz\" (UID: \"09d4e116-fd32-41e2-a463-b1ff1a023828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" Oct 30 20:17:54.397458 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.397431 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d4e116-fd32-41e2-a463-b1ff1a023828-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz\" (UID: \"09d4e116-fd32-41e2-a463-b1ff1a023828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" Oct 30 20:17:54.499248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.499217 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d4e116-fd32-41e2-a463-b1ff1a023828-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz\" (UID: \"09d4e116-fd32-41e2-a463-b1ff1a023828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" Oct 30 20:17:54.499440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.499420 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d4e116-fd32-41e2-a463-b1ff1a023828-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz\" (UID: \"09d4e116-fd32-41e2-a463-b1ff1a023828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" Oct 30 20:17:54.499549 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.499529 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs4fd\" (UniqueName: \"kubernetes.io/projected/09d4e116-fd32-41e2-a463-b1ff1a023828-kube-api-access-vs4fd\") pod \"maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz\" (UID: \"09d4e116-fd32-41e2-a463-b1ff1a023828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" Oct 30 20:17:54.500113 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.500090 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d4e116-fd32-41e2-a463-b1ff1a023828-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz\" (UID: \"09d4e116-fd32-41e2-a463-b1ff1a023828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" Oct 30 20:17:54.501783 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.501765 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d4e116-fd32-41e2-a463-b1ff1a023828-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz\" (UID: \"09d4e116-fd32-41e2-a463-b1ff1a023828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" Oct 30 20:17:54.513830 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.513806 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs4fd\" (UniqueName: \"kubernetes.io/projected/09d4e116-fd32-41e2-a463-b1ff1a023828-kube-api-access-vs4fd\") pod \"maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz\" (UID: \"09d4e116-fd32-41e2-a463-b1ff1a023828\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" Oct 30 20:17:54.662013 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.661969 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" Oct 30 20:17:54.838348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.838134 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:54.897339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.897300 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74"] Oct 30 20:17:54.908711 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.908667 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm"] Oct 30 20:17:54.970838 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:54.970809 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz"] Oct 30 20:17:54.972777 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:17:54.972750 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d4e116_fd32_41e2_a463_b1ff1a023828.slice/crio-10341210594e35847abde8c32227c2514aac8be2d8f27132f7cf55b8eaa4178f WatchSource:0}: Error finding container 10341210594e35847abde8c32227c2514aac8be2d8f27132f7cf55b8eaa4178f: Status 404 returned error can't find the container with id 10341210594e35847abde8c32227c2514aac8be2d8f27132f7cf55b8eaa4178f Oct 30 20:17:55.014123 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:55.014092 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:55.059429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:55.059389 5091 patch_prober.go:29] interesting pod/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:17:55.059598 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:55.059456 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" podUID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:17:55.181321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:55.181279 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:55.803139 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:55.798625 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:55.911230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:55.911191 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" event={"ID":"09d4e116-fd32-41e2-a463-b1ff1a023828","Type":"ContainerStarted","Data":"2d8f71f98dfedac8f0fc007bf784fa01416e8f2e5d0973d333ba1f26ecdc5a89"} Oct 30 20:17:55.911417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:55.911252 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" event={"ID":"09d4e116-fd32-41e2-a463-b1ff1a023828","Type":"ContainerStarted","Data":"10341210594e35847abde8c32227c2514aac8be2d8f27132f7cf55b8eaa4178f"} Oct 30 20:17:55.953892 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:55.953841 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" podStartSLOduration=1.953827613 podStartE2EDuration="1.953827613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:55.952954618 +0000 UTC m=+2795.648162569" watchObservedRunningTime="2026-10-30 20:17:55.953827613 +0000 UTC m=+2795.649035570" Oct 30 20:17:56.318784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:56.318691 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:56.552295 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:56.552260 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq"] Oct 30 20:17:56.552545 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:56.552513 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" podUID="872504eb-20ef-411c-b526-61e05a15a9c4" containerName="istio-proxy" containerID="cri-o://b80337cf3ed50a192c528978b244cd2870ec86fa9a3c2958258414a5ad2cc0ef" gracePeriod=30 Oct 30 20:17:56.561046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:56.561017 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:56.732231 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:56.730075 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:56.935968 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:56.935922 5091 generic.go:356] "Generic (PLEG): container finished" podID="09d4e116-fd32-41e2-a463-b1ff1a023828" containerID="2d8f71f98dfedac8f0fc007bf784fa01416e8f2e5d0973d333ba1f26ecdc5a89" exitCode=0 Oct 30 20:17:56.936501 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:56.935975 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" event={"ID":"09d4e116-fd32-41e2-a463-b1ff1a023828","Type":"ContainerDied","Data":"2d8f71f98dfedac8f0fc007bf784fa01416e8f2e5d0973d333ba1f26ecdc5a89"} Oct 30 20:17:56.949060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:56.949017 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:57.109352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.109311 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:57.222465 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.222425 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59"] Oct 30 20:17:57.223262 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.223246 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d750d29-4fd8-4bd5-b879-ff33ba319afc" containerName="istio-proxy" Oct 30 20:17:57.226952 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.226936 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" Oct 30 20:17:57.238806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.238781 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59"] Oct 30 20:17:57.263004 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.262976 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11dc06d8-aaf3-49bb-a6e9-3120b363a462-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59\" (UID: \"11dc06d8-aaf3-49bb-a6e9-3120b363a462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" Oct 30 20:17:57.263199 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.263039 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6gbt\" (UniqueName: \"kubernetes.io/projected/11dc06d8-aaf3-49bb-a6e9-3120b363a462-kube-api-access-p6gbt\") pod \"maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59\" (UID: \"11dc06d8-aaf3-49bb-a6e9-3120b363a462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" Oct 30 20:17:57.263397 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.263365 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11dc06d8-aaf3-49bb-a6e9-3120b363a462-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59\" (UID: \"11dc06d8-aaf3-49bb-a6e9-3120b363a462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" Oct 30 20:17:57.364344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.364259 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6gbt\" (UniqueName: \"kubernetes.io/projected/11dc06d8-aaf3-49bb-a6e9-3120b363a462-kube-api-access-p6gbt\") pod \"maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59\" (UID: \"11dc06d8-aaf3-49bb-a6e9-3120b363a462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" Oct 30 20:17:57.364545 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.364519 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11dc06d8-aaf3-49bb-a6e9-3120b363a462-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59\" (UID: \"11dc06d8-aaf3-49bb-a6e9-3120b363a462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" Oct 30 20:17:57.364655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.364637 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11dc06d8-aaf3-49bb-a6e9-3120b363a462-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59\" (UID: \"11dc06d8-aaf3-49bb-a6e9-3120b363a462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" Oct 30 20:17:57.365354 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.365327 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11dc06d8-aaf3-49bb-a6e9-3120b363a462-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59\" (UID: \"11dc06d8-aaf3-49bb-a6e9-3120b363a462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" Oct 30 20:17:57.367529 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.367504 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11dc06d8-aaf3-49bb-a6e9-3120b363a462-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59\" (UID: \"11dc06d8-aaf3-49bb-a6e9-3120b363a462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" Oct 30 20:17:57.378332 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.378307 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6gbt\" (UniqueName: \"kubernetes.io/projected/11dc06d8-aaf3-49bb-a6e9-3120b363a462-kube-api-access-p6gbt\") pod \"maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59\" (UID: \"11dc06d8-aaf3-49bb-a6e9-3120b363a462\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" Oct 30 20:17:57.544692 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.544654 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" Oct 30 20:17:57.832668 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:17:57.832632 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11dc06d8_aaf3_49bb_a6e9_3120b363a462.slice/crio-85adf20ce5790c68e8a4a2f7bf3f8b37a78f0fa48ec2fb223c5decfc108c1f23 WatchSource:0}: Error finding container 85adf20ce5790c68e8a4a2f7bf3f8b37a78f0fa48ec2fb223c5decfc108c1f23: Status 404 returned error can't find the container with id 85adf20ce5790c68e8a4a2f7bf3f8b37a78f0fa48ec2fb223c5decfc108c1f23 Oct 30 20:17:57.833060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.833035 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59"] Oct 30 20:17:57.946615 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.946574 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" event={"ID":"11dc06d8-aaf3-49bb-a6e9-3120b363a462","Type":"ContainerStarted","Data":"85adf20ce5790c68e8a4a2f7bf3f8b37a78f0fa48ec2fb223c5decfc108c1f23"} Oct 30 20:17:57.968925 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:57.968888 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:58.276044 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.276014 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" Oct 30 20:17:58.388852 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.388814 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d4e116-fd32-41e2-a463-b1ff1a023828-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d4e116-fd32-41e2-a463-b1ff1a023828-maas-api-openshift-ca\") pod \"09d4e116-fd32-41e2-a463-b1ff1a023828\" (UID: \"09d4e116-fd32-41e2-a463-b1ff1a023828\") " Oct 30 20:17:58.388996 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.388909 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09d4e116-fd32-41e2-a463-b1ff1a023828-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d4e116-fd32-41e2-a463-b1ff1a023828-maas-api-xks-ca\") pod \"09d4e116-fd32-41e2-a463-b1ff1a023828\" (UID: \"09d4e116-fd32-41e2-a463-b1ff1a023828\") " Oct 30 20:17:58.389075 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.389014 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d4e116-fd32-41e2-a463-b1ff1a023828-kube-api-access-vs4fd\" (UniqueName: \"kubernetes.io/projected/09d4e116-fd32-41e2-a463-b1ff1a023828-kube-api-access-vs4fd\") pod \"09d4e116-fd32-41e2-a463-b1ff1a023828\" (UID: \"09d4e116-fd32-41e2-a463-b1ff1a023828\") " Oct 30 20:17:58.389267 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.389235 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d4e116-fd32-41e2-a463-b1ff1a023828-maas-api-openshift-ca" pod "09d4e116-fd32-41e2-a463-b1ff1a023828" (UID: "09d4e116-fd32-41e2-a463-b1ff1a023828"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:58.389721 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.389687 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09d4e116-fd32-41e2-a463-b1ff1a023828-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:58.391604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.391579 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09d4e116-fd32-41e2-a463-b1ff1a023828-maas-api-xks-ca" pod "09d4e116-fd32-41e2-a463-b1ff1a023828" (UID: "09d4e116-fd32-41e2-a463-b1ff1a023828"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:17:58.391867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.391844 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d4e116-fd32-41e2-a463-b1ff1a023828-kube-api-access-vs4fd" pod "09d4e116-fd32-41e2-a463-b1ff1a023828" (UID: "09d4e116-fd32-41e2-a463-b1ff1a023828"). InnerVolumeSpecName "kube-api-access-vs4fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:58.475794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.475759 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:58.491535 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.491508 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09d4e116-fd32-41e2-a463-b1ff1a023828-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:58.491535 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.491534 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs4fd\" (UniqueName: \"kubernetes.io/projected/09d4e116-fd32-41e2-a463-b1ff1a023828-kube-api-access-vs4fd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:58.641262 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.639583 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:58.768304 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.768268 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" Oct 30 20:17:58.769254 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.769233 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" Oct 30 20:17:58.916941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.916910 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 20:17:58.962916 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.962875 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:58.993348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.988741 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" event={"ID":"09d4e116-fd32-41e2-a463-b1ff1a023828","Type":"ContainerDied","Data":"10341210594e35847abde8c32227c2514aac8be2d8f27132f7cf55b8eaa4178f"} Oct 30 20:17:58.993348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.988804 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10341210594e35847abde8c32227c2514aac8be2d8f27132f7cf55b8eaa4178f" Oct 30 20:17:58.993348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.988759 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz" Oct 30 20:17:58.993348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.991641 5091 generic.go:356] "Generic (PLEG): container finished" podID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerID="8a34b82f37a51d39c907a01d4354f8fe58d3a160a7cf56f98e696787ace7a106" exitCode=0 Oct 30 20:17:58.993348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.991791 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" event={"ID":"c222e9d2-821b-4c1c-92d2-88297be8f52d","Type":"ContainerDied","Data":"8a34b82f37a51d39c907a01d4354f8fe58d3a160a7cf56f98e696787ace7a106"} Oct 30 20:17:58.993348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.991836 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" event={"ID":"c222e9d2-821b-4c1c-92d2-88297be8f52d","Type":"ContainerDied","Data":"cbe66a692da1fa9bcf48c8e3d201084a6439889654b2c51eb89e60f56c459e7f"} Oct 30 20:17:58.993348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.991863 5091 scope.go:122] "RemoveContainer" containerID="8a34b82f37a51d39c907a01d4354f8fe58d3a160a7cf56f98e696787ace7a106" Oct 30 20:17:58.993348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:58.992011 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m" Oct 30 20:17:59.003484 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.003448 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" event={"ID":"11dc06d8-aaf3-49bb-a6e9-3120b363a462","Type":"ContainerStarted","Data":"552b6a834aae880c4927eb3a4ad5f5ac76db57f7b85f7d17695e85a43233d80d"} Oct 30 20:17:59.027805 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.027772 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c222e9d2-821b-4c1c-92d2-88297be8f52d-kube-api-access-wj4vd\" (UniqueName: \"kubernetes.io/projected/c222e9d2-821b-4c1c-92d2-88297be8f52d-kube-api-access-wj4vd\") pod \"c222e9d2-821b-4c1c-92d2-88297be8f52d\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " Oct 30 20:17:59.027964 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.027826 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-envoy\") pod \"c222e9d2-821b-4c1c-92d2-88297be8f52d\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " Oct 30 20:17:59.027964 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.027868 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-workload-socket\") pod \"c222e9d2-821b-4c1c-92d2-88297be8f52d\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " Oct 30 20:17:59.028100 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.027962 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-workload-certs\") pod \"c222e9d2-821b-4c1c-92d2-88297be8f52d\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " Oct 30 20:17:59.028100 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.027996 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-token\" (UniqueName: \"kubernetes.io/projected/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-token\") pod \"c222e9d2-821b-4c1c-92d2-88297be8f52d\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " Oct 30 20:17:59.028234 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.028127 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c222e9d2-821b-4c1c-92d2-88297be8f52d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c222e9d2-821b-4c1c-92d2-88297be8f52d-istiod-ca-cert\") pod \"c222e9d2-821b-4c1c-92d2-88297be8f52d\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " Oct 30 20:17:59.028291 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.028265 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-data\") pod \"c222e9d2-821b-4c1c-92d2-88297be8f52d\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " Oct 30 20:17:59.028353 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.028302 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-podinfo\") pod \"c222e9d2-821b-4c1c-92d2-88297be8f52d\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " Oct 30 20:17:59.028450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.028435 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-credential-socket\") pod \"c222e9d2-821b-4c1c-92d2-88297be8f52d\" (UID: \"c222e9d2-821b-4c1c-92d2-88297be8f52d\") " Oct 30 20:17:59.029361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.028899 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-data" pod "c222e9d2-821b-4c1c-92d2-88297be8f52d" (UID: "c222e9d2-821b-4c1c-92d2-88297be8f52d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:59.029361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.029107 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-workload-certs" pod "c222e9d2-821b-4c1c-92d2-88297be8f52d" (UID: "c222e9d2-821b-4c1c-92d2-88297be8f52d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:59.029361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.029170 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:59.029361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.029275 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-workload-socket" pod "c222e9d2-821b-4c1c-92d2-88297be8f52d" (UID: "c222e9d2-821b-4c1c-92d2-88297be8f52d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:59.029361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.029280 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c222e9d2-821b-4c1c-92d2-88297be8f52d-istiod-ca-cert" pod "c222e9d2-821b-4c1c-92d2-88297be8f52d" (UID: "c222e9d2-821b-4c1c-92d2-88297be8f52d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:17:59.029361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.029338 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-credential-socket" pod "c222e9d2-821b-4c1c-92d2-88297be8f52d" (UID: "c222e9d2-821b-4c1c-92d2-88297be8f52d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:59.031627 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.031598 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c222e9d2-821b-4c1c-92d2-88297be8f52d-kube-api-access-wj4vd" pod "c222e9d2-821b-4c1c-92d2-88297be8f52d" (UID: "c222e9d2-821b-4c1c-92d2-88297be8f52d"). InnerVolumeSpecName "kube-api-access-wj4vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:59.032443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.032398 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-podinfo" pod "c222e9d2-821b-4c1c-92d2-88297be8f52d" (UID: "c222e9d2-821b-4c1c-92d2-88297be8f52d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:17:59.034528 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.034483 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" podStartSLOduration=2.03447053 podStartE2EDuration="2.03447053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:17:59.030618391 +0000 UTC m=+2798.725826350" watchObservedRunningTime="2026-10-30 20:17:59.03447053 +0000 UTC m=+2798.729678503" Oct 30 20:17:59.035557 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.035528 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-envoy" pod "c222e9d2-821b-4c1c-92d2-88297be8f52d" (UID: "c222e9d2-821b-4c1c-92d2-88297be8f52d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:17:59.037039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.036998 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-token" pod "c222e9d2-821b-4c1c-92d2-88297be8f52d" (UID: "c222e9d2-821b-4c1c-92d2-88297be8f52d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:17:59.054960 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.054939 5091 scope.go:122] "RemoveContainer" containerID="7f5d4e0b0ede7cef957dc6d33cfe4195cd797972df41d573d4b6d2102d1f7dcc" Oct 30 20:17:59.065523 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.065503 5091 scope.go:122] "RemoveContainer" containerID="8a34b82f37a51d39c907a01d4354f8fe58d3a160a7cf56f98e696787ace7a106" Oct 30 20:17:59.065775 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:59.065757 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a34b82f37a51d39c907a01d4354f8fe58d3a160a7cf56f98e696787ace7a106\": container with ID starting with 8a34b82f37a51d39c907a01d4354f8fe58d3a160a7cf56f98e696787ace7a106 not found: ID does not exist" containerID="8a34b82f37a51d39c907a01d4354f8fe58d3a160a7cf56f98e696787ace7a106" Oct 30 20:17:59.065832 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.065784 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a34b82f37a51d39c907a01d4354f8fe58d3a160a7cf56f98e696787ace7a106"} err="failed to get container status \"8a34b82f37a51d39c907a01d4354f8fe58d3a160a7cf56f98e696787ace7a106\": rpc error: code = NotFound desc = could not find container \"8a34b82f37a51d39c907a01d4354f8fe58d3a160a7cf56f98e696787ace7a106\": container with ID starting with 8a34b82f37a51d39c907a01d4354f8fe58d3a160a7cf56f98e696787ace7a106 not found: ID does not exist" Oct 30 20:17:59.065832 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.065803 5091 scope.go:122] "RemoveContainer" containerID="7f5d4e0b0ede7cef957dc6d33cfe4195cd797972df41d573d4b6d2102d1f7dcc" Oct 30 20:17:59.066017 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:17:59.065996 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f5d4e0b0ede7cef957dc6d33cfe4195cd797972df41d573d4b6d2102d1f7dcc\": container with ID starting with 7f5d4e0b0ede7cef957dc6d33cfe4195cd797972df41d573d4b6d2102d1f7dcc not found: ID does not exist" containerID="7f5d4e0b0ede7cef957dc6d33cfe4195cd797972df41d573d4b6d2102d1f7dcc" Oct 30 20:17:59.066057 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.066024 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f5d4e0b0ede7cef957dc6d33cfe4195cd797972df41d573d4b6d2102d1f7dcc"} err="failed to get container status \"7f5d4e0b0ede7cef957dc6d33cfe4195cd797972df41d573d4b6d2102d1f7dcc\": rpc error: code = NotFound desc = could not find container \"7f5d4e0b0ede7cef957dc6d33cfe4195cd797972df41d573d4b6d2102d1f7dcc\": container with ID starting with 7f5d4e0b0ede7cef957dc6d33cfe4195cd797972df41d573d4b6d2102d1f7dcc not found: ID does not exist" Oct 30 20:17:59.131200 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.131171 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:59.131200 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.131198 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:59.131402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.131230 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:59.131402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.131256 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:59.131402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.131267 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c222e9d2-821b-4c1c-92d2-88297be8f52d-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:59.131402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.131276 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c222e9d2-821b-4c1c-92d2-88297be8f52d-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:59.131402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.131289 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c222e9d2-821b-4c1c-92d2-88297be8f52d-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:59.131402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.131300 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wj4vd\" (UniqueName: \"kubernetes.io/projected/c222e9d2-821b-4c1c-92d2-88297be8f52d-kube-api-access-wj4vd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:17:59.203304 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.203259 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:17:59.325776 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.325743 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m"] Oct 30 20:17:59.333235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.333206 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-54d4c4-openshift-default-78b74cbf7-dxq2m"] Oct 30 20:17:59.917166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.912209 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" podUID="f76728a5-e155-435a-a692-cc9afee10063" containerName="payload-processing" containerID="cri-o://da0387542447795de10080e05ff16384c4ec925ebb6021183f68263114349a9a" gracePeriod=25 Oct 30 20:17:59.917166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.912808 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" podUID="02aaf432-1c15-4e88-9978-23dc3f9ee27d" containerName="payload-pre-processing" containerID="cri-o://88fde8d5cb82274a117df04703609db069be952cfb2c7a947c3f76c20d9f8f76" gracePeriod=25 Oct 30 20:17:59.926182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:17:59.922507 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:00.016034 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:00.016000 5091 generic.go:356] "Generic (PLEG): container finished" podID="11dc06d8-aaf3-49bb-a6e9-3120b363a462" containerID="552b6a834aae880c4927eb3a4ad5f5ac76db57f7b85f7d17695e85a43233d80d" exitCode=0 Oct 30 20:18:00.016545 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:00.016100 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" event={"ID":"11dc06d8-aaf3-49bb-a6e9-3120b363a462","Type":"ContainerDied","Data":"552b6a834aae880c4927eb3a4ad5f5ac76db57f7b85f7d17695e85a43233d80d"} Oct 30 20:18:00.444412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:00.444376 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c222e9d2-821b-4c1c-92d2-88297be8f52d" path="/var/lib/kubelet/pods/c222e9d2-821b-4c1c-92d2-88297be8f52d/volumes" Oct 30 20:18:00.828179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:00.828105 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" podUID="02aaf432-1c15-4e88-9978-23dc3f9ee27d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:18:01.068527 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.068403 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:01.235030 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.234978 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:01.268645 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.268616 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" Oct 30 20:18:01.366325 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.366290 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11dc06d8-aaf3-49bb-a6e9-3120b363a462-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11dc06d8-aaf3-49bb-a6e9-3120b363a462-maas-api-openshift-ca\") pod \"11dc06d8-aaf3-49bb-a6e9-3120b363a462\" (UID: \"11dc06d8-aaf3-49bb-a6e9-3120b363a462\") " Oct 30 20:18:01.366531 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.366357 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11dc06d8-aaf3-49bb-a6e9-3120b363a462-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11dc06d8-aaf3-49bb-a6e9-3120b363a462-maas-api-xks-ca\") pod \"11dc06d8-aaf3-49bb-a6e9-3120b363a462\" (UID: \"11dc06d8-aaf3-49bb-a6e9-3120b363a462\") " Oct 30 20:18:01.366531 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.366462 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11dc06d8-aaf3-49bb-a6e9-3120b363a462-kube-api-access-p6gbt\" (UniqueName: \"kubernetes.io/projected/11dc06d8-aaf3-49bb-a6e9-3120b363a462-kube-api-access-p6gbt\") pod \"11dc06d8-aaf3-49bb-a6e9-3120b363a462\" (UID: \"11dc06d8-aaf3-49bb-a6e9-3120b363a462\") " Oct 30 20:18:01.366720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.366698 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11dc06d8-aaf3-49bb-a6e9-3120b363a462-maas-api-openshift-ca" pod "11dc06d8-aaf3-49bb-a6e9-3120b363a462" (UID: "11dc06d8-aaf3-49bb-a6e9-3120b363a462"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:01.367206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.367175 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11dc06d8-aaf3-49bb-a6e9-3120b363a462-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:01.368736 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.368706 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11dc06d8-aaf3-49bb-a6e9-3120b363a462-maas-api-xks-ca" pod "11dc06d8-aaf3-49bb-a6e9-3120b363a462" (UID: "11dc06d8-aaf3-49bb-a6e9-3120b363a462"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:01.368853 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.368836 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dc06d8-aaf3-49bb-a6e9-3120b363a462-kube-api-access-p6gbt" pod "11dc06d8-aaf3-49bb-a6e9-3120b363a462" (UID: "11dc06d8-aaf3-49bb-a6e9-3120b363a462"). InnerVolumeSpecName "kube-api-access-p6gbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:01.398822 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.398751 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:01.468612 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.468568 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11dc06d8-aaf3-49bb-a6e9-3120b363a462-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:01.468612 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.468610 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6gbt\" (UniqueName: \"kubernetes.io/projected/11dc06d8-aaf3-49bb-a6e9-3120b363a462-kube-api-access-p6gbt\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:01.832422 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.832250 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" podUID="f76728a5-e155-435a-a692-cc9afee10063" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:18:01.841839 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.841814 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 20:18:01.989212 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.989090 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-workload-certs\") pod \"872504eb-20ef-411c-b526-61e05a15a9c4\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " Oct 30 20:18:01.989391 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.989215 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872504eb-20ef-411c-b526-61e05a15a9c4-kube-api-access-s4645\" (UniqueName: \"kubernetes.io/projected/872504eb-20ef-411c-b526-61e05a15a9c4-kube-api-access-s4645\") pod \"872504eb-20ef-411c-b526-61e05a15a9c4\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " Oct 30 20:18:01.989391 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.989306 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/872504eb-20ef-411c-b526-61e05a15a9c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872504eb-20ef-411c-b526-61e05a15a9c4-istiod-ca-cert\") pod \"872504eb-20ef-411c-b526-61e05a15a9c4\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " Oct 30 20:18:01.989391 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.989339 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/872504eb-20ef-411c-b526-61e05a15a9c4-istio-token\" (UniqueName: \"kubernetes.io/projected/872504eb-20ef-411c-b526-61e05a15a9c4-istio-token\") pod \"872504eb-20ef-411c-b526-61e05a15a9c4\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " Oct 30 20:18:01.989595 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.989496 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-credential-socket\") pod \"872504eb-20ef-411c-b526-61e05a15a9c4\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " Oct 30 20:18:01.989595 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.989565 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-workload-certs" pod "872504eb-20ef-411c-b526-61e05a15a9c4" (UID: "872504eb-20ef-411c-b526-61e05a15a9c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:01.989716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.989662 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-workload-socket\") pod \"872504eb-20ef-411c-b526-61e05a15a9c4\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " Oct 30 20:18:01.989716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.989699 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-istio-envoy\") pod \"872504eb-20ef-411c-b526-61e05a15a9c4\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " Oct 30 20:18:01.989838 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.989728 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-istio-data\") pod \"872504eb-20ef-411c-b526-61e05a15a9c4\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " Oct 30 20:18:01.989838 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.989798 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/872504eb-20ef-411c-b526-61e05a15a9c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872504eb-20ef-411c-b526-61e05a15a9c4-istio-podinfo\") pod \"872504eb-20ef-411c-b526-61e05a15a9c4\" (UID: \"872504eb-20ef-411c-b526-61e05a15a9c4\") " Oct 30 20:18:01.990676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.990336 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-credential-socket" pod "872504eb-20ef-411c-b526-61e05a15a9c4" (UID: "872504eb-20ef-411c-b526-61e05a15a9c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:01.990676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.990485 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:01.990676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.990503 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:01.991021 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.990977 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-istio-data" pod "872504eb-20ef-411c-b526-61e05a15a9c4" (UID: "872504eb-20ef-411c-b526-61e05a15a9c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:01.991447 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.991414 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/872504eb-20ef-411c-b526-61e05a15a9c4-istiod-ca-cert" pod "872504eb-20ef-411c-b526-61e05a15a9c4" (UID: "872504eb-20ef-411c-b526-61e05a15a9c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:01.992071 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.992030 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-workload-socket" pod "872504eb-20ef-411c-b526-61e05a15a9c4" (UID: "872504eb-20ef-411c-b526-61e05a15a9c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:01.992446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.992421 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872504eb-20ef-411c-b526-61e05a15a9c4-istio-token" pod "872504eb-20ef-411c-b526-61e05a15a9c4" (UID: "872504eb-20ef-411c-b526-61e05a15a9c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:01.992638 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.992604 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/872504eb-20ef-411c-b526-61e05a15a9c4-kube-api-access-s4645" pod "872504eb-20ef-411c-b526-61e05a15a9c4" (UID: "872504eb-20ef-411c-b526-61e05a15a9c4"). InnerVolumeSpecName "kube-api-access-s4645". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:01.992780 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.992762 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/872504eb-20ef-411c-b526-61e05a15a9c4-istio-podinfo" pod "872504eb-20ef-411c-b526-61e05a15a9c4" (UID: "872504eb-20ef-411c-b526-61e05a15a9c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:18:01.994192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:01.994161 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-istio-envoy" pod "872504eb-20ef-411c-b526-61e05a15a9c4" (UID: "872504eb-20ef-411c-b526-61e05a15a9c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:02.025842 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.025806 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2"] Oct 30 20:18:02.042973 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.042930 5091 generic.go:356] "Generic (PLEG): container finished" podID="872504eb-20ef-411c-b526-61e05a15a9c4" containerID="b80337cf3ed50a192c528978b244cd2870ec86fa9a3c2958258414a5ad2cc0ef" exitCode=0 Oct 30 20:18:02.043188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.042980 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" Oct 30 20:18:02.043188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.043065 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" event={"ID":"872504eb-20ef-411c-b526-61e05a15a9c4","Type":"ContainerDied","Data":"b80337cf3ed50a192c528978b244cd2870ec86fa9a3c2958258414a5ad2cc0ef"} Oct 30 20:18:02.043345 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.043135 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq" event={"ID":"872504eb-20ef-411c-b526-61e05a15a9c4","Type":"ContainerDied","Data":"40655185fad6456e015f451368ec7b4be22516ddd9520a7189dd6c702a308de0"} Oct 30 20:18:02.043345 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.043170 5091 scope.go:122] "RemoveContainer" containerID="b80337cf3ed50a192c528978b244cd2870ec86fa9a3c2958258414a5ad2cc0ef" Oct 30 20:18:02.053653 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.052057 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" Oct 30 20:18:02.053653 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.052272 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59" event={"ID":"11dc06d8-aaf3-49bb-a6e9-3120b363a462","Type":"ContainerDied","Data":"85adf20ce5790c68e8a4a2f7bf3f8b37a78f0fa48ec2fb223c5decfc108c1f23"} Oct 30 20:18:02.053653 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.052318 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85adf20ce5790c68e8a4a2f7bf3f8b37a78f0fa48ec2fb223c5decfc108c1f23" Oct 30 20:18:02.068656 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.068638 5091 scope.go:122] "RemoveContainer" containerID="8fdff38d73f5853ed799dedad4be18e5e92ca0a473371521325500a93cc767a7" Oct 30 20:18:02.080788 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.080766 5091 scope.go:122] "RemoveContainer" containerID="b80337cf3ed50a192c528978b244cd2870ec86fa9a3c2958258414a5ad2cc0ef" Oct 30 20:18:02.081105 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:18:02.081081 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80337cf3ed50a192c528978b244cd2870ec86fa9a3c2958258414a5ad2cc0ef\": container with ID starting with b80337cf3ed50a192c528978b244cd2870ec86fa9a3c2958258414a5ad2cc0ef not found: ID does not exist" containerID="b80337cf3ed50a192c528978b244cd2870ec86fa9a3c2958258414a5ad2cc0ef" Oct 30 20:18:02.081228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.081115 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80337cf3ed50a192c528978b244cd2870ec86fa9a3c2958258414a5ad2cc0ef"} err="failed to get container status \"b80337cf3ed50a192c528978b244cd2870ec86fa9a3c2958258414a5ad2cc0ef\": rpc error: code = NotFound desc = could not find container \"b80337cf3ed50a192c528978b244cd2870ec86fa9a3c2958258414a5ad2cc0ef\": container with ID starting with b80337cf3ed50a192c528978b244cd2870ec86fa9a3c2958258414a5ad2cc0ef not found: ID does not exist" Oct 30 20:18:02.081228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.081140 5091 scope.go:122] "RemoveContainer" containerID="8fdff38d73f5853ed799dedad4be18e5e92ca0a473371521325500a93cc767a7" Oct 30 20:18:02.081481 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:18:02.081461 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fdff38d73f5853ed799dedad4be18e5e92ca0a473371521325500a93cc767a7\": container with ID starting with 8fdff38d73f5853ed799dedad4be18e5e92ca0a473371521325500a93cc767a7 not found: ID does not exist" containerID="8fdff38d73f5853ed799dedad4be18e5e92ca0a473371521325500a93cc767a7" Oct 30 20:18:02.081544 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.081484 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fdff38d73f5853ed799dedad4be18e5e92ca0a473371521325500a93cc767a7"} err="failed to get container status \"8fdff38d73f5853ed799dedad4be18e5e92ca0a473371521325500a93cc767a7\": rpc error: code = NotFound desc = could not find container \"8fdff38d73f5853ed799dedad4be18e5e92ca0a473371521325500a93cc767a7\": container with ID starting with 8fdff38d73f5853ed799dedad4be18e5e92ca0a473371521325500a93cc767a7 not found: ID does not exist" Oct 30 20:18:02.092688 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.092664 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:02.092792 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.092690 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:02.092792 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.092700 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/872504eb-20ef-411c-b526-61e05a15a9c4-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:02.092792 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.092709 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/872504eb-20ef-411c-b526-61e05a15a9c4-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:02.092792 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.092718 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4645\" (UniqueName: \"kubernetes.io/projected/872504eb-20ef-411c-b526-61e05a15a9c4-kube-api-access-s4645\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:02.092792 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.092727 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/872504eb-20ef-411c-b526-61e05a15a9c4-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:02.092792 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.092736 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/872504eb-20ef-411c-b526-61e05a15a9c4-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:02.101921 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.101898 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq"] Oct 30 20:18:02.106678 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.106648 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-779fa3-openshift-default-b556b9b45-mnmwq"] Oct 30 20:18:02.107087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.107052 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:02.280711 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.280672 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:02.443714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:02.443678 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="872504eb-20ef-411c-b526-61e05a15a9c4" path="/var/lib/kubelet/pods/872504eb-20ef-411c-b526-61e05a15a9c4/volumes" Oct 30 20:18:03.109177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:03.100133 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:04.065105 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.065057 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:04.493982 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.493904 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs"] Oct 30 20:18:04.600447 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.600404 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:04.640928 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.640890 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" containerID="cri-o://e9e294f0b6c8cece4596bd4ba4e0be3f916c0bdd065cab3999f38d68eba0013a" gracePeriod=45 Oct 30 20:18:04.696449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.696419 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp"] Oct 30 20:18:04.697171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697126 5091 cpu_manager.go:414] "containerMap: removing container" podUID="872504eb-20ef-411c-b526-61e05a15a9c4" containerName="istio-proxy" Oct 30 20:18:04.697171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697159 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872504eb-20ef-411c-b526-61e05a15a9c4" containerName="istio-proxy" Oct 30 20:18:04.697283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697197 5091 cpu_manager.go:414] "containerMap: removing container" podUID="09d4e116-fd32-41e2-a463-b1ff1a023828" containerName="revoke-keys" Oct 30 20:18:04.697283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697202 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d4e116-fd32-41e2-a463-b1ff1a023828" containerName="revoke-keys" Oct 30 20:18:04.697283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697212 5091 cpu_manager.go:414] "containerMap: removing container" podUID="11dc06d8-aaf3-49bb-a6e9-3120b363a462" containerName="revoke-keys" Oct 30 20:18:04.697283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697217 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11dc06d8-aaf3-49bb-a6e9-3120b363a462" containerName="revoke-keys" Oct 30 20:18:04.697283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697227 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerName="istio-proxy" Oct 30 20:18:04.697283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697232 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerName="istio-proxy" Oct 30 20:18:04.697283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697243 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerName="istio-proxy" Oct 30 20:18:04.697283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697248 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerName="istio-proxy" Oct 30 20:18:04.697283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697255 5091 cpu_manager.go:414] "containerMap: removing container" podUID="872504eb-20ef-411c-b526-61e05a15a9c4" containerName="istio-proxy" Oct 30 20:18:04.697283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697259 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="872504eb-20ef-411c-b526-61e05a15a9c4" containerName="istio-proxy" Oct 30 20:18:04.697591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697365 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerName="istio-proxy" Oct 30 20:18:04.697591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697375 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="872504eb-20ef-411c-b526-61e05a15a9c4" containerName="istio-proxy" Oct 30 20:18:04.697591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697389 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d4e116-fd32-41e2-a463-b1ff1a023828" containerName="revoke-keys" Oct 30 20:18:04.697591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.697394 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="11dc06d8-aaf3-49bb-a6e9-3120b363a462" containerName="revoke-keys" Oct 30 20:18:04.701067 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.701051 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" Oct 30 20:18:04.710999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.710971 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv"] Oct 30 20:18:04.711746 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.711723 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c222e9d2-821b-4c1c-92d2-88297be8f52d" containerName="istio-proxy" Oct 30 20:18:04.711851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.711752 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="872504eb-20ef-411c-b526-61e05a15a9c4" containerName="istio-proxy" Oct 30 20:18:04.713914 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.713889 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:18:04.714040 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.713919 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:18:04.715963 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.715942 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp"] Oct 30 20:18:04.716050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.716024 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" Oct 30 20:18:04.732321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.732291 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv"] Oct 30 20:18:04.856924 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.856883 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:04.857914 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.857878 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7mrw\" (UniqueName: \"kubernetes.io/projected/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-kube-api-access-n7mrw\") pod \"maas-api-revoke-sub-4d11adfc-lc7bp\" (UID: \"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" Oct 30 20:18:04.858023 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.857920 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps46h\" (UniqueName: \"kubernetes.io/projected/c4d0a6d4-3f36-4614-be89-64a2585183ca-kube-api-access-ps46h\") pod \"maas-api-revoke-sub-64d5e97e-zszcv\" (UID: \"c4d0a6d4-3f36-4614-be89-64a2585183ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" Oct 30 20:18:04.858023 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.858006 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d11adfc-lc7bp\" (UID: \"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" Oct 30 20:18:04.858180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.858077 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d0a6d4-3f36-4614-be89-64a2585183ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64d5e97e-zszcv\" (UID: \"c4d0a6d4-3f36-4614-be89-64a2585183ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" Oct 30 20:18:04.858180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.858160 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d0a6d4-3f36-4614-be89-64a2585183ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64d5e97e-zszcv\" (UID: \"c4d0a6d4-3f36-4614-be89-64a2585183ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" Oct 30 20:18:04.858463 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.858432 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d11adfc-lc7bp\" (UID: \"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" Oct 30 20:18:04.964264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.959776 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d0a6d4-3f36-4614-be89-64a2585183ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64d5e97e-zszcv\" (UID: \"c4d0a6d4-3f36-4614-be89-64a2585183ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" Oct 30 20:18:04.964264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.959917 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d11adfc-lc7bp\" (UID: \"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" Oct 30 20:18:04.964264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.959985 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7mrw\" (UniqueName: \"kubernetes.io/projected/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-kube-api-access-n7mrw\") pod \"maas-api-revoke-sub-4d11adfc-lc7bp\" (UID: \"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" Oct 30 20:18:04.964264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.960002 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps46h\" (UniqueName: \"kubernetes.io/projected/c4d0a6d4-3f36-4614-be89-64a2585183ca-kube-api-access-ps46h\") pod \"maas-api-revoke-sub-64d5e97e-zszcv\" (UID: \"c4d0a6d4-3f36-4614-be89-64a2585183ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" Oct 30 20:18:04.964264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.960063 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d11adfc-lc7bp\" (UID: \"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" Oct 30 20:18:04.964264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.960296 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d0a6d4-3f36-4614-be89-64a2585183ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64d5e97e-zszcv\" (UID: \"c4d0a6d4-3f36-4614-be89-64a2585183ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" Oct 30 20:18:04.964264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.962315 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d0a6d4-3f36-4614-be89-64a2585183ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-64d5e97e-zszcv\" (UID: \"c4d0a6d4-3f36-4614-be89-64a2585183ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" Oct 30 20:18:04.964264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.963198 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d11adfc-lc7bp\" (UID: \"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" Oct 30 20:18:04.969175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.965808 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d0a6d4-3f36-4614-be89-64a2585183ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-64d5e97e-zszcv\" (UID: \"c4d0a6d4-3f36-4614-be89-64a2585183ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" Oct 30 20:18:04.978170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.972714 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d11adfc-lc7bp\" (UID: \"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" Oct 30 20:18:04.979197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.978616 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7mrw\" (UniqueName: \"kubernetes.io/projected/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-kube-api-access-n7mrw\") pod \"maas-api-revoke-sub-4d11adfc-lc7bp\" (UID: \"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" Oct 30 20:18:04.983868 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:04.983822 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps46h\" (UniqueName: \"kubernetes.io/projected/c4d0a6d4-3f36-4614-be89-64a2585183ca-kube-api-access-ps46h\") pod \"maas-api-revoke-sub-64d5e97e-zszcv\" (UID: \"c4d0a6d4-3f36-4614-be89-64a2585183ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" Oct 30 20:18:05.097179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.090614 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" Oct 30 20:18:05.113031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.107632 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" Oct 30 20:18:05.118815 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.118444 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:05.164567 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.164461 5091 generic.go:356] "Generic (PLEG): container finished" podID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerID="e9e294f0b6c8cece4596bd4ba4e0be3f916c0bdd065cab3999f38d68eba0013a" exitCode=0 Oct 30 20:18:05.164654 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.164564 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" event={"ID":"6425ff67-d04c-4a3f-a5df-fda1bf36872f","Type":"ContainerDied","Data":"e9e294f0b6c8cece4596bd4ba4e0be3f916c0bdd065cab3999f38d68eba0013a"} Oct 30 20:18:05.197104 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.197076 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 20:18:05.277324 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.277247 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6425ff67-d04c-4a3f-a5df-fda1bf36872f-tls-certs\" (UniqueName: \"kubernetes.io/secret/6425ff67-d04c-4a3f-a5df-fda1bf36872f-tls-certs\") pod \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " Oct 30 20:18:05.277761 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.277734 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6425ff67-d04c-4a3f-a5df-fda1bf36872f-kube-api-access-xvvwx\" (UniqueName: \"kubernetes.io/projected/6425ff67-d04c-4a3f-a5df-fda1bf36872f-kube-api-access-xvvwx\") pod \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " Oct 30 20:18:05.278127 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.278104 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-tmp-dir\") pod \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " Oct 30 20:18:05.278494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.278468 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-tmp-dir" pod "6425ff67-d04c-4a3f-a5df-fda1bf36872f" (UID: "6425ff67-d04c-4a3f-a5df-fda1bf36872f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:05.278634 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.278617 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-dshm\") pod \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " Oct 30 20:18:05.279008 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.278988 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-home\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-home\") pod \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " Oct 30 20:18:05.279284 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.279261 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-home" pod "6425ff67-d04c-4a3f-a5df-fda1bf36872f" (UID: "6425ff67-d04c-4a3f-a5df-fda1bf36872f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:05.279571 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.279552 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-model-cache\") pod \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\" (UID: \"6425ff67-d04c-4a3f-a5df-fda1bf36872f\") " Oct 30 20:18:05.279798 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.279776 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-model-cache" pod "6425ff67-d04c-4a3f-a5df-fda1bf36872f" (UID: "6425ff67-d04c-4a3f-a5df-fda1bf36872f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:05.280397 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.280374 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-model-cache\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:05.280485 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.280434 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-tmp-dir\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:05.280485 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.280449 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-home\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:05.286538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.286508 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6425ff67-d04c-4a3f-a5df-fda1bf36872f-tls-certs" pod "6425ff67-d04c-4a3f-a5df-fda1bf36872f" (UID: "6425ff67-d04c-4a3f-a5df-fda1bf36872f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:05.287729 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.287673 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6425ff67-d04c-4a3f-a5df-fda1bf36872f-kube-api-access-xvvwx" pod "6425ff67-d04c-4a3f-a5df-fda1bf36872f" (UID: "6425ff67-d04c-4a3f-a5df-fda1bf36872f"). InnerVolumeSpecName "kube-api-access-xvvwx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:05.287975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.287941 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-dshm" pod "6425ff67-d04c-4a3f-a5df-fda1bf36872f" (UID: "6425ff67-d04c-4a3f-a5df-fda1bf36872f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:05.381934 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.381890 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6425ff67-d04c-4a3f-a5df-fda1bf36872f-tls-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:05.381934 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.381933 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvvwx\" (UniqueName: \"kubernetes.io/projected/6425ff67-d04c-4a3f-a5df-fda1bf36872f-kube-api-access-xvvwx\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:05.382216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.381950 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6425ff67-d04c-4a3f-a5df-fda1bf36872f-dshm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:05.494621 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:18:05.494590 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2b74c7_1f59_4e6f_ba87_eb7f9ea6b829.slice/crio-177e2e4d111db17802ef0fc92b2db6159b4b764555b669c888147adf8c84cec8 WatchSource:0}: Error finding container 177e2e4d111db17802ef0fc92b2db6159b4b764555b669c888147adf8c84cec8: Status 404 returned error can't find the container with id 177e2e4d111db17802ef0fc92b2db6159b4b764555b669c888147adf8c84cec8 Oct 30 20:18:05.495989 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:18:05.495833 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d0a6d4_3f36_4614_be89_64a2585183ca.slice/crio-3fc5aee42af7e0a7ab79bb11a097754249ea9023476bcf5bbe7ba0ccff1706c0 WatchSource:0}: Error finding container 3fc5aee42af7e0a7ab79bb11a097754249ea9023476bcf5bbe7ba0ccff1706c0: Status 404 returned error can't find the container with id 3fc5aee42af7e0a7ab79bb11a097754249ea9023476bcf5bbe7ba0ccff1706c0 Oct 30 20:18:05.496594 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.496567 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp"] Oct 30 20:18:05.501658 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.501635 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv"] Oct 30 20:18:05.961728 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:05.958776 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:06.176258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:06.176172 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" event={"ID":"c4d0a6d4-3f36-4614-be89-64a2585183ca","Type":"ContainerStarted","Data":"7e64a415ab07cd8b9b9ee59426d369d8ddc318594220d0cfeddd2083a35299c5"} Oct 30 20:18:06.176258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:06.176231 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" event={"ID":"c4d0a6d4-3f36-4614-be89-64a2585183ca","Type":"ContainerStarted","Data":"3fc5aee42af7e0a7ab79bb11a097754249ea9023476bcf5bbe7ba0ccff1706c0"} Oct 30 20:18:06.178050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:06.178018 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" Oct 30 20:18:06.178188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:06.178050 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld" event={"ID":"6425ff67-d04c-4a3f-a5df-fda1bf36872f","Type":"ContainerDied","Data":"d020aae7fa2db36a11f1fba8051dddc1ea9c0e9e0864126c4cdbd56c9939f65a"} Oct 30 20:18:06.178188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:06.178093 5091 scope.go:122] "RemoveContainer" containerID="e9e294f0b6c8cece4596bd4ba4e0be3f916c0bdd065cab3999f38d68eba0013a" Oct 30 20:18:06.180329 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:06.180267 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" event={"ID":"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829","Type":"ContainerStarted","Data":"6c8c09df7713428c2e52a25c7d2d396b66b141c585a1754c412bafb54cd22128"} Oct 30 20:18:06.180479 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:06.180354 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" event={"ID":"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829","Type":"ContainerStarted","Data":"177e2e4d111db17802ef0fc92b2db6159b4b764555b669c888147adf8c84cec8"} Oct 30 20:18:06.200099 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:06.200056 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" podStartSLOduration=2.200044287 podStartE2EDuration="2.200044287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:06.198032794 +0000 UTC m=+2805.893240754" watchObservedRunningTime="2026-10-30 20:18:06.200044287 +0000 UTC m=+2805.895252286" Oct 30 20:18:06.222380 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:06.222339 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" podStartSLOduration=2.22232919 podStartE2EDuration="2.22232919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:06.219663111 +0000 UTC m=+2805.914871068" watchObservedRunningTime="2026-10-30 20:18:06.22232919 +0000 UTC m=+2805.917537147" Oct 30 20:18:06.240677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:06.240651 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld"] Oct 30 20:18:06.247352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:06.247323 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-2a3e1b/simulator-w4-2a3e1b-kserve-545994d9d5-zxcld"] Oct 30 20:18:06.442924 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:06.442845 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" path="/var/lib/kubelet/pods/6425ff67-d04c-4a3f-a5df-fda1bf36872f/volumes" Oct 30 20:18:06.783721 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:06.783682 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:07.191831 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:07.191736 5091 generic.go:356] "Generic (PLEG): container finished" podID="c4d0a6d4-3f36-4614-be89-64a2585183ca" containerID="7e64a415ab07cd8b9b9ee59426d369d8ddc318594220d0cfeddd2083a35299c5" exitCode=0 Oct 30 20:18:07.191985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:07.191825 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" event={"ID":"c4d0a6d4-3f36-4614-be89-64a2585183ca","Type":"ContainerDied","Data":"7e64a415ab07cd8b9b9ee59426d369d8ddc318594220d0cfeddd2083a35299c5"} Oct 30 20:18:07.195049 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:07.195021 5091 generic.go:356] "Generic (PLEG): container finished" podID="bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829" containerID="6c8c09df7713428c2e52a25c7d2d396b66b141c585a1754c412bafb54cd22128" exitCode=0 Oct 30 20:18:07.195181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:07.195085 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" event={"ID":"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829","Type":"ContainerDied","Data":"6c8c09df7713428c2e52a25c7d2d396b66b141c585a1754c412bafb54cd22128"} Oct 30 20:18:07.496120 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:07.496009 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:07.667409 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:07.667352 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:07.842167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:07.840823 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:08.391762 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.391729 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" Oct 30 20:18:08.429527 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.429500 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" Oct 30 20:18:08.431961 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.431939 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d0a6d4-3f36-4614-be89-64a2585183ca-kube-api-access-ps46h\" (UniqueName: \"kubernetes.io/projected/c4d0a6d4-3f36-4614-be89-64a2585183ca-kube-api-access-ps46h\") pod \"c4d0a6d4-3f36-4614-be89-64a2585183ca\" (UID: \"c4d0a6d4-3f36-4614-be89-64a2585183ca\") " Oct 30 20:18:08.432111 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.432097 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c4d0a6d4-3f36-4614-be89-64a2585183ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d0a6d4-3f36-4614-be89-64a2585183ca-maas-api-xks-ca\") pod \"c4d0a6d4-3f36-4614-be89-64a2585183ca\" (UID: \"c4d0a6d4-3f36-4614-be89-64a2585183ca\") " Oct 30 20:18:08.432209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.432196 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d0a6d4-3f36-4614-be89-64a2585183ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d0a6d4-3f36-4614-be89-64a2585183ca-maas-api-openshift-ca\") pod \"c4d0a6d4-3f36-4614-be89-64a2585183ca\" (UID: \"c4d0a6d4-3f36-4614-be89-64a2585183ca\") " Oct 30 20:18:08.432611 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.432579 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d0a6d4-3f36-4614-be89-64a2585183ca-maas-api-openshift-ca" pod "c4d0a6d4-3f36-4614-be89-64a2585183ca" (UID: "c4d0a6d4-3f36-4614-be89-64a2585183ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:08.432838 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.432821 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c4d0a6d4-3f36-4614-be89-64a2585183ca-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:08.434218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.434192 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d0a6d4-3f36-4614-be89-64a2585183ca-kube-api-access-ps46h" pod "c4d0a6d4-3f36-4614-be89-64a2585183ca" (UID: "c4d0a6d4-3f36-4614-be89-64a2585183ca"). InnerVolumeSpecName "kube-api-access-ps46h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:08.434301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.434283 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c4d0a6d4-3f36-4614-be89-64a2585183ca-maas-api-xks-ca" pod "c4d0a6d4-3f36-4614-be89-64a2585183ca" (UID: "c4d0a6d4-3f36-4614-be89-64a2585183ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:08.534264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.534232 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-kube-api-access-n7mrw\" (UniqueName: \"kubernetes.io/projected/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-kube-api-access-n7mrw\") pod \"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829\" (UID: \"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829\") " Oct 30 20:18:08.534456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.534278 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-maas-api-openshift-ca\") pod \"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829\" (UID: \"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829\") " Oct 30 20:18:08.534456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.534333 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-maas-api-xks-ca\") pod \"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829\" (UID: \"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829\") " Oct 30 20:18:08.534720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.534693 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-maas-api-openshift-ca" pod "bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829" (UID: "bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:08.535012 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.534989 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c4d0a6d4-3f36-4614-be89-64a2585183ca-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:08.535161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.535015 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps46h\" (UniqueName: \"kubernetes.io/projected/c4d0a6d4-3f36-4614-be89-64a2585183ca-kube-api-access-ps46h\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:08.535161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.535031 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:08.536665 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.536641 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-maas-api-xks-ca" pod "bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829" (UID: "bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:08.536779 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.536658 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-kube-api-access-n7mrw" pod "bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829" (UID: "bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829"). InnerVolumeSpecName "kube-api-access-n7mrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:08.638698 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.638655 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7mrw\" (UniqueName: \"kubernetes.io/projected/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-kube-api-access-n7mrw\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:08.638698 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.638698 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:08.752903 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:08.752865 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:09.213100 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:09.213015 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" Oct 30 20:18:09.213100 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:09.213057 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp" event={"ID":"bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829","Type":"ContainerDied","Data":"177e2e4d111db17802ef0fc92b2db6159b4b764555b669c888147adf8c84cec8"} Oct 30 20:18:09.213596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:09.213116 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="177e2e4d111db17802ef0fc92b2db6159b4b764555b669c888147adf8c84cec8" Oct 30 20:18:09.214713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:09.214688 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" Oct 30 20:18:09.214856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:09.214722 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv" event={"ID":"c4d0a6d4-3f36-4614-be89-64a2585183ca","Type":"ContainerDied","Data":"3fc5aee42af7e0a7ab79bb11a097754249ea9023476bcf5bbe7ba0ccff1706c0"} Oct 30 20:18:09.214856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:09.214764 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fc5aee42af7e0a7ab79bb11a097754249ea9023476bcf5bbe7ba0ccff1706c0" Oct 30 20:18:09.494320 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:09.494205 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" podUID="77c5214d-bdc6-4294-b702-704c79eaa0aa" containerName="payload-pre-processing" containerID="cri-o://fe31f14a2f911716cde58898ba1e7e75f331e4b36ad089cf3915394547996c9c" gracePeriod=25 Oct 30 20:18:09.622889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:09.621814 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:09.813649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:09.813611 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" podUID="77c5214d-bdc6-4294-b702-704c79eaa0aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:18:10.203855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.203779 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk"] Oct 30 20:18:10.204527 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.204510 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" Oct 30 20:18:10.204527 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.204527 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" Oct 30 20:18:10.204661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.204539 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c4d0a6d4-3f36-4614-be89-64a2585183ca" containerName="revoke-keys" Oct 30 20:18:10.204661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.204545 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d0a6d4-3f36-4614-be89-64a2585183ca" containerName="revoke-keys" Oct 30 20:18:10.204661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.204555 5091 cpu_manager.go:414] "containerMap: removing container" podUID="bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829" containerName="revoke-keys" Oct 30 20:18:10.204661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.204560 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829" containerName="revoke-keys" Oct 30 20:18:10.204799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.204664 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6425ff67-d04c-4a3f-a5df-fda1bf36872f" containerName="main" Oct 30 20:18:10.204799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.204675 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d0a6d4-3f36-4614-be89-64a2585183ca" containerName="revoke-keys" Oct 30 20:18:10.204799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.204685 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829" containerName="revoke-keys" Oct 30 20:18:10.208215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.208199 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" Oct 30 20:18:10.214673 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.214634 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc"] Oct 30 20:18:10.219994 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.219969 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" Oct 30 20:18:10.220646 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.220619 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk"] Oct 30 20:18:10.227228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.227184 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:18:10.227604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.227582 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:18:10.242371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.242340 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc"] Oct 30 20:18:10.262797 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.262767 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bsr5\" (UniqueName: \"kubernetes.io/projected/c187d6c7-f5d8-41ee-a906-11e33e892cd5-kube-api-access-9bsr5\") pod \"maas-api-revoke-sub-dc99947e-fsdlc\" (UID: \"c187d6c7-f5d8-41ee-a906-11e33e892cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" Oct 30 20:18:10.262977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.262809 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c187d6c7-f5d8-41ee-a906-11e33e892cd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc99947e-fsdlc\" (UID: \"c187d6c7-f5d8-41ee-a906-11e33e892cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" Oct 30 20:18:10.262977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.262838 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c187d6c7-f5d8-41ee-a906-11e33e892cd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc99947e-fsdlc\" (UID: \"c187d6c7-f5d8-41ee-a906-11e33e892cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" Oct 30 20:18:10.262977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.262868 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47417445-63ec-4320-8296-59c85ec05fa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5e44932-59sgk\" (UID: \"47417445-63ec-4320-8296-59c85ec05fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" Oct 30 20:18:10.262977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.262955 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47417445-63ec-4320-8296-59c85ec05fa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5e44932-59sgk\" (UID: \"47417445-63ec-4320-8296-59c85ec05fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" Oct 30 20:18:10.263236 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.263106 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xk2h\" (UniqueName: \"kubernetes.io/projected/47417445-63ec-4320-8296-59c85ec05fa3-kube-api-access-4xk2h\") pod \"maas-api-revoke-sub-a5e44932-59sgk\" (UID: \"47417445-63ec-4320-8296-59c85ec05fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" Oct 30 20:18:10.364658 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.364618 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xk2h\" (UniqueName: \"kubernetes.io/projected/47417445-63ec-4320-8296-59c85ec05fa3-kube-api-access-4xk2h\") pod \"maas-api-revoke-sub-a5e44932-59sgk\" (UID: \"47417445-63ec-4320-8296-59c85ec05fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" Oct 30 20:18:10.364858 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.364818 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bsr5\" (UniqueName: \"kubernetes.io/projected/c187d6c7-f5d8-41ee-a906-11e33e892cd5-kube-api-access-9bsr5\") pod \"maas-api-revoke-sub-dc99947e-fsdlc\" (UID: \"c187d6c7-f5d8-41ee-a906-11e33e892cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" Oct 30 20:18:10.364858 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.364853 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c187d6c7-f5d8-41ee-a906-11e33e892cd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc99947e-fsdlc\" (UID: \"c187d6c7-f5d8-41ee-a906-11e33e892cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" Oct 30 20:18:10.364993 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.364878 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c187d6c7-f5d8-41ee-a906-11e33e892cd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc99947e-fsdlc\" (UID: \"c187d6c7-f5d8-41ee-a906-11e33e892cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" Oct 30 20:18:10.364993 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.364908 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47417445-63ec-4320-8296-59c85ec05fa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5e44932-59sgk\" (UID: \"47417445-63ec-4320-8296-59c85ec05fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" Oct 30 20:18:10.365110 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.365004 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47417445-63ec-4320-8296-59c85ec05fa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5e44932-59sgk\" (UID: \"47417445-63ec-4320-8296-59c85ec05fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" Oct 30 20:18:10.365755 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.365724 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c187d6c7-f5d8-41ee-a906-11e33e892cd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc99947e-fsdlc\" (UID: \"c187d6c7-f5d8-41ee-a906-11e33e892cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" Oct 30 20:18:10.366114 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.366086 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47417445-63ec-4320-8296-59c85ec05fa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5e44932-59sgk\" (UID: \"47417445-63ec-4320-8296-59c85ec05fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" Oct 30 20:18:10.367667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.367638 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c187d6c7-f5d8-41ee-a906-11e33e892cd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc99947e-fsdlc\" (UID: \"c187d6c7-f5d8-41ee-a906-11e33e892cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" Oct 30 20:18:10.367852 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.367832 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47417445-63ec-4320-8296-59c85ec05fa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5e44932-59sgk\" (UID: \"47417445-63ec-4320-8296-59c85ec05fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" Oct 30 20:18:10.379422 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.379400 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xk2h\" (UniqueName: \"kubernetes.io/projected/47417445-63ec-4320-8296-59c85ec05fa3-kube-api-access-4xk2h\") pod \"maas-api-revoke-sub-a5e44932-59sgk\" (UID: \"47417445-63ec-4320-8296-59c85ec05fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" Oct 30 20:18:10.381013 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.380985 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bsr5\" (UniqueName: \"kubernetes.io/projected/c187d6c7-f5d8-41ee-a906-11e33e892cd5-kube-api-access-9bsr5\") pod \"maas-api-revoke-sub-dc99947e-fsdlc\" (UID: \"c187d6c7-f5d8-41ee-a906-11e33e892cd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" Oct 30 20:18:10.526625 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.526585 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" Oct 30 20:18:10.539286 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.539258 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" Oct 30 20:18:10.575172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.573210 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:10.827719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.827643 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" podUID="02aaf432-1c15-4e88-9978-23dc3f9ee27d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:18:10.860350 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.860320 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk"] Oct 30 20:18:10.860880 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:18:10.860853 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47417445_63ec_4320_8296_59c85ec05fa3.slice/crio-ef2c9237f031e8705820140896958cf85253e051775949447bb46037e43165c0 WatchSource:0}: Error finding container ef2c9237f031e8705820140896958cf85253e051775949447bb46037e43165c0: Status 404 returned error can't find the container with id ef2c9237f031e8705820140896958cf85253e051775949447bb46037e43165c0 Oct 30 20:18:10.893846 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:10.893821 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc"] Oct 30 20:18:10.897643 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:18:10.897619 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc187d6c7_f5d8_41ee_a906_11e33e892cd5.slice/crio-4fde33b970f89806f9742b69c19565b1b1c7a4e44901d2dd08064465f19746a8 WatchSource:0}: Error finding container 4fde33b970f89806f9742b69c19565b1b1c7a4e44901d2dd08064465f19746a8: Status 404 returned error can't find the container with id 4fde33b970f89806f9742b69c19565b1b1c7a4e44901d2dd08064465f19746a8 Oct 30 20:18:11.235042 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:11.234964 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" event={"ID":"47417445-63ec-4320-8296-59c85ec05fa3","Type":"ContainerStarted","Data":"ef2c9237f031e8705820140896958cf85253e051775949447bb46037e43165c0"} Oct 30 20:18:11.236613 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:11.236575 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" event={"ID":"c187d6c7-f5d8-41ee-a906-11e33e892cd5","Type":"ContainerStarted","Data":"4fde33b970f89806f9742b69c19565b1b1c7a4e44901d2dd08064465f19746a8"} Oct 30 20:18:11.485802 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:11.482200 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:11.831770 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:11.831736 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" podUID="f76728a5-e155-435a-a692-cc9afee10063" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:18:12.131073 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.130992 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl"] Oct 30 20:18:12.136415 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.136388 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" Oct 30 20:18:12.142391 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.142354 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2"] Oct 30 20:18:12.147682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.147656 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl"] Oct 30 20:18:12.147848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.147786 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" Oct 30 20:18:12.179116 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.179090 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2"] Oct 30 20:18:12.199277 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.199251 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97462176-f0ce-492c-bcad-1e125c73a4cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a852f6-47svl\" (UID: \"97462176-f0ce-492c-bcad-1e125c73a4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" Oct 30 20:18:12.199417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.199327 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62a6025-b883-4a28-bdea-ab4078ffc1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a61e133f-h95p2\" (UID: \"b62a6025-b883-4a28-bdea-ab4078ffc1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" Oct 30 20:18:12.199470 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.199457 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97462176-f0ce-492c-bcad-1e125c73a4cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a852f6-47svl\" (UID: \"97462176-f0ce-492c-bcad-1e125c73a4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" Oct 30 20:18:12.199513 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.199481 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbn9v\" (UniqueName: \"kubernetes.io/projected/b62a6025-b883-4a28-bdea-ab4078ffc1bf-kube-api-access-tbn9v\") pod \"maas-api-revoke-sub-a61e133f-h95p2\" (UID: \"b62a6025-b883-4a28-bdea-ab4078ffc1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" Oct 30 20:18:12.199513 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.199501 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz98z\" (UniqueName: \"kubernetes.io/projected/97462176-f0ce-492c-bcad-1e125c73a4cd-kube-api-access-fz98z\") pod \"maas-api-revoke-sub-61a852f6-47svl\" (UID: \"97462176-f0ce-492c-bcad-1e125c73a4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" Oct 30 20:18:12.199669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.199646 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62a6025-b883-4a28-bdea-ab4078ffc1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a61e133f-h95p2\" (UID: \"b62a6025-b883-4a28-bdea-ab4078ffc1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" Oct 30 20:18:12.247022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.246988 5091 generic.go:356] "Generic (PLEG): container finished" podID="c187d6c7-f5d8-41ee-a906-11e33e892cd5" containerID="2d439bf3480137cf2e074914cb35272876fc70124dde01a359029617cdd14bfe" exitCode=0 Oct 30 20:18:12.247433 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.247071 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" event={"ID":"c187d6c7-f5d8-41ee-a906-11e33e892cd5","Type":"ContainerDied","Data":"2d439bf3480137cf2e074914cb35272876fc70124dde01a359029617cdd14bfe"} Oct 30 20:18:12.248890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.248866 5091 generic.go:356] "Generic (PLEG): container finished" podID="47417445-63ec-4320-8296-59c85ec05fa3" containerID="ce4be51ca712a33e348cd4945f66e24c83927f389a021e8756f7b22ecdf6e6bd" exitCode=0 Oct 30 20:18:12.248999 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.248941 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" event={"ID":"47417445-63ec-4320-8296-59c85ec05fa3","Type":"ContainerDied","Data":"ce4be51ca712a33e348cd4945f66e24c83927f389a021e8756f7b22ecdf6e6bd"} Oct 30 20:18:12.301301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.301257 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97462176-f0ce-492c-bcad-1e125c73a4cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a852f6-47svl\" (UID: \"97462176-f0ce-492c-bcad-1e125c73a4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" Oct 30 20:18:12.301452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.301352 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62a6025-b883-4a28-bdea-ab4078ffc1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a61e133f-h95p2\" (UID: \"b62a6025-b883-4a28-bdea-ab4078ffc1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" Oct 30 20:18:12.301499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.301459 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97462176-f0ce-492c-bcad-1e125c73a4cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a852f6-47svl\" (UID: \"97462176-f0ce-492c-bcad-1e125c73a4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" Oct 30 20:18:12.301499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.301480 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbn9v\" (UniqueName: \"kubernetes.io/projected/b62a6025-b883-4a28-bdea-ab4078ffc1bf-kube-api-access-tbn9v\") pod \"maas-api-revoke-sub-a61e133f-h95p2\" (UID: \"b62a6025-b883-4a28-bdea-ab4078ffc1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" Oct 30 20:18:12.301585 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.301499 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz98z\" (UniqueName: \"kubernetes.io/projected/97462176-f0ce-492c-bcad-1e125c73a4cd-kube-api-access-fz98z\") pod \"maas-api-revoke-sub-61a852f6-47svl\" (UID: \"97462176-f0ce-492c-bcad-1e125c73a4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" Oct 30 20:18:12.301693 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.301670 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62a6025-b883-4a28-bdea-ab4078ffc1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a61e133f-h95p2\" (UID: \"b62a6025-b883-4a28-bdea-ab4078ffc1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" Oct 30 20:18:12.302057 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.302036 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97462176-f0ce-492c-bcad-1e125c73a4cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61a852f6-47svl\" (UID: \"97462176-f0ce-492c-bcad-1e125c73a4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" Oct 30 20:18:12.302275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.302255 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62a6025-b883-4a28-bdea-ab4078ffc1bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a61e133f-h95p2\" (UID: \"b62a6025-b883-4a28-bdea-ab4078ffc1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" Oct 30 20:18:12.303746 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.303727 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62a6025-b883-4a28-bdea-ab4078ffc1bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a61e133f-h95p2\" (UID: \"b62a6025-b883-4a28-bdea-ab4078ffc1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" Oct 30 20:18:12.303896 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.303874 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97462176-f0ce-492c-bcad-1e125c73a4cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61a852f6-47svl\" (UID: \"97462176-f0ce-492c-bcad-1e125c73a4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" Oct 30 20:18:12.315700 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.315677 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz98z\" (UniqueName: \"kubernetes.io/projected/97462176-f0ce-492c-bcad-1e125c73a4cd-kube-api-access-fz98z\") pod \"maas-api-revoke-sub-61a852f6-47svl\" (UID: \"97462176-f0ce-492c-bcad-1e125c73a4cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" Oct 30 20:18:12.315830 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.315812 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbn9v\" (UniqueName: \"kubernetes.io/projected/b62a6025-b883-4a28-bdea-ab4078ffc1bf-kube-api-access-tbn9v\") pod \"maas-api-revoke-sub-a61e133f-h95p2\" (UID: \"b62a6025-b883-4a28-bdea-ab4078ffc1bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" Oct 30 20:18:12.481201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.477122 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" Oct 30 20:18:12.481201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.478342 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:12.484648 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.484602 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" Oct 30 20:18:12.794718 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:18:12.794683 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb62a6025_b883_4a28_bdea_ab4078ffc1bf.slice/crio-e442d0c0ae5bcac1953b540c9a3b589ca4f1145ea10df18a667c9253abe56a72 WatchSource:0}: Error finding container e442d0c0ae5bcac1953b540c9a3b589ca4f1145ea10df18a667c9253abe56a72: Status 404 returned error can't find the container with id e442d0c0ae5bcac1953b540c9a3b589ca4f1145ea10df18a667c9253abe56a72 Oct 30 20:18:12.795475 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.795446 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2"] Oct 30 20:18:12.802533 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:18:12.802509 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97462176_f0ce_492c_bcad_1e125c73a4cd.slice/crio-35f8a4237371685d3b68cd6a21525cdc9313858d6624b50a459dda52bde62bd8 WatchSource:0}: Error finding container 35f8a4237371685d3b68cd6a21525cdc9313858d6624b50a459dda52bde62bd8: Status 404 returned error can't find the container with id 35f8a4237371685d3b68cd6a21525cdc9313858d6624b50a459dda52bde62bd8 Oct 30 20:18:12.803990 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:12.803966 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl"] Oct 30 20:18:13.264727 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.264684 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" event={"ID":"97462176-f0ce-492c-bcad-1e125c73a4cd","Type":"ContainerStarted","Data":"04f582944081d056e3030050db4075515f3234af2aaf13a9bb11a45ec67c7990"} Oct 30 20:18:13.265199 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.264756 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" event={"ID":"97462176-f0ce-492c-bcad-1e125c73a4cd","Type":"ContainerStarted","Data":"35f8a4237371685d3b68cd6a21525cdc9313858d6624b50a459dda52bde62bd8"} Oct 30 20:18:13.267636 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.267604 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" event={"ID":"b62a6025-b883-4a28-bdea-ab4078ffc1bf","Type":"ContainerStarted","Data":"d1e173bad783867e3714f7f0d26f940e7b9a186246d5c75a47bb641888913445"} Oct 30 20:18:13.267783 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.267657 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" event={"ID":"b62a6025-b883-4a28-bdea-ab4078ffc1bf","Type":"ContainerStarted","Data":"e442d0c0ae5bcac1953b540c9a3b589ca4f1145ea10df18a667c9253abe56a72"} Oct 30 20:18:13.312285 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.312175 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" podStartSLOduration=1.312156659 podStartE2EDuration="1.312156659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:13.287292242 +0000 UTC m=+2812.982500204" watchObservedRunningTime="2026-10-30 20:18:13.312156659 +0000 UTC m=+2813.007364609" Oct 30 20:18:13.319550 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.319475 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" podStartSLOduration=1.31946281 podStartE2EDuration="1.31946281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:13.317188504 +0000 UTC m=+2813.012396463" watchObservedRunningTime="2026-10-30 20:18:13.31946281 +0000 UTC m=+2813.014670807" Oct 30 20:18:13.395263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.384789 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:13.468263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.468228 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" Oct 30 20:18:13.515775 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.515743 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c187d6c7-f5d8-41ee-a906-11e33e892cd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c187d6c7-f5d8-41ee-a906-11e33e892cd5-maas-api-xks-ca\") pod \"c187d6c7-f5d8-41ee-a906-11e33e892cd5\" (UID: \"c187d6c7-f5d8-41ee-a906-11e33e892cd5\") " Oct 30 20:18:13.515775 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.515780 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c187d6c7-f5d8-41ee-a906-11e33e892cd5-kube-api-access-9bsr5\" (UniqueName: \"kubernetes.io/projected/c187d6c7-f5d8-41ee-a906-11e33e892cd5-kube-api-access-9bsr5\") pod \"c187d6c7-f5d8-41ee-a906-11e33e892cd5\" (UID: \"c187d6c7-f5d8-41ee-a906-11e33e892cd5\") " Oct 30 20:18:13.516025 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.515856 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c187d6c7-f5d8-41ee-a906-11e33e892cd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c187d6c7-f5d8-41ee-a906-11e33e892cd5-maas-api-openshift-ca\") pod \"c187d6c7-f5d8-41ee-a906-11e33e892cd5\" (UID: \"c187d6c7-f5d8-41ee-a906-11e33e892cd5\") " Oct 30 20:18:13.517275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.517246 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c187d6c7-f5d8-41ee-a906-11e33e892cd5-maas-api-openshift-ca" pod "c187d6c7-f5d8-41ee-a906-11e33e892cd5" (UID: "c187d6c7-f5d8-41ee-a906-11e33e892cd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:13.520020 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.519995 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c187d6c7-f5d8-41ee-a906-11e33e892cd5-maas-api-xks-ca" pod "c187d6c7-f5d8-41ee-a906-11e33e892cd5" (UID: "c187d6c7-f5d8-41ee-a906-11e33e892cd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:13.520498 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.520463 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c187d6c7-f5d8-41ee-a906-11e33e892cd5-kube-api-access-9bsr5" pod "c187d6c7-f5d8-41ee-a906-11e33e892cd5" (UID: "c187d6c7-f5d8-41ee-a906-11e33e892cd5"). InnerVolumeSpecName "kube-api-access-9bsr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:13.557011 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.556986 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" Oct 30 20:18:13.617466 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.617381 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47417445-63ec-4320-8296-59c85ec05fa3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47417445-63ec-4320-8296-59c85ec05fa3-maas-api-xks-ca\") pod \"47417445-63ec-4320-8296-59c85ec05fa3\" (UID: \"47417445-63ec-4320-8296-59c85ec05fa3\") " Oct 30 20:18:13.617648 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.617631 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47417445-63ec-4320-8296-59c85ec05fa3-kube-api-access-4xk2h\" (UniqueName: \"kubernetes.io/projected/47417445-63ec-4320-8296-59c85ec05fa3-kube-api-access-4xk2h\") pod \"47417445-63ec-4320-8296-59c85ec05fa3\" (UID: \"47417445-63ec-4320-8296-59c85ec05fa3\") " Oct 30 20:18:13.618031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.618004 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47417445-63ec-4320-8296-59c85ec05fa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47417445-63ec-4320-8296-59c85ec05fa3-maas-api-openshift-ca\") pod \"47417445-63ec-4320-8296-59c85ec05fa3\" (UID: \"47417445-63ec-4320-8296-59c85ec05fa3\") " Oct 30 20:18:13.618696 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.618659 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47417445-63ec-4320-8296-59c85ec05fa3-maas-api-openshift-ca" pod "47417445-63ec-4320-8296-59c85ec05fa3" (UID: "47417445-63ec-4320-8296-59c85ec05fa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:13.618822 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.618748 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c187d6c7-f5d8-41ee-a906-11e33e892cd5-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:13.618822 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.618775 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bsr5\" (UniqueName: \"kubernetes.io/projected/c187d6c7-f5d8-41ee-a906-11e33e892cd5-kube-api-access-9bsr5\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:13.618822 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.618792 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c187d6c7-f5d8-41ee-a906-11e33e892cd5-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:13.620762 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.620393 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47417445-63ec-4320-8296-59c85ec05fa3-kube-api-access-4xk2h" pod "47417445-63ec-4320-8296-59c85ec05fa3" (UID: "47417445-63ec-4320-8296-59c85ec05fa3"). InnerVolumeSpecName "kube-api-access-4xk2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:13.620762 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.620640 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47417445-63ec-4320-8296-59c85ec05fa3-maas-api-xks-ca" pod "47417445-63ec-4320-8296-59c85ec05fa3" (UID: "47417445-63ec-4320-8296-59c85ec05fa3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:13.720298 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.720259 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47417445-63ec-4320-8296-59c85ec05fa3-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:13.720298 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.720294 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xk2h\" (UniqueName: \"kubernetes.io/projected/47417445-63ec-4320-8296-59c85ec05fa3-kube-api-access-4xk2h\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:13.720298 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:13.720305 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47417445-63ec-4320-8296-59c85ec05fa3-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:14.189285 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:14.188205 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:14.278826 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:14.278793 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" Oct 30 20:18:14.278826 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:14.278815 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk" event={"ID":"47417445-63ec-4320-8296-59c85ec05fa3","Type":"ContainerDied","Data":"ef2c9237f031e8705820140896958cf85253e051775949447bb46037e43165c0"} Oct 30 20:18:14.279386 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:14.278867 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef2c9237f031e8705820140896958cf85253e051775949447bb46037e43165c0" Oct 30 20:18:14.281111 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:14.281085 5091 generic.go:356] "Generic (PLEG): container finished" podID="97462176-f0ce-492c-bcad-1e125c73a4cd" containerID="04f582944081d056e3030050db4075515f3234af2aaf13a9bb11a45ec67c7990" exitCode=0 Oct 30 20:18:14.281254 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:14.281182 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" event={"ID":"97462176-f0ce-492c-bcad-1e125c73a4cd","Type":"ContainerDied","Data":"04f582944081d056e3030050db4075515f3234af2aaf13a9bb11a45ec67c7990"} Oct 30 20:18:14.283033 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:14.283007 5091 generic.go:356] "Generic (PLEG): container finished" podID="b62a6025-b883-4a28-bdea-ab4078ffc1bf" containerID="d1e173bad783867e3714f7f0d26f940e7b9a186246d5c75a47bb641888913445" exitCode=0 Oct 30 20:18:14.283186 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:14.283096 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" event={"ID":"b62a6025-b883-4a28-bdea-ab4078ffc1bf","Type":"ContainerDied","Data":"d1e173bad783867e3714f7f0d26f940e7b9a186246d5c75a47bb641888913445"} Oct 30 20:18:14.284852 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:14.284831 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" Oct 30 20:18:14.284966 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:14.284854 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc" event={"ID":"c187d6c7-f5d8-41ee-a906-11e33e892cd5","Type":"ContainerDied","Data":"4fde33b970f89806f9742b69c19565b1b1c7a4e44901d2dd08064465f19746a8"} Oct 30 20:18:14.284966 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:14.284895 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fde33b970f89806f9742b69c19565b1b1c7a4e44901d2dd08064465f19746a8" Oct 30 20:18:14.994219 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:14.993849 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:15.220172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.219907 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:15.493227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.493203 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" Oct 30 20:18:15.517290 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.517216 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" Oct 30 20:18:15.551853 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.551775 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97462176-f0ce-492c-bcad-1e125c73a4cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97462176-f0ce-492c-bcad-1e125c73a4cd-maas-api-openshift-ca\") pod \"97462176-f0ce-492c-bcad-1e125c73a4cd\" (UID: \"97462176-f0ce-492c-bcad-1e125c73a4cd\") " Oct 30 20:18:15.551853 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.551840 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97462176-f0ce-492c-bcad-1e125c73a4cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97462176-f0ce-492c-bcad-1e125c73a4cd-maas-api-xks-ca\") pod \"97462176-f0ce-492c-bcad-1e125c73a4cd\" (UID: \"97462176-f0ce-492c-bcad-1e125c73a4cd\") " Oct 30 20:18:15.552078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.551924 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97462176-f0ce-492c-bcad-1e125c73a4cd-kube-api-access-fz98z\" (UniqueName: \"kubernetes.io/projected/97462176-f0ce-492c-bcad-1e125c73a4cd-kube-api-access-fz98z\") pod \"97462176-f0ce-492c-bcad-1e125c73a4cd\" (UID: \"97462176-f0ce-492c-bcad-1e125c73a4cd\") " Oct 30 20:18:15.553367 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.553335 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97462176-f0ce-492c-bcad-1e125c73a4cd-maas-api-openshift-ca" pod "97462176-f0ce-492c-bcad-1e125c73a4cd" (UID: "97462176-f0ce-492c-bcad-1e125c73a4cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:15.559391 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.559360 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97462176-f0ce-492c-bcad-1e125c73a4cd-kube-api-access-fz98z" pod "97462176-f0ce-492c-bcad-1e125c73a4cd" (UID: "97462176-f0ce-492c-bcad-1e125c73a4cd"). InnerVolumeSpecName "kube-api-access-fz98z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:15.559797 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.559774 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97462176-f0ce-492c-bcad-1e125c73a4cd-maas-api-xks-ca" pod "97462176-f0ce-492c-bcad-1e125c73a4cd" (UID: "97462176-f0ce-492c-bcad-1e125c73a4cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:15.653764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.653733 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b62a6025-b883-4a28-bdea-ab4078ffc1bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62a6025-b883-4a28-bdea-ab4078ffc1bf-maas-api-openshift-ca\") pod \"b62a6025-b883-4a28-bdea-ab4078ffc1bf\" (UID: \"b62a6025-b883-4a28-bdea-ab4078ffc1bf\") " Oct 30 20:18:15.653915 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.653882 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b62a6025-b883-4a28-bdea-ab4078ffc1bf-kube-api-access-tbn9v\" (UniqueName: \"kubernetes.io/projected/b62a6025-b883-4a28-bdea-ab4078ffc1bf-kube-api-access-tbn9v\") pod \"b62a6025-b883-4a28-bdea-ab4078ffc1bf\" (UID: \"b62a6025-b883-4a28-bdea-ab4078ffc1bf\") " Oct 30 20:18:15.654016 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.653998 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b62a6025-b883-4a28-bdea-ab4078ffc1bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62a6025-b883-4a28-bdea-ab4078ffc1bf-maas-api-xks-ca\") pod \"b62a6025-b883-4a28-bdea-ab4078ffc1bf\" (UID: \"b62a6025-b883-4a28-bdea-ab4078ffc1bf\") " Oct 30 20:18:15.654190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.654142 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b62a6025-b883-4a28-bdea-ab4078ffc1bf-maas-api-openshift-ca" pod "b62a6025-b883-4a28-bdea-ab4078ffc1bf" (UID: "b62a6025-b883-4a28-bdea-ab4078ffc1bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:15.654908 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.654873 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b62a6025-b883-4a28-bdea-ab4078ffc1bf-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:15.654908 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.654904 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97462176-f0ce-492c-bcad-1e125c73a4cd-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:15.655104 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.654920 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97462176-f0ce-492c-bcad-1e125c73a4cd-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:15.655104 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.654934 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz98z\" (UniqueName: \"kubernetes.io/projected/97462176-f0ce-492c-bcad-1e125c73a4cd-kube-api-access-fz98z\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:15.656006 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.655982 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b62a6025-b883-4a28-bdea-ab4078ffc1bf-kube-api-access-tbn9v" pod "b62a6025-b883-4a28-bdea-ab4078ffc1bf" (UID: "b62a6025-b883-4a28-bdea-ab4078ffc1bf"). InnerVolumeSpecName "kube-api-access-tbn9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:15.656282 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.656261 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b62a6025-b883-4a28-bdea-ab4078ffc1bf-maas-api-xks-ca" pod "b62a6025-b883-4a28-bdea-ab4078ffc1bf" (UID: "b62a6025-b883-4a28-bdea-ab4078ffc1bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:15.756740 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.756710 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbn9v\" (UniqueName: \"kubernetes.io/projected/b62a6025-b883-4a28-bdea-ab4078ffc1bf-kube-api-access-tbn9v\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:15.756879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:15.756737 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b62a6025-b883-4a28-bdea-ab4078ffc1bf-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:15.976910 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:18:15.976827 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:18:16.165036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:16.165000 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:16.329989 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:16.329948 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:16.333059 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:16.333031 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" Oct 30 20:18:16.333059 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:16.333046 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl" event={"ID":"97462176-f0ce-492c-bcad-1e125c73a4cd","Type":"ContainerDied","Data":"35f8a4237371685d3b68cd6a21525cdc9313858d6624b50a459dda52bde62bd8"} Oct 30 20:18:16.333310 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:16.333104 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35f8a4237371685d3b68cd6a21525cdc9313858d6624b50a459dda52bde62bd8" Oct 30 20:18:16.335261 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:16.335237 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" Oct 30 20:18:16.335399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:16.335271 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2" event={"ID":"b62a6025-b883-4a28-bdea-ab4078ffc1bf","Type":"ContainerDied","Data":"e442d0c0ae5bcac1953b540c9a3b589ca4f1145ea10df18a667c9253abe56a72"} Oct 30 20:18:16.335399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:16.335324 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e442d0c0ae5bcac1953b540c9a3b589ca4f1145ea10df18a667c9253abe56a72" Oct 30 20:18:17.059221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.059179 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" containerID="cri-o://00c2909f65532ffbe960d01bbb286fe704307163200c69bdcb2e5eae85b26c7b" gracePeriod=45 Oct 30 20:18:17.227039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.226998 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:17.348871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.348789 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" event={"ID":"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb","Type":"ContainerDied","Data":"00c2909f65532ffbe960d01bbb286fe704307163200c69bdcb2e5eae85b26c7b"} Oct 30 20:18:17.348871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.348814 5091 generic.go:356] "Generic (PLEG): container finished" podID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerID="00c2909f65532ffbe960d01bbb286fe704307163200c69bdcb2e5eae85b26c7b" exitCode=0 Oct 30 20:18:17.393637 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.393601 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 20:18:17.508693 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.508661 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-kube-api-access-fq5xw\" (UniqueName: \"kubernetes.io/projected/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-kube-api-access-fq5xw\") pod \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " Oct 30 20:18:17.508865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.508737 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-home\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-home\") pod \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " Oct 30 20:18:17.508865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.508760 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-tls-certs\" (UniqueName: \"kubernetes.io/secret/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-tls-certs\") pod \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " Oct 30 20:18:17.508865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.508780 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-model-cache\") pod \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " Oct 30 20:18:17.508865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.508816 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-tmp-dir\") pod \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " Oct 30 20:18:17.509098 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.508942 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-dshm\") pod \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\" (UID: \"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb\") " Oct 30 20:18:17.509215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.509087 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-home" pod "4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" (UID: "4c14c99d-b8f0-4b81-ab2c-723529cb9dbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:17.509297 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.509220 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-model-cache" pod "4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" (UID: "4c14c99d-b8f0-4b81-ab2c-723529cb9dbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:17.509676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.509647 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-tmp-dir" pod "4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" (UID: "4c14c99d-b8f0-4b81-ab2c-723529cb9dbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:17.509791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.509706 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-home\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:17.509791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.509733 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-model-cache\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:17.511355 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.511333 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-tls-certs" pod "4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" (UID: "4c14c99d-b8f0-4b81-ab2c-723529cb9dbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:18:17.511425 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.511386 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-kube-api-access-fq5xw" pod "4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" (UID: "4c14c99d-b8f0-4b81-ab2c-723529cb9dbb"). InnerVolumeSpecName "kube-api-access-fq5xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:17.514466 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.514446 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-dshm" pod "4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" (UID: "4c14c99d-b8f0-4b81-ab2c-723529cb9dbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:17.611053 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.610965 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq5xw\" (UniqueName: \"kubernetes.io/projected/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-kube-api-access-fq5xw\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:17.611053 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.610996 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-tls-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:17.611053 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.611011 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-tmp-dir\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:17.611053 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:17.611023 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb-dshm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:18.257298 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:18.257194 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:18.355425 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:18.355372 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" probeResult="failure" output="Get \"https://10.132.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 20:18:18.359389 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:18.359361 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" Oct 30 20:18:18.359555 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:18.359388 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2" event={"ID":"4c14c99d-b8f0-4b81-ab2c-723529cb9dbb","Type":"ContainerDied","Data":"11b4b7c636d4410fbdf8fe486bebf1ccf34d77757cbb91e7b56daadfe7d67226"} Oct 30 20:18:18.359555 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:18.359435 5091 scope.go:122] "RemoveContainer" containerID="00c2909f65532ffbe960d01bbb286fe704307163200c69bdcb2e5eae85b26c7b" Oct 30 20:18:18.390245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:18.390222 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2"] Oct 30 20:18:18.394125 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:18.394102 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-c1fdd6/premium-w3-c1fdd6-kserve-54d49787bc-msgg2"] Oct 30 20:18:18.443183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:18.443137 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" path="/var/lib/kubelet/pods/4c14c99d-b8f0-4b81-ab2c-723529cb9dbb/volumes" Oct 30 20:18:19.142067 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:19.142013 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:19.813933 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:19.813892 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" podUID="77c5214d-bdc6-4294-b702-704c79eaa0aa" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:18:20.018538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.018448 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:20.239807 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.239781 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" Oct 30 20:18:20.250404 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.250357 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76728a5-e155-435a-a692-cc9afee10063-kube-api-access-s2v7v\" (UniqueName: \"kubernetes.io/projected/f76728a5-e155-435a-a692-cc9afee10063-kube-api-access-s2v7v\") pod \"f76728a5-e155-435a-a692-cc9afee10063\" (UID: \"f76728a5-e155-435a-a692-cc9afee10063\") " Oct 30 20:18:20.250605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.250581 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f76728a5-e155-435a-a692-cc9afee10063-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76728a5-e155-435a-a692-cc9afee10063-plugins-config-volume\") pod \"f76728a5-e155-435a-a692-cc9afee10063\" (UID: \"f76728a5-e155-435a-a692-cc9afee10063\") " Oct 30 20:18:20.251119 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.251086 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f76728a5-e155-435a-a692-cc9afee10063-plugins-config-volume" pod "f76728a5-e155-435a-a692-cc9afee10063" (UID: "f76728a5-e155-435a-a692-cc9afee10063"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:20.251287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.251268 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f76728a5-e155-435a-a692-cc9afee10063-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:20.253199 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.253171 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76728a5-e155-435a-a692-cc9afee10063-kube-api-access-s2v7v" pod "f76728a5-e155-435a-a692-cc9afee10063" (UID: "f76728a5-e155-435a-a692-cc9afee10063"). InnerVolumeSpecName "kube-api-access-s2v7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:20.280443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.280406 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" Oct 30 20:18:20.351702 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.351666 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02aaf432-1c15-4e88-9978-23dc3f9ee27d-kube-api-access-pfq6h\" (UniqueName: \"kubernetes.io/projected/02aaf432-1c15-4e88-9978-23dc3f9ee27d-kube-api-access-pfq6h\") pod \"02aaf432-1c15-4e88-9978-23dc3f9ee27d\" (UID: \"02aaf432-1c15-4e88-9978-23dc3f9ee27d\") " Oct 30 20:18:20.351967 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.351946 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02aaf432-1c15-4e88-9978-23dc3f9ee27d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02aaf432-1c15-4e88-9978-23dc3f9ee27d-plugins-config-volume\") pod \"02aaf432-1c15-4e88-9978-23dc3f9ee27d\" (UID: \"02aaf432-1c15-4e88-9978-23dc3f9ee27d\") " Oct 30 20:18:20.352421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.352389 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02aaf432-1c15-4e88-9978-23dc3f9ee27d-plugins-config-volume" pod "02aaf432-1c15-4e88-9978-23dc3f9ee27d" (UID: "02aaf432-1c15-4e88-9978-23dc3f9ee27d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:20.352560 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.352538 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2v7v\" (UniqueName: \"kubernetes.io/projected/f76728a5-e155-435a-a692-cc9afee10063-kube-api-access-s2v7v\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:20.352641 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.352561 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02aaf432-1c15-4e88-9978-23dc3f9ee27d-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:20.353972 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.353951 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02aaf432-1c15-4e88-9978-23dc3f9ee27d-kube-api-access-pfq6h" pod "02aaf432-1c15-4e88-9978-23dc3f9ee27d" (UID: "02aaf432-1c15-4e88-9978-23dc3f9ee27d"). InnerVolumeSpecName "kube-api-access-pfq6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:20.384985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.384899 5091 generic.go:356] "Generic (PLEG): container finished" podID="f76728a5-e155-435a-a692-cc9afee10063" containerID="da0387542447795de10080e05ff16384c4ec925ebb6021183f68263114349a9a" exitCode=0 Oct 30 20:18:20.384985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.384949 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" Oct 30 20:18:20.385217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.384985 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" event={"ID":"f76728a5-e155-435a-a692-cc9afee10063","Type":"ContainerDied","Data":"da0387542447795de10080e05ff16384c4ec925ebb6021183f68263114349a9a"} Oct 30 20:18:20.385217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.385049 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74" event={"ID":"f76728a5-e155-435a-a692-cc9afee10063","Type":"ContainerDied","Data":"54b4a301931fc47759dd68e1ade1cca0761eb75049324aac448c33d335b829c1"} Oct 30 20:18:20.385217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.385097 5091 scope.go:122] "RemoveContainer" containerID="da0387542447795de10080e05ff16384c4ec925ebb6021183f68263114349a9a" Oct 30 20:18:20.387463 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.387437 5091 generic.go:356] "Generic (PLEG): container finished" podID="02aaf432-1c15-4e88-9978-23dc3f9ee27d" containerID="88fde8d5cb82274a117df04703609db069be952cfb2c7a947c3f76c20d9f8f76" exitCode=0 Oct 30 20:18:20.387613 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.387596 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" event={"ID":"02aaf432-1c15-4e88-9978-23dc3f9ee27d","Type":"ContainerDied","Data":"88fde8d5cb82274a117df04703609db069be952cfb2c7a947c3f76c20d9f8f76"} Oct 30 20:18:20.387702 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.387639 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" event={"ID":"02aaf432-1c15-4e88-9978-23dc3f9ee27d","Type":"ContainerDied","Data":"e83b7b522f4c50d320db6a00d61eb11b4d15b350ebfd383d77cfce5eb4690de5"} Oct 30 20:18:20.387766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.387708 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm" Oct 30 20:18:20.398197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.398174 5091 scope.go:122] "RemoveContainer" containerID="da0387542447795de10080e05ff16384c4ec925ebb6021183f68263114349a9a" Oct 30 20:18:20.398512 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:18:20.398484 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da0387542447795de10080e05ff16384c4ec925ebb6021183f68263114349a9a\": container with ID starting with da0387542447795de10080e05ff16384c4ec925ebb6021183f68263114349a9a not found: ID does not exist" containerID="da0387542447795de10080e05ff16384c4ec925ebb6021183f68263114349a9a" Oct 30 20:18:20.398589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.398524 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da0387542447795de10080e05ff16384c4ec925ebb6021183f68263114349a9a"} err="failed to get container status \"da0387542447795de10080e05ff16384c4ec925ebb6021183f68263114349a9a\": rpc error: code = NotFound desc = could not find container \"da0387542447795de10080e05ff16384c4ec925ebb6021183f68263114349a9a\": container with ID starting with da0387542447795de10080e05ff16384c4ec925ebb6021183f68263114349a9a not found: ID does not exist" Oct 30 20:18:20.398589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.398547 5091 scope.go:122] "RemoveContainer" containerID="88fde8d5cb82274a117df04703609db069be952cfb2c7a947c3f76c20d9f8f76" Oct 30 20:18:20.409124 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.409103 5091 scope.go:122] "RemoveContainer" containerID="88fde8d5cb82274a117df04703609db069be952cfb2c7a947c3f76c20d9f8f76" Oct 30 20:18:20.409413 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:18:20.409395 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88fde8d5cb82274a117df04703609db069be952cfb2c7a947c3f76c20d9f8f76\": container with ID starting with 88fde8d5cb82274a117df04703609db069be952cfb2c7a947c3f76c20d9f8f76 not found: ID does not exist" containerID="88fde8d5cb82274a117df04703609db069be952cfb2c7a947c3f76c20d9f8f76" Oct 30 20:18:20.409473 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.409423 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88fde8d5cb82274a117df04703609db069be952cfb2c7a947c3f76c20d9f8f76"} err="failed to get container status \"88fde8d5cb82274a117df04703609db069be952cfb2c7a947c3f76c20d9f8f76\": rpc error: code = NotFound desc = could not find container \"88fde8d5cb82274a117df04703609db069be952cfb2c7a947c3f76c20d9f8f76\": container with ID starting with 88fde8d5cb82274a117df04703609db069be952cfb2c7a947c3f76c20d9f8f76 not found: ID does not exist" Oct 30 20:18:20.421532 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.421506 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74"] Oct 30 20:18:20.448947 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.448917 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2a3e1b-5bff84b45d-sdp74"] Oct 30 20:18:20.449074 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.448957 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm"] Oct 30 20:18:20.453994 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.453966 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2a3e1b-7dd684b74c-gjmfm"] Oct 30 20:18:20.454337 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.454314 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfq6h\" (UniqueName: \"kubernetes.io/projected/02aaf432-1c15-4e88-9978-23dc3f9ee27d-kube-api-access-pfq6h\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:20.547811 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.541331 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:20.886176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:20.885997 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:21.031865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.031832 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:21.353109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.353072 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww"] Oct 30 20:18:21.354171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354130 5091 cpu_manager.go:414] "containerMap: removing container" podUID="97462176-f0ce-492c-bcad-1e125c73a4cd" containerName="revoke-keys" Oct 30 20:18:21.354228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354174 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97462176-f0ce-492c-bcad-1e125c73a4cd" containerName="revoke-keys" Oct 30 20:18:21.354264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354226 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c187d6c7-f5d8-41ee-a906-11e33e892cd5" containerName="revoke-keys" Oct 30 20:18:21.354264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354234 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c187d6c7-f5d8-41ee-a906-11e33e892cd5" containerName="revoke-keys" Oct 30 20:18:21.354264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354258 5091 cpu_manager.go:414] "containerMap: removing container" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" Oct 30 20:18:21.354407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354266 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" Oct 30 20:18:21.354407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354296 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b62a6025-b883-4a28-bdea-ab4078ffc1bf" containerName="revoke-keys" Oct 30 20:18:21.354407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354303 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b62a6025-b883-4a28-bdea-ab4078ffc1bf" containerName="revoke-keys" Oct 30 20:18:21.354407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354321 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f76728a5-e155-435a-a692-cc9afee10063" containerName="payload-processing" Oct 30 20:18:21.354407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354329 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76728a5-e155-435a-a692-cc9afee10063" containerName="payload-processing" Oct 30 20:18:21.354407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354358 5091 cpu_manager.go:414] "containerMap: removing container" podUID="02aaf432-1c15-4e88-9978-23dc3f9ee27d" containerName="payload-pre-processing" Oct 30 20:18:21.354407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354366 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02aaf432-1c15-4e88-9978-23dc3f9ee27d" containerName="payload-pre-processing" Oct 30 20:18:21.354407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354376 5091 cpu_manager.go:414] "containerMap: removing container" podUID="47417445-63ec-4320-8296-59c85ec05fa3" containerName="revoke-keys" Oct 30 20:18:21.354407 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354383 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47417445-63ec-4320-8296-59c85ec05fa3" containerName="revoke-keys" Oct 30 20:18:21.354724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354537 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b62a6025-b883-4a28-bdea-ab4078ffc1bf" containerName="revoke-keys" Oct 30 20:18:21.354724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354551 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76728a5-e155-435a-a692-cc9afee10063" containerName="payload-processing" Oct 30 20:18:21.354724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354565 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="47417445-63ec-4320-8296-59c85ec05fa3" containerName="revoke-keys" Oct 30 20:18:21.354724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354575 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="02aaf432-1c15-4e88-9978-23dc3f9ee27d" containerName="payload-pre-processing" Oct 30 20:18:21.354724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354586 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="97462176-f0ce-492c-bcad-1e125c73a4cd" containerName="revoke-keys" Oct 30 20:18:21.354724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354604 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c187d6c7-f5d8-41ee-a906-11e33e892cd5" containerName="revoke-keys" Oct 30 20:18:21.354724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.354614 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c14c99d-b8f0-4b81-ab2c-723529cb9dbb" containerName="main" Oct 30 20:18:21.372049 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.371849 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww"] Oct 30 20:18:21.372049 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.371982 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.394700 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.394665 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-credential-socket\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.394828 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.394761 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-data\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.394889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.394857 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-token\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.394937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.394903 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-workload-certs\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.394984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.394970 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istiod-ca-cert\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.395027 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.395011 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4hjr\" (UniqueName: \"kubernetes.io/projected/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-kube-api-access-t4hjr\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.395087 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.395072 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-envoy\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.395130 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.395102 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-podinfo\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.395193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.395172 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-workload-socket\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.438167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.437410 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:21.482170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.481396 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-bdf9e6-openshift-default-dockercfg-ffn9f\"" Oct 30 20:18:21.526171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.523587 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-token\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.526171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.523661 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-workload-certs\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.526171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.523754 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istiod-ca-cert\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.526171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.523795 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4hjr\" (UniqueName: \"kubernetes.io/projected/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-kube-api-access-t4hjr\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.526171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.523855 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-envoy\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.526171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.523885 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-podinfo\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.526171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.523938 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-workload-socket\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.526171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.524046 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-credential-socket\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.526171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.524133 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-data\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.526171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.524646 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-data\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.526171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.525310 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-workload-certs\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.533721 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.532640 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-workload-socket\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.533721 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.533342 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-credential-socket\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.553042 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.544390 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istiod-ca-cert\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.555930 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.555903 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-envoy\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.556323 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.556053 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-podinfo\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.567456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.560929 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-token\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.584523 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.584297 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4hjr\" (UniqueName: \"kubernetes.io/projected/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-kube-api-access-t4hjr\") pod \"e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.701785 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.701693 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:21.801196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.799458 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:21.940981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:21.940944 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:22.037687 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:18:22.037657 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6f27a0f_15c8_4266_b1ea_ced8d23c985b.slice/crio-bb8dbf65a3aaf3df8998e57adf45ab303ee97d0d48f75a2e5f5010b345867977 WatchSource:0}: Error finding container bb8dbf65a3aaf3df8998e57adf45ab303ee97d0d48f75a2e5f5010b345867977: Status 404 returned error can't find the container with id bb8dbf65a3aaf3df8998e57adf45ab303ee97d0d48f75a2e5f5010b345867977 Oct 30 20:18:22.039823 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:22.039795 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww"] Oct 30 20:18:22.040736 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:22.040704 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:18:22.040843 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:22.040781 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:18:22.040843 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:22.040827 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:18:22.412499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:22.412415 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" event={"ID":"a6f27a0f-15c8-4266-b1ea-ced8d23c985b","Type":"ContainerStarted","Data":"d985f317ad1a4318547f90ccecf7923b1d7a892a12ff5e2bc36ecabb6b47b9ce"} Oct 30 20:18:22.412499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:22.412492 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" event={"ID":"a6f27a0f-15c8-4266-b1ea-ced8d23c985b","Type":"ContainerStarted","Data":"bb8dbf65a3aaf3df8998e57adf45ab303ee97d0d48f75a2e5f5010b345867977"} Oct 30 20:18:22.442284 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:22.442227 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" podStartSLOduration=1.442212978 podStartE2EDuration="1.442212978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:22.439909435 +0000 UTC m=+2822.135117392" watchObservedRunningTime="2026-10-30 20:18:22.442212978 +0000 UTC m=+2822.137420935" Oct 30 20:18:22.443353 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:22.443330 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02aaf432-1c15-4e88-9978-23dc3f9ee27d" path="/var/lib/kubelet/pods/02aaf432-1c15-4e88-9978-23dc3f9ee27d/volumes" Oct 30 20:18:22.443791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:22.443768 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76728a5-e155-435a-a692-cc9afee10063" path="/var/lib/kubelet/pods/f76728a5-e155-435a-a692-cc9afee10063/volumes" Oct 30 20:18:22.702362 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:22.702278 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:22.966945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:22.966852 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:23.691845 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:23.691801 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:23.707924 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:23.707900 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:24.068851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:24.068811 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz"] Oct 30 20:18:24.074608 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:24.074583 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f06af6-7srzz"] Oct 30 20:18:24.082379 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:24.082342 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:24.233945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:24.233909 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:24.428646 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:24.428525 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:24.429579 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:24.429553 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:18:24.443509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:24.443480 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d4e116-fd32-41e2-a463-b1ff1a023828" path="/var/lib/kubelet/pods/09d4e116-fd32-41e2-a463-b1ff1a023828/volumes" Oct 30 20:18:24.573012 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:24.572972 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:24.839744 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:24.839710 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:25.149110 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:25.149030 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:25.302626 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:25.302587 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:26.108423 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.108381 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:26.243857 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.243826 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn"] Oct 30 20:18:26.276259 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.274529 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn"] Oct 30 20:18:26.276259 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.274583 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" Oct 30 20:18:26.292577 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.289459 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:26.308478 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.308106 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkj26\" (UniqueName: \"kubernetes.io/projected/71b363c1-4123-454f-86aa-daab26507f6e-kube-api-access-bkj26\") pod \"payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn\" (UID: \"71b363c1-4123-454f-86aa-daab26507f6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" Oct 30 20:18:26.308478 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.308193 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b363c1-4123-454f-86aa-daab26507f6e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn\" (UID: \"71b363c1-4123-454f-86aa-daab26507f6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" Oct 30 20:18:26.309039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.309016 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bdf9e6\"" Oct 30 20:18:26.309314 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.309294 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bdf9e6-dockercfg-btwc5\"" Oct 30 20:18:26.445274 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.427336 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkj26\" (UniqueName: \"kubernetes.io/projected/71b363c1-4123-454f-86aa-daab26507f6e-kube-api-access-bkj26\") pod \"payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn\" (UID: \"71b363c1-4123-454f-86aa-daab26507f6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" Oct 30 20:18:26.445274 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.427416 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b363c1-4123-454f-86aa-daab26507f6e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn\" (UID: \"71b363c1-4123-454f-86aa-daab26507f6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" Oct 30 20:18:26.477169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.446510 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b363c1-4123-454f-86aa-daab26507f6e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn\" (UID: \"71b363c1-4123-454f-86aa-daab26507f6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" Oct 30 20:18:26.477169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.450929 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:26.516167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.503849 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkj26\" (UniqueName: \"kubernetes.io/projected/71b363c1-4123-454f-86aa-daab26507f6e-kube-api-access-bkj26\") pod \"payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn\" (UID: \"71b363c1-4123-454f-86aa-daab26507f6e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" Oct 30 20:18:26.568170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.564679 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59"] Oct 30 20:18:26.568170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.564734 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-c1fdd6-rrf59"] Oct 30 20:18:26.587557 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.578731 5091 scope.go:122] "RemoveContainer" containerID="fbb75e3e5d98b78438aaeb6aef83c661f79a7c3432067bb38ef11af13a58c431" Oct 30 20:18:26.606682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.605390 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" Oct 30 20:18:26.666170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.665742 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:26.791387 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:26.780651 5091 scope.go:122] "RemoveContainer" containerID="dc4cdb8a18358da0f1946858cda5835fa1bef7609b6dac631b5fcf502b5128db" Oct 30 20:18:27.028940 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:27.028820 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:27.131392 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:18:27.131361 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71b363c1_4123_454f_86aa_daab26507f6e.slice/crio-edd163560e9081ff275c11345359ebfb399461e56f5d4772e7313ee1e17a7c9f WatchSource:0}: Error finding container edd163560e9081ff275c11345359ebfb399461e56f5d4772e7313ee1e17a7c9f: Status 404 returned error can't find the container with id edd163560e9081ff275c11345359ebfb399461e56f5d4772e7313ee1e17a7c9f Oct 30 20:18:27.131872 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:27.131851 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn"] Oct 30 20:18:27.513608 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:27.513572 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" event={"ID":"71b363c1-4123-454f-86aa-daab26507f6e","Type":"ContainerStarted","Data":"3f1ffb0d23ff1a69fa42df8d63ae842a7ea6768bdfb76f3746def3287920e7a2"} Oct 30 20:18:27.514062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:27.513628 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" event={"ID":"71b363c1-4123-454f-86aa-daab26507f6e","Type":"ContainerStarted","Data":"edd163560e9081ff275c11345359ebfb399461e56f5d4772e7313ee1e17a7c9f"} Oct 30 20:18:27.514062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:27.513845 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" Oct 30 20:18:27.545013 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:27.544960 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" podStartSLOduration=1.5449472850000001 podStartE2EDuration="1.544947285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:18:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:18:27.543671503 +0000 UTC m=+2827.238879462" watchObservedRunningTime="2026-10-30 20:18:27.544947285 +0000 UTC m=+2827.240155243" Oct 30 20:18:27.574482 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:27.574436 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:27.827581 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:27.827499 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:27.986190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:27.984538 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:28.442908 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:28.442872 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11dc06d8-aaf3-49bb-a6e9-3120b363a462" path="/var/lib/kubelet/pods/11dc06d8-aaf3-49bb-a6e9-3120b363a462/volumes" Oct 30 20:18:28.958844 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:28.958803 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:29.811862 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:18:29.811784 5091 logging.go:55] [core] [Channel #9783 SubChannel #9784]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.69:50052", ServerName: "10.132.0.69:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.69:50052: connect: connection refused" Oct 30 20:18:29.965775 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:29.965741 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" Oct 30 20:18:30.036840 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.036782 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c5214d-bdc6-4294-b702-704c79eaa0aa-kube-api-access-wlzv2\" (UniqueName: \"kubernetes.io/projected/77c5214d-bdc6-4294-b702-704c79eaa0aa-kube-api-access-wlzv2\") pod \"77c5214d-bdc6-4294-b702-704c79eaa0aa\" (UID: \"77c5214d-bdc6-4294-b702-704c79eaa0aa\") " Oct 30 20:18:30.037059 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.036928 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c5214d-bdc6-4294-b702-704c79eaa0aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c5214d-bdc6-4294-b702-704c79eaa0aa-plugins-config-volume\") pod \"77c5214d-bdc6-4294-b702-704c79eaa0aa\" (UID: \"77c5214d-bdc6-4294-b702-704c79eaa0aa\") " Oct 30 20:18:30.038706 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.038326 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c5214d-bdc6-4294-b702-704c79eaa0aa-plugins-config-volume" pod "77c5214d-bdc6-4294-b702-704c79eaa0aa" (UID: "77c5214d-bdc6-4294-b702-704c79eaa0aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:30.050882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.050803 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c5214d-bdc6-4294-b702-704c79eaa0aa-kube-api-access-wlzv2" pod "77c5214d-bdc6-4294-b702-704c79eaa0aa" (UID: "77c5214d-bdc6-4294-b702-704c79eaa0aa"). InnerVolumeSpecName "kube-api-access-wlzv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:30.061972 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.061698 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:30.141867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.141789 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlzv2\" (UniqueName: \"kubernetes.io/projected/77c5214d-bdc6-4294-b702-704c79eaa0aa-kube-api-access-wlzv2\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:30.141867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.141835 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/77c5214d-bdc6-4294-b702-704c79eaa0aa-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:30.634226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.634189 5091 generic.go:356] "Generic (PLEG): container finished" podID="77c5214d-bdc6-4294-b702-704c79eaa0aa" containerID="fe31f14a2f911716cde58898ba1e7e75f331e4b36ad089cf3915394547996c9c" exitCode=0 Oct 30 20:18:30.634428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.634375 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" event={"ID":"77c5214d-bdc6-4294-b702-704c79eaa0aa","Type":"ContainerDied","Data":"fe31f14a2f911716cde58898ba1e7e75f331e4b36ad089cf3915394547996c9c"} Oct 30 20:18:30.634504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.634430 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" event={"ID":"77c5214d-bdc6-4294-b702-704c79eaa0aa","Type":"ContainerDied","Data":"d838031941d3f7a1ebd582267d70e7051cf69ba93a858374639071548237e018"} Oct 30 20:18:30.634504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.634458 5091 scope.go:122] "RemoveContainer" containerID="fe31f14a2f911716cde58898ba1e7e75f331e4b36ad089cf3915394547996c9c" Oct 30 20:18:30.634644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.634627 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" Oct 30 20:18:30.662224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.661320 5091 scope.go:122] "RemoveContainer" containerID="fe31f14a2f911716cde58898ba1e7e75f331e4b36ad089cf3915394547996c9c" Oct 30 20:18:30.670402 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:18:30.670282 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe31f14a2f911716cde58898ba1e7e75f331e4b36ad089cf3915394547996c9c\": container with ID starting with fe31f14a2f911716cde58898ba1e7e75f331e4b36ad089cf3915394547996c9c not found: ID does not exist" containerID="fe31f14a2f911716cde58898ba1e7e75f331e4b36ad089cf3915394547996c9c" Oct 30 20:18:30.670402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.670336 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe31f14a2f911716cde58898ba1e7e75f331e4b36ad089cf3915394547996c9c"} err="failed to get container status \"fe31f14a2f911716cde58898ba1e7e75f331e4b36ad089cf3915394547996c9c\": rpc error: code = NotFound desc = could not find container \"fe31f14a2f911716cde58898ba1e7e75f331e4b36ad089cf3915394547996c9c\": container with ID starting with fe31f14a2f911716cde58898ba1e7e75f331e4b36ad089cf3915394547996c9c not found: ID does not exist" Oct 30 20:18:30.688375 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.688336 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs"] Oct 30 20:18:30.700633 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.700595 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs"] Oct 30 20:18:30.708471 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.708438 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:30.812137 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.812041 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f06af6-7fcf74dbd7-s9jrs" podUID="77c5214d-bdc6-4294-b702-704c79eaa0aa" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.69:50052\" within 1s: context deadline exceeded" Oct 30 20:18:30.812394 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:18:30.812132 5091 logging.go:55] [core] [Channel #9783 SubChannel #9784]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.69:50052", ServerName: "10.132.0.69:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.69:50052: operation was canceled" Oct 30 20:18:30.980051 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:30.975523 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:31.371172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:31.365750 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:31.389788 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:31.389742 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f"] Oct 30 20:18:31.394893 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:31.394864 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2a3e1b-xq92f"] Oct 30 20:18:31.539019 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:31.538985 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:31.972193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:31.972130 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:32.477518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:32.477475 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ab86d3-0a1e-44b3-8456-be82ab10e104" path="/var/lib/kubelet/pods/75ab86d3-0a1e-44b3-8456-be82ab10e104/volumes" Oct 30 20:18:32.478191 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:32.478129 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c5214d-bdc6-4294-b702-704c79eaa0aa" path="/var/lib/kubelet/pods/77c5214d-bdc6-4294-b702-704c79eaa0aa/volumes" Oct 30 20:18:32.604808 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:32.604766 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:33.814746 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:33.814703 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:34.867169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:34.856801 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:35.736382 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:35.736343 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:35.910121 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:35.910069 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:36.024983 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:36.024948 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2"] Oct 30 20:18:36.025215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:36.025186 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" podUID="fee7b3f2-9725-488d-8792-357afcb18f32" containerName="istio-proxy" containerID="cri-o://8a2da82dd2e092be74e0a4ee2229ae925dea685b434d5731ece1b163a6a0f59e" gracePeriod=30 Oct 30 20:18:36.147167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:36.137684 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:37.065214 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:37.065183 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:37.235743 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:37.235711 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:37.423011 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:37.422931 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2"] Oct 30 20:18:37.423214 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:37.423184 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerName="istio-proxy" containerID="cri-o://03b59195e4f6f2c3ac93f70feca33c6f70bc8def4f1822008f3b54dfadf4ae65" gracePeriod=30 Oct 30 20:18:37.505135 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:37.505096 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:38.392173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:38.386184 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:38.551964 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:38.551923 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:38.605048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:38.605005 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" Oct 30 20:18:38.718866 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:38.718772 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:38.899134 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:38.893344 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:39.059315 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:39.059281 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:39.494443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:39.494351 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:39.875213 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:39.875176 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:40.402013 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:40.401983 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:40.563416 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:40.563377 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:40.793050 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:40.793013 5091 generic.go:356] "Generic (PLEG): container finished" podID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" containerID="f834c3bc0ffbb915da2643b1434f92a79d4a8a71e0eb0914ad67c8e29cfab6a7" exitCode=0 Oct 30 20:18:40.793295 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:40.793098 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" event={"ID":"320dc7ae-3660-41d1-bc53-49d3427a0b6d","Type":"ContainerDied","Data":"f834c3bc0ffbb915da2643b1434f92a79d4a8a71e0eb0914ad67c8e29cfab6a7"} Oct 30 20:18:40.793677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:40.793656 5091 scope.go:122] "RemoveContainer" containerID="f834c3bc0ffbb915da2643b1434f92a79d4a8a71e0eb0914ad67c8e29cfab6a7" Oct 30 20:18:40.795482 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:40.795451 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:18:40.795574 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:40.795522 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:18:40.795574 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:40.795561 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:18:41.391913 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.391640 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:41.484260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.484233 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 20:18:41.535934 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.535894 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-workload-certs\") pod \"fee7b3f2-9725-488d-8792-357afcb18f32\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " Oct 30 20:18:41.536182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.536009 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-workload-socket\") pod \"fee7b3f2-9725-488d-8792-357afcb18f32\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " Oct 30 20:18:41.536182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.536058 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-credential-socket\") pod \"fee7b3f2-9725-488d-8792-357afcb18f32\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " Oct 30 20:18:41.536319 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.536190 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-istio-data\") pod \"fee7b3f2-9725-488d-8792-357afcb18f32\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " Oct 30 20:18:41.536319 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.536249 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee7b3f2-9725-488d-8792-357afcb18f32-istio-token\" (UniqueName: \"kubernetes.io/projected/fee7b3f2-9725-488d-8792-357afcb18f32-istio-token\") pod \"fee7b3f2-9725-488d-8792-357afcb18f32\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " Oct 30 20:18:41.536319 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.536300 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fee7b3f2-9725-488d-8792-357afcb18f32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fee7b3f2-9725-488d-8792-357afcb18f32-istio-podinfo\") pod \"fee7b3f2-9725-488d-8792-357afcb18f32\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " Oct 30 20:18:41.536479 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.536335 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-istio-envoy\") pod \"fee7b3f2-9725-488d-8792-357afcb18f32\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " Oct 30 20:18:41.536479 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.536372 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee7b3f2-9725-488d-8792-357afcb18f32-kube-api-access-czmxm\" (UniqueName: \"kubernetes.io/projected/fee7b3f2-9725-488d-8792-357afcb18f32-kube-api-access-czmxm\") pod \"fee7b3f2-9725-488d-8792-357afcb18f32\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " Oct 30 20:18:41.536479 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.536437 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee7b3f2-9725-488d-8792-357afcb18f32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fee7b3f2-9725-488d-8792-357afcb18f32-istiod-ca-cert\") pod \"fee7b3f2-9725-488d-8792-357afcb18f32\" (UID: \"fee7b3f2-9725-488d-8792-357afcb18f32\") " Oct 30 20:18:41.536929 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.536899 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-istio-data" pod "fee7b3f2-9725-488d-8792-357afcb18f32" (UID: "fee7b3f2-9725-488d-8792-357afcb18f32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:41.537116 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.537098 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:41.537408 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.537370 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee7b3f2-9725-488d-8792-357afcb18f32-istiod-ca-cert" pod "fee7b3f2-9725-488d-8792-357afcb18f32" (UID: "fee7b3f2-9725-488d-8792-357afcb18f32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:41.537408 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.537388 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-workload-certs" pod "fee7b3f2-9725-488d-8792-357afcb18f32" (UID: "fee7b3f2-9725-488d-8792-357afcb18f32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:41.537592 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.537575 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-workload-socket" pod "fee7b3f2-9725-488d-8792-357afcb18f32" (UID: "fee7b3f2-9725-488d-8792-357afcb18f32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:41.538018 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.537994 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-credential-socket" pod "fee7b3f2-9725-488d-8792-357afcb18f32" (UID: "fee7b3f2-9725-488d-8792-357afcb18f32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:41.540271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.540249 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-istio-envoy" pod "fee7b3f2-9725-488d-8792-357afcb18f32" (UID: "fee7b3f2-9725-488d-8792-357afcb18f32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:41.541260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.541182 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee7b3f2-9725-488d-8792-357afcb18f32-istio-token" pod "fee7b3f2-9725-488d-8792-357afcb18f32" (UID: "fee7b3f2-9725-488d-8792-357afcb18f32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:41.541639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.541619 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee7b3f2-9725-488d-8792-357afcb18f32-kube-api-access-czmxm" pod "fee7b3f2-9725-488d-8792-357afcb18f32" (UID: "fee7b3f2-9725-488d-8792-357afcb18f32"). InnerVolumeSpecName "kube-api-access-czmxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:41.541948 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.541923 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fee7b3f2-9725-488d-8792-357afcb18f32-istio-podinfo" pod "fee7b3f2-9725-488d-8792-357afcb18f32" (UID: "fee7b3f2-9725-488d-8792-357afcb18f32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:18:41.550996 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.550967 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:41.638093 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.638059 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:41.638093 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.638088 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:41.638522 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.638100 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:41.638522 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.638112 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fee7b3f2-9725-488d-8792-357afcb18f32-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:41.638522 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.638121 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fee7b3f2-9725-488d-8792-357afcb18f32-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:41.638522 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.638130 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fee7b3f2-9725-488d-8792-357afcb18f32-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:41.638522 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.638141 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czmxm\" (UniqueName: \"kubernetes.io/projected/fee7b3f2-9725-488d-8792-357afcb18f32-kube-api-access-czmxm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:41.638522 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.638180 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fee7b3f2-9725-488d-8792-357afcb18f32-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:41.702836 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.701869 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:41.803291 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.803254 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" event={"ID":"320dc7ae-3660-41d1-bc53-49d3427a0b6d","Type":"ContainerStarted","Data":"a32916c06e9e991d8c6eb389b20d3666185bfb7e766dc3a3a6511b7c2b8dc3d0"} Oct 30 20:18:41.804887 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.804860 5091 generic.go:356] "Generic (PLEG): container finished" podID="fee7b3f2-9725-488d-8792-357afcb18f32" containerID="8a2da82dd2e092be74e0a4ee2229ae925dea685b434d5731ece1b163a6a0f59e" exitCode=0 Oct 30 20:18:41.805023 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.804900 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" Oct 30 20:18:41.805023 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.804935 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" event={"ID":"fee7b3f2-9725-488d-8792-357afcb18f32","Type":"ContainerDied","Data":"8a2da82dd2e092be74e0a4ee2229ae925dea685b434d5731ece1b163a6a0f59e"} Oct 30 20:18:41.805023 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.804982 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2" event={"ID":"fee7b3f2-9725-488d-8792-357afcb18f32","Type":"ContainerDied","Data":"d5e1ac03298c99b65ba91b93c56af922e47290a1e9e347396fa436dd03ec65e1"} Oct 30 20:18:41.805023 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.805005 5091 scope.go:122] "RemoveContainer" containerID="8a2da82dd2e092be74e0a4ee2229ae925dea685b434d5731ece1b163a6a0f59e" Oct 30 20:18:41.816460 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.816442 5091 scope.go:122] "RemoveContainer" containerID="8a2da82dd2e092be74e0a4ee2229ae925dea685b434d5731ece1b163a6a0f59e" Oct 30 20:18:41.816786 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:18:41.816762 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a2da82dd2e092be74e0a4ee2229ae925dea685b434d5731ece1b163a6a0f59e\": container with ID starting with 8a2da82dd2e092be74e0a4ee2229ae925dea685b434d5731ece1b163a6a0f59e not found: ID does not exist" containerID="8a2da82dd2e092be74e0a4ee2229ae925dea685b434d5731ece1b163a6a0f59e" Oct 30 20:18:41.816848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.816791 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a2da82dd2e092be74e0a4ee2229ae925dea685b434d5731ece1b163a6a0f59e"} err="failed to get container status \"8a2da82dd2e092be74e0a4ee2229ae925dea685b434d5731ece1b163a6a0f59e\": rpc error: code = NotFound desc = could not find container \"8a2da82dd2e092be74e0a4ee2229ae925dea685b434d5731ece1b163a6a0f59e\": container with ID starting with 8a2da82dd2e092be74e0a4ee2229ae925dea685b434d5731ece1b163a6a0f59e not found: ID does not exist" Oct 30 20:18:41.942263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.942176 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2"] Oct 30 20:18:41.946866 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.946828 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f06af6-openshift-default-f48f877dc-97tr2"] Oct 30 20:18:41.986450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.984518 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:18:41.987109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.986777 5091 patch_prober.go:29] interesting pod/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Oct 30 20:18:41.987109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:41.986834 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" podUID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Oct 30 20:18:42.483170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:42.475395 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee7b3f2-9725-488d-8792-357afcb18f32" path="/var/lib/kubelet/pods/fee7b3f2-9725-488d-8792-357afcb18f32/volumes" Oct 30 20:18:42.618027 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:42.617982 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:42.773169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:42.772388 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:42.830754 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:42.830713 5091 generic.go:356] "Generic (PLEG): container finished" podID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerID="03b59195e4f6f2c3ac93f70feca33c6f70bc8def4f1822008f3b54dfadf4ae65" exitCode=0 Oct 30 20:18:42.831020 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:42.830995 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" event={"ID":"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d","Type":"ContainerDied","Data":"03b59195e4f6f2c3ac93f70feca33c6f70bc8def4f1822008f3b54dfadf4ae65"} Oct 30 20:18:42.831082 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:42.831068 5091 scope.go:122] "RemoveContainer" containerID="31057ad55c8374fd1f70af888ca4405fe2163b86852725c4f093f48249ea9145" Oct 30 20:18:42.971170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:42.967340 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 20:18:42.989233 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:42.989198 5091 patch_prober.go:29] interesting pod/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Oct 30 20:18:42.989417 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:42.989251 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" podUID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Oct 30 20:18:42.990648 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:42.990609 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:43.147226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.123058 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istiod-ca-cert\") pod \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " Oct 30 20:18:43.147746 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.147711 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-credential-socket\") pod \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " Oct 30 20:18:43.148011 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.147993 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-envoy\") pod \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " Oct 30 20:18:43.148260 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.148241 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-podinfo\") pod \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " Oct 30 20:18:43.148434 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.148418 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-workload-certs\") pod \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " Oct 30 20:18:43.148544 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.148530 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-kube-api-access-rdcq6\" (UniqueName: \"kubernetes.io/projected/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-kube-api-access-rdcq6\") pod \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " Oct 30 20:18:43.148694 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.148678 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-token\" (UniqueName: \"kubernetes.io/projected/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-token\") pod \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " Oct 30 20:18:43.148851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.148837 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-data\") pod \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " Oct 30 20:18:43.148969 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.148956 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-workload-socket\") pod \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\" (UID: \"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d\") " Oct 30 20:18:43.160168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.157532 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-workload-socket" pod "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" (UID: "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:43.170168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.167309 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-workload-certs" pod "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" (UID: "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:43.170168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.167434 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-data" pod "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" (UID: "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:43.170168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.167658 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-credential-socket" pod "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" (UID: "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:43.180176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.179658 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istiod-ca-cert" pod "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" (UID: "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:18:43.196321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.195534 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-podinfo" pod "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" (UID: "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:18:43.242453 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.239169 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-kube-api-access-rdcq6" pod "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" (UID: "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d"). InnerVolumeSpecName "kube-api-access-rdcq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:43.266195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.249748 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-envoy" pod "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" (UID: "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:18:43.266195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.251119 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:43.276287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.275356 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdcq6\" (UniqueName: \"kubernetes.io/projected/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-kube-api-access-rdcq6\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:43.276287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.275396 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:43.276287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.275410 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:43.276287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.275475 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:43.276287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.275493 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:43.276287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.275508 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:43.276287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.275523 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:43.276287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.276013 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-token" pod "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" (UID: "e65edc8b-97e7-40ac-9f21-e1ffd3aff71d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:18:43.379207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.377450 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:18:43.703249 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.703202 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:43.847515 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.845479 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" event={"ID":"e65edc8b-97e7-40ac-9f21-e1ffd3aff71d","Type":"ContainerDied","Data":"1e51e8015e7655b3aff01b8f125db14e97a13b0fe42b35b52e12ec83d20d237e"} Oct 30 20:18:43.847515 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.845554 5091 scope.go:122] "RemoveContainer" containerID="03b59195e4f6f2c3ac93f70feca33c6f70bc8def4f1822008f3b54dfadf4ae65" Oct 30 20:18:43.847515 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.845675 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2" Oct 30 20:18:43.884358 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.884310 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:43.907554 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.907526 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2"] Oct 30 20:18:43.916883 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.916277 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-c1fdd6-openshift-default-5845b6b8cf-t5cl2"] Oct 30 20:18:43.984658 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.984573 5091 patch_prober.go:29] interesting pod/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Oct 30 20:18:43.984658 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:43.984642 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" podUID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Oct 30 20:18:44.443253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:44.443212 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" path="/var/lib/kubelet/pods/e65edc8b-97e7-40ac-9f21-e1ffd3aff71d/volumes" Oct 30 20:18:44.570405 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:44.570369 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:18:44.796096 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:44.796051 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:44.984756 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:44.984723 5091 patch_prober.go:29] interesting pod/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Oct 30 20:18:44.985252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:44.984804 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" podUID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Oct 30 20:18:45.807170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:45.807093 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:45.984933 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:45.984881 5091 patch_prober.go:29] interesting pod/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Oct 30 20:18:45.985358 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:45.984988 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" podUID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Oct 30 20:18:46.006656 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:18:46.006618 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:18:46.976763 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:46.976727 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:46.984974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:46.984944 5091 patch_prober.go:29] interesting pod/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" start-of-body= Oct 30 20:18:46.985461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:46.985433 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" podUID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.126:15021/healthz/ready\": dial tcp 10.132.1.126:15021: connect: connection refused" Oct 30 20:18:47.989791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:47.989750 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:18:47.990987 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:47.990961 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:18:48.481575 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:48.481535 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:48.722481 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:48.722437 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:49.308178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:49.308119 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:50.189168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:50.183560 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:50.459065 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:50.458955 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:51.096172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:51.092271 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:51.926216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:51.920976 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:52.733271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:52.733218 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:53.713300 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:53.712114 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:54.590649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:54.590612 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:55.578197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:55.576102 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:56.625176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:56.624859 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:57.711720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:57.711679 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:58.669063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:58.669029 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:18:59.552366 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:18:59.552326 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:00.358188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:00.358124 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:00.514129 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:00.514078 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:01.572132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:01.572097 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:02.262559 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:02.262521 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:02.432902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:02.432815 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:03.210210 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:03.210168 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:04.130465 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:04.130204 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:05.007171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:05.005209 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:05.881171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:05.869945 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:06.153503 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:06.153442 5091 generic.go:356] "Generic (PLEG): container finished" podID="a6f27a0f-15c8-4266-b1ea-ced8d23c985b" containerID="d985f317ad1a4318547f90ccecf7923b1d7a892a12ff5e2bc36ecabb6b47b9ce" exitCode=0 Oct 30 20:19:06.153967 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:06.153517 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" event={"ID":"a6f27a0f-15c8-4266-b1ea-ced8d23c985b","Type":"ContainerDied","Data":"d985f317ad1a4318547f90ccecf7923b1d7a892a12ff5e2bc36ecabb6b47b9ce"} Oct 30 20:19:06.154079 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:06.154059 5091 scope.go:122] "RemoveContainer" containerID="d985f317ad1a4318547f90ccecf7923b1d7a892a12ff5e2bc36ecabb6b47b9ce" Oct 30 20:19:06.155977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:06.155946 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:19:06.156108 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:06.156029 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:19:06.156108 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:06.156062 5091 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236084Ki","pods":"250"} Oct 30 20:19:06.806346 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:06.806300 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:07.191985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:07.191898 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" event={"ID":"a6f27a0f-15c8-4266-b1ea-ced8d23c985b","Type":"ContainerStarted","Data":"309a8d7107e3b7b2dc23f168ba3f1cb62096e1692403c34613c7c9959c67b878"} Oct 30 20:19:07.702164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:07.701690 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:07.704341 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:07.703175 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:19:07.708433 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:07.707720 5091 patch_prober.go:29] interesting pod/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body= Oct 30 20:19:07.708433 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:07.707803 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" podUID="a6f27a0f-15c8-4266-b1ea-ced8d23c985b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" Oct 30 20:19:08.510894 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:08.510850 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:08.703132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:08.703094 5091 patch_prober.go:29] interesting pod/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body= Oct 30 20:19:08.703332 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:08.703177 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" podUID="a6f27a0f-15c8-4266-b1ea-ced8d23c985b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" Oct 30 20:19:09.335341 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:09.335300 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:09.429279 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:09.429244 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:19:09.605805 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:09.605716 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:09.705636 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:09.705598 5091 patch_prober.go:29] interesting pod/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body= Oct 30 20:19:09.705925 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:09.705895 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" podUID="a6f27a0f-15c8-4266-b1ea-ced8d23c985b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" Oct 30 20:19:10.488499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:10.488448 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:10.703368 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:10.703331 5091 patch_prober.go:29] interesting pod/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body= Oct 30 20:19:10.703785 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:10.703404 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" podUID="a6f27a0f-15c8-4266-b1ea-ced8d23c985b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" Oct 30 20:19:11.542714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:11.542615 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:11.702799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:11.702762 5091 patch_prober.go:29] interesting pod/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" start-of-body= Oct 30 20:19:11.703022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:11.702825 5091 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" podUID="a6f27a0f-15c8-4266-b1ea-ced8d23c985b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.138:15021/healthz/ready\": dial tcp 10.132.1.138:15021: connect: connection refused" Oct 30 20:19:12.471168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:12.471117 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:12.707271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:12.707239 5091 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:19:12.708265 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:12.708240 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:19:13.363906 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:13.363863 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:14.179349 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:14.179269 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:14.946851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:14.946813 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:15.090129 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:15.090094 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:15.790344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:15.790306 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:15.977401 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:19:15.977359 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:19:16.792029 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:16.791943 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:17.804459 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:17.804419 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:18.682445 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:18.682409 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:19.567167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:19.557100 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:20.379923 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:20.379872 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:21.342406 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:21.342338 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:22.173437 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:22.173395 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:22.966744 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:22.966691 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:23.932174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:23.931601 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:24.797720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:24.795425 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:25.998170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:25.997768 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:26.943706 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:26.943648 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:27.804182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:27.803113 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:28.681171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:28.677282 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:29.657002 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:29.656954 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:30.485513 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:30.485430 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:31.302169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:31.294070 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:32.182017 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:32.181974 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:33.061197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:33.061141 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:33.849538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:33.849500 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:34.659352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:34.659313 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:35.482398 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:35.482357 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:36.322978 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:36.322942 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:37.146741 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:37.146694 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:38.033573 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:38.033531 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp"] Oct 30 20:19:38.043158 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:38.043116 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1252f0d1-ngkjp"] Oct 30 20:19:38.177545 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:38.177507 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:38.442084 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:38.441996 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e26c2b6-aafc-4eec-a7ab-eb24d6c456de" path="/var/lib/kubelet/pods/1e26c2b6-aafc-4eec-a7ab-eb24d6c456de/volumes" Oct 30 20:19:39.175527 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:39.175486 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:40.178763 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:40.176284 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:41.106169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:41.100636 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:41.863791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:41.863755 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:42.629379 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:42.629337 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:43.489360 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:43.489295 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:44.397746 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:44.396223 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:45.284943 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:45.284900 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:45.979761 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:19:45.979720 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:19:46.259003 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:46.258966 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:47.126234 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:47.126196 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:47.954269 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:47.954195 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:48.796476 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:48.796093 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:49.576357 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:49.576319 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:49.815604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:49.815565 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:50.492917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:50.492886 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:51.304258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:51.304219 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:52.135747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:52.132514 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:53.093206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:53.093165 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:53.922268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:53.922217 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:54.855892 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:54.855848 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:55.780170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:55.779168 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:56.709287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:56.709245 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:57.736945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:57.736904 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:58.513921 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:58.513884 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:19:59.392401 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:19:59.392363 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:00.285593 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:00.285545 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:01.132031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.131939 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn"] Oct 30 20:20:01.132668 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.132649 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerName="istio-proxy" Oct 30 20:20:01.132718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.132669 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerName="istio-proxy" Oct 30 20:20:01.132718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.132683 5091 cpu_manager.go:414] "containerMap: removing container" podUID="77c5214d-bdc6-4294-b702-704c79eaa0aa" containerName="payload-pre-processing" Oct 30 20:20:01.132718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.132688 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c5214d-bdc6-4294-b702-704c79eaa0aa" containerName="payload-pre-processing" Oct 30 20:20:01.132718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.132698 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fee7b3f2-9725-488d-8792-357afcb18f32" containerName="istio-proxy" Oct 30 20:20:01.132718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.132703 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee7b3f2-9725-488d-8792-357afcb18f32" containerName="istio-proxy" Oct 30 20:20:01.132886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.132742 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerName="istio-proxy" Oct 30 20:20:01.132886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.132747 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerName="istio-proxy" Oct 30 20:20:01.132886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.132831 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerName="istio-proxy" Oct 30 20:20:01.132886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.132841 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c5214d-bdc6-4294-b702-704c79eaa0aa" containerName="payload-pre-processing" Oct 30 20:20:01.132886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.132850 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee7b3f2-9725-488d-8792-357afcb18f32" containerName="istio-proxy" Oct 30 20:20:01.136912 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.136886 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" Oct 30 20:20:01.143650 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.143610 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:20:01.143822 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.143702 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:20:01.147915 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.147890 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn"] Oct 30 20:20:01.228019 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.227985 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:01.279611 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.279570 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f372eb8c-e213-4899-9819-fb9ecb84ed89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d004ccf-nlptn\" (UID: \"f372eb8c-e213-4899-9819-fb9ecb84ed89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" Oct 30 20:20:01.279784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.279629 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f372eb8c-e213-4899-9819-fb9ecb84ed89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d004ccf-nlptn\" (UID: \"f372eb8c-e213-4899-9819-fb9ecb84ed89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" Oct 30 20:20:01.279784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.279735 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnxd9\" (UniqueName: \"kubernetes.io/projected/f372eb8c-e213-4899-9819-fb9ecb84ed89-kube-api-access-nnxd9\") pod \"maas-api-revoke-sub-7d004ccf-nlptn\" (UID: \"f372eb8c-e213-4899-9819-fb9ecb84ed89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" Oct 30 20:20:01.381672 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.381639 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f372eb8c-e213-4899-9819-fb9ecb84ed89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d004ccf-nlptn\" (UID: \"f372eb8c-e213-4899-9819-fb9ecb84ed89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" Oct 30 20:20:01.381878 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.381696 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f372eb8c-e213-4899-9819-fb9ecb84ed89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d004ccf-nlptn\" (UID: \"f372eb8c-e213-4899-9819-fb9ecb84ed89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" Oct 30 20:20:01.381878 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.381755 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnxd9\" (UniqueName: \"kubernetes.io/projected/f372eb8c-e213-4899-9819-fb9ecb84ed89-kube-api-access-nnxd9\") pod \"maas-api-revoke-sub-7d004ccf-nlptn\" (UID: \"f372eb8c-e213-4899-9819-fb9ecb84ed89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" Oct 30 20:20:01.382512 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.382451 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f372eb8c-e213-4899-9819-fb9ecb84ed89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d004ccf-nlptn\" (UID: \"f372eb8c-e213-4899-9819-fb9ecb84ed89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" Oct 30 20:20:01.383945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.383920 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f372eb8c-e213-4899-9819-fb9ecb84ed89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d004ccf-nlptn\" (UID: \"f372eb8c-e213-4899-9819-fb9ecb84ed89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" Oct 30 20:20:01.396271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.396244 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnxd9\" (UniqueName: \"kubernetes.io/projected/f372eb8c-e213-4899-9819-fb9ecb84ed89-kube-api-access-nnxd9\") pod \"maas-api-revoke-sub-7d004ccf-nlptn\" (UID: \"f372eb8c-e213-4899-9819-fb9ecb84ed89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" Oct 30 20:20:01.454931 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.454895 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" Oct 30 20:20:01.754585 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:20:01.754556 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf372eb8c_e213_4899_9819_fb9ecb84ed89.slice/crio-e7d7c336301eda5c00dbc402a5b6825d74fffc62b33d4c8aa6b7fc0753786008 WatchSource:0}: Error finding container e7d7c336301eda5c00dbc402a5b6825d74fffc62b33d4c8aa6b7fc0753786008: Status 404 returned error can't find the container with id e7d7c336301eda5c00dbc402a5b6825d74fffc62b33d4c8aa6b7fc0753786008 Oct 30 20:20:01.756562 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:01.756536 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn"] Oct 30 20:20:02.215086 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:02.215038 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:02.757128 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:02.757087 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" event={"ID":"f372eb8c-e213-4899-9819-fb9ecb84ed89","Type":"ContainerStarted","Data":"1509a28f18a5ce1274b080ecf181656b721d5dd13fc7bc2f901ca5cc8302b1db"} Oct 30 20:20:02.757344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:02.757167 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" event={"ID":"f372eb8c-e213-4899-9819-fb9ecb84ed89","Type":"ContainerStarted","Data":"e7d7c336301eda5c00dbc402a5b6825d74fffc62b33d4c8aa6b7fc0753786008"} Oct 30 20:20:02.785351 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:02.785299 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" podStartSLOduration=1.785284679 podStartE2EDuration="1.785284679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:20:02.780322175 +0000 UTC m=+2922.475530136" watchObservedRunningTime="2026-10-30 20:20:02.785284679 +0000 UTC m=+2922.480492636" Oct 30 20:20:03.137172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:03.136811 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:03.766797 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:03.766760 5091 generic.go:356] "Generic (PLEG): container finished" podID="f372eb8c-e213-4899-9819-fb9ecb84ed89" containerID="1509a28f18a5ce1274b080ecf181656b721d5dd13fc7bc2f901ca5cc8302b1db" exitCode=0 Oct 30 20:20:03.767291 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:03.766851 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" event={"ID":"f372eb8c-e213-4899-9819-fb9ecb84ed89","Type":"ContainerDied","Data":"1509a28f18a5ce1274b080ecf181656b721d5dd13fc7bc2f901ca5cc8302b1db"} Oct 30 20:20:03.964356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:03.964318 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:04.788172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:04.786253 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:04.965887 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:04.965861 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" Oct 30 20:20:05.048298 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:05.048205 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f372eb8c-e213-4899-9819-fb9ecb84ed89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f372eb8c-e213-4899-9819-fb9ecb84ed89-maas-api-openshift-ca\") pod \"f372eb8c-e213-4899-9819-fb9ecb84ed89\" (UID: \"f372eb8c-e213-4899-9819-fb9ecb84ed89\") " Oct 30 20:20:05.048298 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:05.048275 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f372eb8c-e213-4899-9819-fb9ecb84ed89-kube-api-access-nnxd9\" (UniqueName: \"kubernetes.io/projected/f372eb8c-e213-4899-9819-fb9ecb84ed89-kube-api-access-nnxd9\") pod \"f372eb8c-e213-4899-9819-fb9ecb84ed89\" (UID: \"f372eb8c-e213-4899-9819-fb9ecb84ed89\") " Oct 30 20:20:05.048545 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:05.048335 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f372eb8c-e213-4899-9819-fb9ecb84ed89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f372eb8c-e213-4899-9819-fb9ecb84ed89-maas-api-xks-ca\") pod \"f372eb8c-e213-4899-9819-fb9ecb84ed89\" (UID: \"f372eb8c-e213-4899-9819-fb9ecb84ed89\") " Oct 30 20:20:05.048685 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:05.048654 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f372eb8c-e213-4899-9819-fb9ecb84ed89-maas-api-openshift-ca" pod "f372eb8c-e213-4899-9819-fb9ecb84ed89" (UID: "f372eb8c-e213-4899-9819-fb9ecb84ed89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:05.048995 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:05.048978 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f372eb8c-e213-4899-9819-fb9ecb84ed89-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:05.050477 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:05.050446 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f372eb8c-e213-4899-9819-fb9ecb84ed89-maas-api-xks-ca" pod "f372eb8c-e213-4899-9819-fb9ecb84ed89" (UID: "f372eb8c-e213-4899-9819-fb9ecb84ed89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:20:05.050588 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:05.050560 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f372eb8c-e213-4899-9819-fb9ecb84ed89-kube-api-access-nnxd9" pod "f372eb8c-e213-4899-9819-fb9ecb84ed89" (UID: "f372eb8c-e213-4899-9819-fb9ecb84ed89"). InnerVolumeSpecName "kube-api-access-nnxd9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:05.151123 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:05.151077 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f372eb8c-e213-4899-9819-fb9ecb84ed89-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:05.151123 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:05.151120 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnxd9\" (UniqueName: \"kubernetes.io/projected/f372eb8c-e213-4899-9819-fb9ecb84ed89-kube-api-access-nnxd9\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:05.577065 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:05.577009 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:05.803720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:05.803687 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" event={"ID":"f372eb8c-e213-4899-9819-fb9ecb84ed89","Type":"ContainerDied","Data":"e7d7c336301eda5c00dbc402a5b6825d74fffc62b33d4c8aa6b7fc0753786008"} Oct 30 20:20:05.804159 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:05.803740 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7d7c336301eda5c00dbc402a5b6825d74fffc62b33d4c8aa6b7fc0753786008" Oct 30 20:20:05.804159 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:05.803693 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn" Oct 30 20:20:06.484255 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:06.480715 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:07.393180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:07.392881 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:08.301256 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:08.301158 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:09.154953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:09.154917 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:10.160371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:10.160317 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:10.997667 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:10.997621 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:11.863904 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:11.863859 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:12.040472 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.040421 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:12.259018 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.258983 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj"] Oct 30 20:20:12.259773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.259752 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f372eb8c-e213-4899-9819-fb9ecb84ed89" containerName="revoke-keys" Oct 30 20:20:12.259824 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.259775 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f372eb8c-e213-4899-9819-fb9ecb84ed89" containerName="revoke-keys" Oct 30 20:20:12.259895 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.259886 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f372eb8c-e213-4899-9819-fb9ecb84ed89" containerName="revoke-keys" Oct 30 20:20:12.259933 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.259898 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e65edc8b-97e7-40ac-9f21-e1ffd3aff71d" containerName="istio-proxy" Oct 30 20:20:12.263528 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.263508 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" Oct 30 20:20:12.275645 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.275619 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj"] Oct 30 20:20:12.276472 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.276449 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:20:12.276627 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.276505 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:20:12.363225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.363179 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t47d\" (UniqueName: \"kubernetes.io/projected/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-kube-api-access-9t47d\") pod \"maas-api-revoke-sub-cf4ff9e7-nmmrj\" (UID: \"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" Oct 30 20:20:12.363422 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.363350 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4ff9e7-nmmrj\" (UID: \"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" Oct 30 20:20:12.363492 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.363476 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4ff9e7-nmmrj\" (UID: \"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" Oct 30 20:20:12.464381 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.464340 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4ff9e7-nmmrj\" (UID: \"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" Oct 30 20:20:12.464665 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.464632 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t47d\" (UniqueName: \"kubernetes.io/projected/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-kube-api-access-9t47d\") pod \"maas-api-revoke-sub-cf4ff9e7-nmmrj\" (UID: \"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" Oct 30 20:20:12.464829 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.464727 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4ff9e7-nmmrj\" (UID: \"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" Oct 30 20:20:12.465427 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.465406 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf4ff9e7-nmmrj\" (UID: \"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" Oct 30 20:20:12.467012 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.466991 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf4ff9e7-nmmrj\" (UID: \"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" Oct 30 20:20:12.478399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.478377 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t47d\" (UniqueName: \"kubernetes.io/projected/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-kube-api-access-9t47d\") pod \"maas-api-revoke-sub-cf4ff9e7-nmmrj\" (UID: \"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" Oct 30 20:20:12.584679 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.584602 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" Oct 30 20:20:12.873743 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:20:12.873711 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode468b395_2a21_4cc9_a82f_0e6d1e9ecc2b.slice/crio-0277a494b69dd1caa6dab2b9e058ef96c9257e6d0c65521abdfdf9e06588493f WatchSource:0}: Error finding container 0277a494b69dd1caa6dab2b9e058ef96c9257e6d0c65521abdfdf9e06588493f: Status 404 returned error can't find the container with id 0277a494b69dd1caa6dab2b9e058ef96c9257e6d0c65521abdfdf9e06588493f Oct 30 20:20:12.874457 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:12.874429 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj"] Oct 30 20:20:13.088486 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:13.085174 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:13.876784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:13.876746 5091 generic.go:356] "Generic (PLEG): container finished" podID="e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b" containerID="f5ced9b208480f9d83140d2a2a921465082bd4eedf1d66b4af29c65ea85548ce" exitCode=0 Oct 30 20:20:13.877161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:13.876830 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" event={"ID":"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b","Type":"ContainerDied","Data":"f5ced9b208480f9d83140d2a2a921465082bd4eedf1d66b4af29c65ea85548ce"} Oct 30 20:20:13.877161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:13.876877 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" event={"ID":"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b","Type":"ContainerStarted","Data":"0277a494b69dd1caa6dab2b9e058ef96c9257e6d0c65521abdfdf9e06588493f"} Oct 30 20:20:13.996961 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:13.989130 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:14.903040 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:14.903001 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:15.083398 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:15.083367 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" Oct 30 20:20:15.103830 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:15.103806 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-maas-api-openshift-ca\") pod \"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b\" (UID: \"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b\") " Oct 30 20:20:15.104070 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:15.104051 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-kube-api-access-9t47d\" (UniqueName: \"kubernetes.io/projected/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-kube-api-access-9t47d\") pod \"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b\" (UID: \"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b\") " Oct 30 20:20:15.104179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:15.104141 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-maas-api-openshift-ca" pod "e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b" (UID: "e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:15.104259 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:15.104198 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-maas-api-xks-ca\") pod \"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b\" (UID: \"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b\") " Oct 30 20:20:15.104699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:15.104679 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:15.106029 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:15.106002 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:15.106483 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:15.106459 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-kube-api-access-9t47d" pod "e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b" (UID: "e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b"). InnerVolumeSpecName "kube-api-access-9t47d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:15.106719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:15.106695 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-maas-api-xks-ca" pod "e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b" (UID: "e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:20:15.206255 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:15.206174 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t47d\" (UniqueName: \"kubernetes.io/projected/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-kube-api-access-9t47d\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:15.206255 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:15.206204 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:15.861174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:15.857278 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:15.913560 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:15.913529 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" Oct 30 20:20:15.913982 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:15.913522 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj" event={"ID":"e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b","Type":"ContainerDied","Data":"0277a494b69dd1caa6dab2b9e058ef96c9257e6d0c65521abdfdf9e06588493f"} Oct 30 20:20:15.913982 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:15.913665 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0277a494b69dd1caa6dab2b9e058ef96c9257e6d0c65521abdfdf9e06588493f" Oct 30 20:20:15.976634 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:20:15.976594 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:20:16.881244 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:16.881204 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:17.704812 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:17.704705 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:18.700552 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:18.700512 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:19.713186 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:19.712848 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:20.811211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:20.811173 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:21.554018 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:21.553879 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:22.344235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:22.344199 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:22.868373 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:22.868331 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph"] Oct 30 20:20:22.998845 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:22.998795 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:23.095607 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.095575 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl"] Oct 30 20:20:23.096320 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.096299 5091 cpu_manager.go:414] "containerMap: removing container" podUID="e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b" containerName="revoke-keys" Oct 30 20:20:23.096320 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.096320 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b" containerName="revoke-keys" Oct 30 20:20:23.096494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.096454 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b" containerName="revoke-keys" Oct 30 20:20:23.100505 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.100483 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" Oct 30 20:20:23.106005 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.105982 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:20:23.107134 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.107110 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:20:23.113697 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.113675 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl"] Oct 30 20:20:23.142703 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.142627 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzfq2\" (UniqueName: \"kubernetes.io/projected/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-kube-api-access-tzfq2\") pod \"maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl\" (UID: \"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" Oct 30 20:20:23.142953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.142925 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl\" (UID: \"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" Oct 30 20:20:23.143095 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.143070 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl\" (UID: \"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" Oct 30 20:20:23.244699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.244646 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl\" (UID: \"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" Oct 30 20:20:23.244941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.244746 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl\" (UID: \"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" Oct 30 20:20:23.244941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.244841 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzfq2\" (UniqueName: \"kubernetes.io/projected/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-kube-api-access-tzfq2\") pod \"maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl\" (UID: \"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" Oct 30 20:20:23.245696 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.245667 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl\" (UID: \"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" Oct 30 20:20:23.247291 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.247261 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl\" (UID: \"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" Oct 30 20:20:23.258843 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.258813 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzfq2\" (UniqueName: \"kubernetes.io/projected/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-kube-api-access-tzfq2\") pod \"maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl\" (UID: \"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" Oct 30 20:20:23.427371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.427284 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" Oct 30 20:20:23.568813 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.568756 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:23.718939 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:20:23.718896 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64a10ef_b0a0_4fdf_bb90_a05f8fb9655a.slice/crio-3bac0d856a97c78371b8cd4f2064e7a2ec05a69c05a6e158832f69b2b922d378 WatchSource:0}: Error finding container 3bac0d856a97c78371b8cd4f2064e7a2ec05a69c05a6e158832f69b2b922d378: Status 404 returned error can't find the container with id 3bac0d856a97c78371b8cd4f2064e7a2ec05a69c05a6e158832f69b2b922d378 Oct 30 20:20:23.719716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:23.719687 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl"] Oct 30 20:20:24.035228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:24.035189 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" event={"ID":"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a","Type":"ContainerStarted","Data":"3bac0d856a97c78371b8cd4f2064e7a2ec05a69c05a6e158832f69b2b922d378"} Oct 30 20:20:24.359648 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:24.358731 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:25.044695 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:25.044659 5091 generic.go:356] "Generic (PLEG): container finished" podID="c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a" containerID="9cb9882f85b37193b1dbaebdd822335a65cb05b915204fd83edad804077e1059" exitCode=0 Oct 30 20:20:25.045091 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:25.044721 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" event={"ID":"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a","Type":"ContainerDied","Data":"9cb9882f85b37193b1dbaebdd822335a65cb05b915204fd83edad804077e1059"} Oct 30 20:20:25.298395 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:25.298279 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:26.180330 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:26.180294 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:26.275839 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:26.275803 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" Oct 30 20:20:26.401739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:26.401641 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-maas-api-xks-ca\") pod \"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a\" (UID: \"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a\") " Oct 30 20:20:26.401943 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:26.401809 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-kube-api-access-tzfq2\" (UniqueName: \"kubernetes.io/projected/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-kube-api-access-tzfq2\") pod \"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a\" (UID: \"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a\") " Oct 30 20:20:26.401943 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:26.401855 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-maas-api-openshift-ca\") pod \"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a\" (UID: \"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a\") " Oct 30 20:20:26.402283 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:26.402245 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-maas-api-openshift-ca" pod "c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a" (UID: "c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:26.402552 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:26.402521 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:26.403973 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:26.403947 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-maas-api-xks-ca" pod "c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a" (UID: "c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:20:26.404245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:26.404223 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-kube-api-access-tzfq2" pod "c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a" (UID: "c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a"). InnerVolumeSpecName "kube-api-access-tzfq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:26.503629 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:26.503593 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:26.503798 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:26.503632 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzfq2\" (UniqueName: \"kubernetes.io/projected/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a-kube-api-access-tzfq2\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:27.063869 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:27.063714 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" Oct 30 20:20:27.063869 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:27.063734 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl" event={"ID":"c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a","Type":"ContainerDied","Data":"3bac0d856a97c78371b8cd4f2064e7a2ec05a69c05a6e158832f69b2b922d378"} Oct 30 20:20:27.063869 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:27.063793 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bac0d856a97c78371b8cd4f2064e7a2ec05a69c05a6e158832f69b2b922d378" Oct 30 20:20:27.172164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:27.172099 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:28.111714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:28.111679 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:28.937707 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:28.935231 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:29.801188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:29.800176 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:31.102998 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:31.102952 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:31.390615 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:31.390474 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:32.346378 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:32.346341 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:33.029405 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:33.029367 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7"] Oct 30 20:20:33.033108 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:33.033081 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-90f64cb7-cjtb7"] Oct 30 20:20:33.196309 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:33.192173 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:33.388669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:33.388576 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:33.530885 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:33.530841 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:34.032831 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:34.032794 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7"] Oct 30 20:20:34.036445 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:34.036423 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7c2cfa27-lqvt7"] Oct 30 20:20:34.169171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:34.165759 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:34.441688 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:34.441613 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0308bc8c-4d95-4bb8-a116-5359e5df4d52" path="/var/lib/kubelet/pods/0308bc8c-4d95-4bb8-a116-5359e5df4d52/volumes" Oct 30 20:20:34.442068 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:34.442053 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d906bfd5-8723-4c63-9f12-bb8f55a4e7e5" path="/var/lib/kubelet/pods/d906bfd5-8723-4c63-9f12-bb8f55a4e7e5/volumes" Oct 30 20:20:35.044719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:35.044684 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c"] Oct 30 20:20:35.048882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:35.048846 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ab6bf3-qlf7c"] Oct 30 20:20:35.082439 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:35.082401 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:35.879749 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:35.879714 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:36.441813 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:36.441781 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0a7eb30-eb93-41c4-aabf-ffda909ccb39" path="/var/lib/kubelet/pods/d0a7eb30-eb93-41c4-aabf-ffda909ccb39/volumes" Oct 30 20:20:36.773431 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:36.773393 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:37.490199 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:37.477476 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:37.898764 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:37.898699 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" podUID="176a900b-0376-4b44-b443-09f050c96daf" containerName="main" containerID="cri-o://02c5678127773ae6d48174b611fad090ce97b6aca773ae541ef0e36de18bde9d" gracePeriod=45 Oct 30 20:20:38.160326 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.160254 5091 generic.go:356] "Generic (PLEG): container finished" podID="176a900b-0376-4b44-b443-09f050c96daf" containerID="02c5678127773ae6d48174b611fad090ce97b6aca773ae541ef0e36de18bde9d" exitCode=0 Oct 30 20:20:38.160469 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.160323 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" event={"ID":"176a900b-0376-4b44-b443-09f050c96daf","Type":"ContainerDied","Data":"02c5678127773ae6d48174b611fad090ce97b6aca773ae541ef0e36de18bde9d"} Oct 30 20:20:38.160469 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.160369 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" event={"ID":"176a900b-0376-4b44-b443-09f050c96daf","Type":"ContainerDied","Data":"0a765f1dbe2cd9c2d88ed222223ef4b5d8ed7551074e4f78db9a6d666d1b14fa"} Oct 30 20:20:38.160469 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.160382 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a765f1dbe2cd9c2d88ed222223ef4b5d8ed7551074e4f78db9a6d666d1b14fa" Oct 30 20:20:38.190634 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.190608 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:20:38.237009 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.236973 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-tmp-dir\") pod \"176a900b-0376-4b44-b443-09f050c96daf\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " Oct 30 20:20:38.237220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.237043 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-home\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-home\") pod \"176a900b-0376-4b44-b443-09f050c96daf\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " Oct 30 20:20:38.237220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.237075 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/176a900b-0376-4b44-b443-09f050c96daf-tls-certs\" (UniqueName: \"kubernetes.io/secret/176a900b-0376-4b44-b443-09f050c96daf-tls-certs\") pod \"176a900b-0376-4b44-b443-09f050c96daf\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " Oct 30 20:20:38.237220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.237137 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/176a900b-0376-4b44-b443-09f050c96daf-kube-api-access-t6b4k\" (UniqueName: \"kubernetes.io/projected/176a900b-0376-4b44-b443-09f050c96daf-kube-api-access-t6b4k\") pod \"176a900b-0376-4b44-b443-09f050c96daf\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " Oct 30 20:20:38.237413 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.237248 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-dshm\") pod \"176a900b-0376-4b44-b443-09f050c96daf\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " Oct 30 20:20:38.237620 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.237417 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-home" pod "176a900b-0376-4b44-b443-09f050c96daf" (UID: "176a900b-0376-4b44-b443-09f050c96daf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:38.237620 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.237537 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-model-cache\") pod \"176a900b-0376-4b44-b443-09f050c96daf\" (UID: \"176a900b-0376-4b44-b443-09f050c96daf\") " Oct 30 20:20:38.238221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.238196 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-tmp-dir" pod "176a900b-0376-4b44-b443-09f050c96daf" (UID: "176a900b-0376-4b44-b443-09f050c96daf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:38.238394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.238265 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-model-cache" pod "176a900b-0376-4b44-b443-09f050c96daf" (UID: "176a900b-0376-4b44-b443-09f050c96daf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:38.238533 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.238314 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-home\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:38.239686 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.239657 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/176a900b-0376-4b44-b443-09f050c96daf-tls-certs" pod "176a900b-0376-4b44-b443-09f050c96daf" (UID: "176a900b-0376-4b44-b443-09f050c96daf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:20:38.239871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.239851 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/176a900b-0376-4b44-b443-09f050c96daf-kube-api-access-t6b4k" pod "176a900b-0376-4b44-b443-09f050c96daf" (UID: "176a900b-0376-4b44-b443-09f050c96daf"). InnerVolumeSpecName "kube-api-access-t6b4k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:38.239933 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.239909 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-dshm" pod "176a900b-0376-4b44-b443-09f050c96daf" (UID: "176a900b-0376-4b44-b443-09f050c96daf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:20:38.339836 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.339804 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-model-cache\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:38.339836 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.339834 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-tmp-dir\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:38.339836 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.339844 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/176a900b-0376-4b44-b443-09f050c96daf-tls-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:38.340084 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.339855 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6b4k\" (UniqueName: \"kubernetes.io/projected/176a900b-0376-4b44-b443-09f050c96daf-kube-api-access-t6b4k\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:38.340084 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.339864 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/176a900b-0376-4b44-b443-09f050c96daf-dshm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:38.405705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:38.405668 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:39.099806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:39.099751 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" podUID="176a900b-0376-4b44-b443-09f050c96daf" containerName="main" probeResult="failure" output="Get \"https://10.132.1.118:8000/ready\": context deadline exceeded" Oct 30 20:20:39.171041 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:39.171007 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph" Oct 30 20:20:39.200705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:39.200665 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:39.222293 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:39.222248 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph"] Oct 30 20:20:39.230503 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:39.230467 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c257e7/test-model-c257e7-kserve-6c88cc97f7-h9wph"] Oct 30 20:20:40.179452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:40.177643 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:40.442472 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:40.442384 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="176a900b-0376-4b44-b443-09f050c96daf" path="/var/lib/kubelet/pods/176a900b-0376-4b44-b443-09f050c96daf/volumes" Oct 30 20:20:41.041170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:41.032262 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:41.920801 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:41.920748 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:42.581802 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.581764 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6"] Oct 30 20:20:42.582587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.582562 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a" containerName="revoke-keys" Oct 30 20:20:42.582587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.582584 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a" containerName="revoke-keys" Oct 30 20:20:42.582813 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.582603 5091 cpu_manager.go:414] "containerMap: removing container" podUID="176a900b-0376-4b44-b443-09f050c96daf" containerName="main" Oct 30 20:20:42.582813 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.582611 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="176a900b-0376-4b44-b443-09f050c96daf" containerName="main" Oct 30 20:20:42.582813 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.582798 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a" containerName="revoke-keys" Oct 30 20:20:42.582980 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.582814 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="176a900b-0376-4b44-b443-09f050c96daf" containerName="main" Oct 30 20:20:42.588047 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.588020 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" Oct 30 20:20:42.593714 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.593690 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:20:42.594856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.594832 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6"] Oct 30 20:20:42.595820 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.595794 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:20:42.697089 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.697053 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj58m\" (UniqueName: \"kubernetes.io/projected/0855b356-7f3a-41aa-a78a-ab8366b566b3-kube-api-access-cj58m\") pod \"maas-api-revoke-sub-6b105644-dbmr6\" (UID: \"0855b356-7f3a-41aa-a78a-ab8366b566b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" Oct 30 20:20:42.697273 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.697242 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0855b356-7f3a-41aa-a78a-ab8366b566b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b105644-dbmr6\" (UID: \"0855b356-7f3a-41aa-a78a-ab8366b566b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" Oct 30 20:20:42.697385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.697363 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0855b356-7f3a-41aa-a78a-ab8366b566b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b105644-dbmr6\" (UID: \"0855b356-7f3a-41aa-a78a-ab8366b566b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" Oct 30 20:20:42.799211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.799064 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj58m\" (UniqueName: \"kubernetes.io/projected/0855b356-7f3a-41aa-a78a-ab8366b566b3-kube-api-access-cj58m\") pod \"maas-api-revoke-sub-6b105644-dbmr6\" (UID: \"0855b356-7f3a-41aa-a78a-ab8366b566b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" Oct 30 20:20:42.799437 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.799311 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0855b356-7f3a-41aa-a78a-ab8366b566b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b105644-dbmr6\" (UID: \"0855b356-7f3a-41aa-a78a-ab8366b566b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" Oct 30 20:20:42.799437 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.799397 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0855b356-7f3a-41aa-a78a-ab8366b566b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b105644-dbmr6\" (UID: \"0855b356-7f3a-41aa-a78a-ab8366b566b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" Oct 30 20:20:42.803738 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.803707 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0855b356-7f3a-41aa-a78a-ab8366b566b3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b105644-dbmr6\" (UID: \"0855b356-7f3a-41aa-a78a-ab8366b566b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" Oct 30 20:20:42.814891 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.814864 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0855b356-7f3a-41aa-a78a-ab8366b566b3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b105644-dbmr6\" (UID: \"0855b356-7f3a-41aa-a78a-ab8366b566b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" Oct 30 20:20:42.815758 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.815734 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj58m\" (UniqueName: \"kubernetes.io/projected/0855b356-7f3a-41aa-a78a-ab8366b566b3-kube-api-access-cj58m\") pod \"maas-api-revoke-sub-6b105644-dbmr6\" (UID: \"0855b356-7f3a-41aa-a78a-ab8366b566b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" Oct 30 20:20:42.864814 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.864736 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:42.905288 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:42.905257 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" Oct 30 20:20:43.188472 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:20:43.188438 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0855b356_7f3a_41aa_a78a_ab8366b566b3.slice/crio-11341a59636ae0ea4c588ae9c7cbfe1f1e016885654aa7ff156f081197d75c7f WatchSource:0}: Error finding container 11341a59636ae0ea4c588ae9c7cbfe1f1e016885654aa7ff156f081197d75c7f: Status 404 returned error can't find the container with id 11341a59636ae0ea4c588ae9c7cbfe1f1e016885654aa7ff156f081197d75c7f Oct 30 20:20:43.189440 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:43.189411 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6"] Oct 30 20:20:43.234947 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:43.234916 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" event={"ID":"0855b356-7f3a-41aa-a78a-ab8366b566b3","Type":"ContainerStarted","Data":"11341a59636ae0ea4c588ae9c7cbfe1f1e016885654aa7ff156f081197d75c7f"} Oct 30 20:20:43.794832 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:43.794726 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:44.245218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:44.245181 5091 generic.go:356] "Generic (PLEG): container finished" podID="0855b356-7f3a-41aa-a78a-ab8366b566b3" containerID="09bb04abeef7b4404d355fcfc8dcf189c54e16c2d4981a8c9deb4ecca1c4fb7a" exitCode=0 Oct 30 20:20:44.245616 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:44.245254 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" event={"ID":"0855b356-7f3a-41aa-a78a-ab8366b566b3","Type":"ContainerDied","Data":"09bb04abeef7b4404d355fcfc8dcf189c54e16c2d4981a8c9deb4ecca1c4fb7a"} Oct 30 20:20:44.611973 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:44.611918 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:45.427698 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:45.427672 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" Oct 30 20:20:45.508224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:45.498737 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:45.537173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:45.535828 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0855b356-7f3a-41aa-a78a-ab8366b566b3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0855b356-7f3a-41aa-a78a-ab8366b566b3-maas-api-openshift-ca\") pod \"0855b356-7f3a-41aa-a78a-ab8366b566b3\" (UID: \"0855b356-7f3a-41aa-a78a-ab8366b566b3\") " Oct 30 20:20:45.537173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:45.535943 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0855b356-7f3a-41aa-a78a-ab8366b566b3-kube-api-access-cj58m\" (UniqueName: \"kubernetes.io/projected/0855b356-7f3a-41aa-a78a-ab8366b566b3-kube-api-access-cj58m\") pod \"0855b356-7f3a-41aa-a78a-ab8366b566b3\" (UID: \"0855b356-7f3a-41aa-a78a-ab8366b566b3\") " Oct 30 20:20:45.537173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:45.536131 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0855b356-7f3a-41aa-a78a-ab8366b566b3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0855b356-7f3a-41aa-a78a-ab8366b566b3-maas-api-xks-ca\") pod \"0855b356-7f3a-41aa-a78a-ab8366b566b3\" (UID: \"0855b356-7f3a-41aa-a78a-ab8366b566b3\") " Oct 30 20:20:45.537580 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:45.537552 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0855b356-7f3a-41aa-a78a-ab8366b566b3-maas-api-openshift-ca" pod "0855b356-7f3a-41aa-a78a-ab8366b566b3" (UID: "0855b356-7f3a-41aa-a78a-ab8366b566b3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:45.545743 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:45.545713 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0855b356-7f3a-41aa-a78a-ab8366b566b3-maas-api-xks-ca" pod "0855b356-7f3a-41aa-a78a-ab8366b566b3" (UID: "0855b356-7f3a-41aa-a78a-ab8366b566b3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:20:45.550617 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:45.550583 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0855b356-7f3a-41aa-a78a-ab8366b566b3-kube-api-access-cj58m" pod "0855b356-7f3a-41aa-a78a-ab8366b566b3" (UID: "0855b356-7f3a-41aa-a78a-ab8366b566b3"). InnerVolumeSpecName "kube-api-access-cj58m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:45.637916 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:45.637883 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj58m\" (UniqueName: \"kubernetes.io/projected/0855b356-7f3a-41aa-a78a-ab8366b566b3-kube-api-access-cj58m\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:45.637916 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:45.637913 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0855b356-7f3a-41aa-a78a-ab8366b566b3-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:45.638112 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:45.637925 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0855b356-7f3a-41aa-a78a-ab8366b566b3-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:45.979328 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:20:45.979234 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:20:46.232442 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:46.232357 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:46.266895 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:46.266859 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" event={"ID":"0855b356-7f3a-41aa-a78a-ab8366b566b3","Type":"ContainerDied","Data":"11341a59636ae0ea4c588ae9c7cbfe1f1e016885654aa7ff156f081197d75c7f"} Oct 30 20:20:46.266895 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:46.266890 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b105644-dbmr6" Oct 30 20:20:46.267189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:46.266911 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11341a59636ae0ea4c588ae9c7cbfe1f1e016885654aa7ff156f081197d75c7f" Oct 30 20:20:46.459168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:46.459103 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:46.513118 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:46.513082 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl"] Oct 30 20:20:46.522274 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:46.522243 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f63d20-vhwjl"] Oct 30 20:20:46.634106 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:46.634070 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:47.095549 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:47.095512 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:47.694874 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:47.694833 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:48.441983 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:48.441946 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a" path="/var/lib/kubelet/pods/c64a10ef-b0a0-4fdf-bb90-a05f8fb9655a/volumes" Oct 30 20:20:48.661983 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:48.649447 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:49.514307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:49.514267 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:50.327172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:50.327030 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:51.293175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:51.287765 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:52.158471 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:52.158382 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:53.133169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.128072 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:53.354917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.354884 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:53.683630 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.683596 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd"] Oct 30 20:20:53.684293 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.684276 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0855b356-7f3a-41aa-a78a-ab8366b566b3" containerName="revoke-keys" Oct 30 20:20:53.684354 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.684295 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0855b356-7f3a-41aa-a78a-ab8366b566b3" containerName="revoke-keys" Oct 30 20:20:53.684410 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.684401 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0855b356-7f3a-41aa-a78a-ab8366b566b3" containerName="revoke-keys" Oct 30 20:20:53.687844 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.687825 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" Oct 30 20:20:53.702669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.701850 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd"] Oct 30 20:20:53.702669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.701980 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:20:53.702669 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.701881 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:20:53.788578 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.788539 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19fc6e23-7fad-4948-8650-7835db35d161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99d640d0-t9pmd\" (UID: \"19fc6e23-7fad-4948-8650-7835db35d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" Oct 30 20:20:53.788738 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.788675 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q6nk\" (UniqueName: \"kubernetes.io/projected/19fc6e23-7fad-4948-8650-7835db35d161-kube-api-access-5q6nk\") pod \"maas-api-revoke-sub-99d640d0-t9pmd\" (UID: \"19fc6e23-7fad-4948-8650-7835db35d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" Oct 30 20:20:53.788738 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.788720 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19fc6e23-7fad-4948-8650-7835db35d161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99d640d0-t9pmd\" (UID: \"19fc6e23-7fad-4948-8650-7835db35d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" Oct 30 20:20:53.889653 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.889618 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q6nk\" (UniqueName: \"kubernetes.io/projected/19fc6e23-7fad-4948-8650-7835db35d161-kube-api-access-5q6nk\") pod \"maas-api-revoke-sub-99d640d0-t9pmd\" (UID: \"19fc6e23-7fad-4948-8650-7835db35d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" Oct 30 20:20:53.889834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.889670 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19fc6e23-7fad-4948-8650-7835db35d161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99d640d0-t9pmd\" (UID: \"19fc6e23-7fad-4948-8650-7835db35d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" Oct 30 20:20:53.889884 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.889831 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19fc6e23-7fad-4948-8650-7835db35d161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99d640d0-t9pmd\" (UID: \"19fc6e23-7fad-4948-8650-7835db35d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" Oct 30 20:20:53.890395 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.890371 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19fc6e23-7fad-4948-8650-7835db35d161-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-99d640d0-t9pmd\" (UID: \"19fc6e23-7fad-4948-8650-7835db35d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" Oct 30 20:20:53.892281 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.892257 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19fc6e23-7fad-4948-8650-7835db35d161-maas-api-xks-ca\") pod \"maas-api-revoke-sub-99d640d0-t9pmd\" (UID: \"19fc6e23-7fad-4948-8650-7835db35d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" Oct 30 20:20:53.904004 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:53.903971 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q6nk\" (UniqueName: \"kubernetes.io/projected/19fc6e23-7fad-4948-8650-7835db35d161-kube-api-access-5q6nk\") pod \"maas-api-revoke-sub-99d640d0-t9pmd\" (UID: \"19fc6e23-7fad-4948-8650-7835db35d161\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" Oct 30 20:20:54.014785 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:54.014755 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" Oct 30 20:20:54.177291 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:54.177256 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:54.345593 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:20:54.345560 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19fc6e23_7fad_4948_8650_7835db35d161.slice/crio-e44fb2d335d25446a59143b567424beb83858a38aa332056fbf9d3c2214a6ac1 WatchSource:0}: Error finding container e44fb2d335d25446a59143b567424beb83858a38aa332056fbf9d3c2214a6ac1: Status 404 returned error can't find the container with id e44fb2d335d25446a59143b567424beb83858a38aa332056fbf9d3c2214a6ac1 Oct 30 20:20:54.346122 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:54.346096 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd"] Oct 30 20:20:54.460580 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:54.460547 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" event={"ID":"19fc6e23-7fad-4948-8650-7835db35d161","Type":"ContainerStarted","Data":"e44fb2d335d25446a59143b567424beb83858a38aa332056fbf9d3c2214a6ac1"} Oct 30 20:20:54.875184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:54.875119 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:55.055782 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:55.055742 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:55.205559 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:55.205524 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:55.363166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:55.362932 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:55.470461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:55.470423 5091 generic.go:356] "Generic (PLEG): container finished" podID="19fc6e23-7fad-4948-8650-7835db35d161" containerID="31a84549eecd68774611c1203274d224a908008a24e9ef98c5afab0402c5c48c" exitCode=0 Oct 30 20:20:55.470655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:55.470521 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" event={"ID":"19fc6e23-7fad-4948-8650-7835db35d161","Type":"ContainerDied","Data":"31a84549eecd68774611c1203274d224a908008a24e9ef98c5afab0402c5c48c"} Oct 30 20:20:55.763061 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:55.763024 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj"] Oct 30 20:20:55.767545 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:55.767521 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" Oct 30 20:20:55.784110 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:55.784069 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj"] Oct 30 20:20:55.923672 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:55.923641 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9149aa30-03f0-4385-bbd8-28a4e00c40e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj\" (UID: \"9149aa30-03f0-4385-bbd8-28a4e00c40e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" Oct 30 20:20:55.923854 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:55.923800 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97qf9\" (UniqueName: \"kubernetes.io/projected/9149aa30-03f0-4385-bbd8-28a4e00c40e8-kube-api-access-97qf9\") pod \"maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj\" (UID: \"9149aa30-03f0-4385-bbd8-28a4e00c40e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" Oct 30 20:20:55.923916 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:55.923902 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9149aa30-03f0-4385-bbd8-28a4e00c40e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj\" (UID: \"9149aa30-03f0-4385-bbd8-28a4e00c40e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" Oct 30 20:20:56.025509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.025424 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9149aa30-03f0-4385-bbd8-28a4e00c40e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj\" (UID: \"9149aa30-03f0-4385-bbd8-28a4e00c40e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" Oct 30 20:20:56.025509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.025469 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9149aa30-03f0-4385-bbd8-28a4e00c40e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj\" (UID: \"9149aa30-03f0-4385-bbd8-28a4e00c40e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" Oct 30 20:20:56.025737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.025629 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97qf9\" (UniqueName: \"kubernetes.io/projected/9149aa30-03f0-4385-bbd8-28a4e00c40e8-kube-api-access-97qf9\") pod \"maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj\" (UID: \"9149aa30-03f0-4385-bbd8-28a4e00c40e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" Oct 30 20:20:56.026093 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.026070 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9149aa30-03f0-4385-bbd8-28a4e00c40e8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj\" (UID: \"9149aa30-03f0-4385-bbd8-28a4e00c40e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" Oct 30 20:20:56.027946 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.027922 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9149aa30-03f0-4385-bbd8-28a4e00c40e8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj\" (UID: \"9149aa30-03f0-4385-bbd8-28a4e00c40e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" Oct 30 20:20:56.040690 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.040669 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97qf9\" (UniqueName: \"kubernetes.io/projected/9149aa30-03f0-4385-bbd8-28a4e00c40e8-kube-api-access-97qf9\") pod \"maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj\" (UID: \"9149aa30-03f0-4385-bbd8-28a4e00c40e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" Oct 30 20:20:56.083725 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.083701 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" Oct 30 20:20:56.263227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.263139 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:56.387822 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.387787 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj"] Oct 30 20:20:56.388698 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:20:56.388664 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9149aa30_03f0_4385_bbd8_28a4e00c40e8.slice/crio-af186e744df8b7a323f0eab97c5bb624b39f43ea3f6486a1af2acbe2c8c068fe WatchSource:0}: Error finding container af186e744df8b7a323f0eab97c5bb624b39f43ea3f6486a1af2acbe2c8c068fe: Status 404 returned error can't find the container with id af186e744df8b7a323f0eab97c5bb624b39f43ea3f6486a1af2acbe2c8c068fe Oct 30 20:20:56.483900 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.483862 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" event={"ID":"9149aa30-03f0-4385-bbd8-28a4e00c40e8","Type":"ContainerStarted","Data":"af186e744df8b7a323f0eab97c5bb624b39f43ea3f6486a1af2acbe2c8c068fe"} Oct 30 20:20:56.709794 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.709764 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" Oct 30 20:20:56.839169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.839095 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19fc6e23-7fad-4948-8650-7835db35d161-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19fc6e23-7fad-4948-8650-7835db35d161-maas-api-xks-ca\") pod \"19fc6e23-7fad-4948-8650-7835db35d161\" (UID: \"19fc6e23-7fad-4948-8650-7835db35d161\") " Oct 30 20:20:56.839413 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.839391 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19fc6e23-7fad-4948-8650-7835db35d161-kube-api-access-5q6nk\" (UniqueName: \"kubernetes.io/projected/19fc6e23-7fad-4948-8650-7835db35d161-kube-api-access-5q6nk\") pod \"19fc6e23-7fad-4948-8650-7835db35d161\" (UID: \"19fc6e23-7fad-4948-8650-7835db35d161\") " Oct 30 20:20:56.839570 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.839456 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19fc6e23-7fad-4948-8650-7835db35d161-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19fc6e23-7fad-4948-8650-7835db35d161-maas-api-openshift-ca\") pod \"19fc6e23-7fad-4948-8650-7835db35d161\" (UID: \"19fc6e23-7fad-4948-8650-7835db35d161\") " Oct 30 20:20:56.839955 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.839928 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19fc6e23-7fad-4948-8650-7835db35d161-maas-api-openshift-ca" pod "19fc6e23-7fad-4948-8650-7835db35d161" (UID: "19fc6e23-7fad-4948-8650-7835db35d161"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:56.841376 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.841351 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19fc6e23-7fad-4948-8650-7835db35d161-maas-api-xks-ca" pod "19fc6e23-7fad-4948-8650-7835db35d161" (UID: "19fc6e23-7fad-4948-8650-7835db35d161"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:20:56.841486 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.841466 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19fc6e23-7fad-4948-8650-7835db35d161-kube-api-access-5q6nk" pod "19fc6e23-7fad-4948-8650-7835db35d161" (UID: "19fc6e23-7fad-4948-8650-7835db35d161"). InnerVolumeSpecName "kube-api-access-5q6nk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:56.941004 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.940919 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q6nk\" (UniqueName: \"kubernetes.io/projected/19fc6e23-7fad-4948-8650-7835db35d161-kube-api-access-5q6nk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:56.941004 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.940950 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19fc6e23-7fad-4948-8650-7835db35d161-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:56.941004 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:56.940960 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19fc6e23-7fad-4948-8650-7835db35d161-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:57.208815 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:57.205283 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:57.495490 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:57.495404 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" Oct 30 20:20:57.495490 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:57.495433 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-99d640d0-t9pmd" event={"ID":"19fc6e23-7fad-4948-8650-7835db35d161","Type":"ContainerDied","Data":"e44fb2d335d25446a59143b567424beb83858a38aa332056fbf9d3c2214a6ac1"} Oct 30 20:20:57.495490 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:57.495484 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e44fb2d335d25446a59143b567424beb83858a38aa332056fbf9d3c2214a6ac1" Oct 30 20:20:57.497367 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:57.497337 5091 generic.go:356] "Generic (PLEG): container finished" podID="9149aa30-03f0-4385-bbd8-28a4e00c40e8" containerID="1bd4c2709a8399192f084401427353b10dfabe80589a727737d3c18ac5cffe04" exitCode=0 Oct 30 20:20:57.497534 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:57.497398 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" event={"ID":"9149aa30-03f0-4385-bbd8-28a4e00c40e8","Type":"ContainerDied","Data":"1bd4c2709a8399192f084401427353b10dfabe80589a727737d3c18ac5cffe04"} Oct 30 20:20:58.187193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:58.187139 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:58.670792 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:58.670755 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" Oct 30 20:20:58.765923 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:58.765826 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9149aa30-03f0-4385-bbd8-28a4e00c40e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9149aa30-03f0-4385-bbd8-28a4e00c40e8-maas-api-openshift-ca\") pod \"9149aa30-03f0-4385-bbd8-28a4e00c40e8\" (UID: \"9149aa30-03f0-4385-bbd8-28a4e00c40e8\") " Oct 30 20:20:58.766094 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:58.766075 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9149aa30-03f0-4385-bbd8-28a4e00c40e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9149aa30-03f0-4385-bbd8-28a4e00c40e8-maas-api-xks-ca\") pod \"9149aa30-03f0-4385-bbd8-28a4e00c40e8\" (UID: \"9149aa30-03f0-4385-bbd8-28a4e00c40e8\") " Oct 30 20:20:58.766182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:58.766111 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9149aa30-03f0-4385-bbd8-28a4e00c40e8-kube-api-access-97qf9\" (UniqueName: \"kubernetes.io/projected/9149aa30-03f0-4385-bbd8-28a4e00c40e8-kube-api-access-97qf9\") pod \"9149aa30-03f0-4385-bbd8-28a4e00c40e8\" (UID: \"9149aa30-03f0-4385-bbd8-28a4e00c40e8\") " Oct 30 20:20:58.766263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:58.766238 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9149aa30-03f0-4385-bbd8-28a4e00c40e8-maas-api-openshift-ca" pod "9149aa30-03f0-4385-bbd8-28a4e00c40e8" (UID: "9149aa30-03f0-4385-bbd8-28a4e00c40e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:20:58.766651 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:58.766629 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9149aa30-03f0-4385-bbd8-28a4e00c40e8-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:58.768455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:58.768427 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9149aa30-03f0-4385-bbd8-28a4e00c40e8-kube-api-access-97qf9" pod "9149aa30-03f0-4385-bbd8-28a4e00c40e8" (UID: "9149aa30-03f0-4385-bbd8-28a4e00c40e8"). InnerVolumeSpecName "kube-api-access-97qf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:20:58.768556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:58.768497 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9149aa30-03f0-4385-bbd8-28a4e00c40e8-maas-api-xks-ca" pod "9149aa30-03f0-4385-bbd8-28a4e00c40e8" (UID: "9149aa30-03f0-4385-bbd8-28a4e00c40e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:20:58.867879 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:58.867848 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9149aa30-03f0-4385-bbd8-28a4e00c40e8-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:58.868005 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:58.867884 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97qf9\" (UniqueName: \"kubernetes.io/projected/9149aa30-03f0-4385-bbd8-28a4e00c40e8-kube-api-access-97qf9\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:20:59.081224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:59.071757 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:20:59.516157 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:59.516112 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" Oct 30 20:20:59.516157 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:59.516137 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj" event={"ID":"9149aa30-03f0-4385-bbd8-28a4e00c40e8","Type":"ContainerDied","Data":"af186e744df8b7a323f0eab97c5bb624b39f43ea3f6486a1af2acbe2c8c068fe"} Oct 30 20:20:59.516546 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:59.516205 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af186e744df8b7a323f0eab97c5bb624b39f43ea3f6486a1af2acbe2c8c068fe" Oct 30 20:20:59.956227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:20:59.955598 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:00.766175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:00.763122 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:01.834421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:01.830829 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:02.782167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:02.778267 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:03.573167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:03.572200 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:04.413225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.411056 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:04.561197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.560892 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:04.727301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.727216 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6"] Oct 30 20:21:04.728135 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.728110 5091 cpu_manager.go:414] "containerMap: removing container" podUID="19fc6e23-7fad-4948-8650-7835db35d161" containerName="revoke-keys" Oct 30 20:21:04.728135 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.728137 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19fc6e23-7fad-4948-8650-7835db35d161" containerName="revoke-keys" Oct 30 20:21:04.728339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.728183 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9149aa30-03f0-4385-bbd8-28a4e00c40e8" containerName="revoke-keys" Oct 30 20:21:04.728339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.728192 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9149aa30-03f0-4385-bbd8-28a4e00c40e8" containerName="revoke-keys" Oct 30 20:21:04.728455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.728360 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="19fc6e23-7fad-4948-8650-7835db35d161" containerName="revoke-keys" Oct 30 20:21:04.728455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.728378 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9149aa30-03f0-4385-bbd8-28a4e00c40e8" containerName="revoke-keys" Oct 30 20:21:04.732307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.732286 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" Oct 30 20:21:04.737618 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.737596 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:21:04.739541 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.739521 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:21:04.744053 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.744025 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6"] Oct 30 20:21:04.778769 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.778720 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtjcp\" (UniqueName: \"kubernetes.io/projected/46afd14f-e8da-4be4-9bf7-9c8bac157021-kube-api-access-dtjcp\") pod \"maas-api-revoke-keys-e2e-inf-a-89c124-cdng6\" (UID: \"46afd14f-e8da-4be4-9bf7-9c8bac157021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" Oct 30 20:21:04.778953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.778860 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46afd14f-e8da-4be4-9bf7-9c8bac157021-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-89c124-cdng6\" (UID: \"46afd14f-e8da-4be4-9bf7-9c8bac157021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" Oct 30 20:21:04.779015 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.778970 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46afd14f-e8da-4be4-9bf7-9c8bac157021-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-89c124-cdng6\" (UID: \"46afd14f-e8da-4be4-9bf7-9c8bac157021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" Oct 30 20:21:04.880587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.880535 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtjcp\" (UniqueName: \"kubernetes.io/projected/46afd14f-e8da-4be4-9bf7-9c8bac157021-kube-api-access-dtjcp\") pod \"maas-api-revoke-keys-e2e-inf-a-89c124-cdng6\" (UID: \"46afd14f-e8da-4be4-9bf7-9c8bac157021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" Oct 30 20:21:04.880816 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.880681 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46afd14f-e8da-4be4-9bf7-9c8bac157021-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-89c124-cdng6\" (UID: \"46afd14f-e8da-4be4-9bf7-9c8bac157021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" Oct 30 20:21:04.880816 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.880791 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46afd14f-e8da-4be4-9bf7-9c8bac157021-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-89c124-cdng6\" (UID: \"46afd14f-e8da-4be4-9bf7-9c8bac157021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" Oct 30 20:21:04.881712 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.881683 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46afd14f-e8da-4be4-9bf7-9c8bac157021-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-89c124-cdng6\" (UID: \"46afd14f-e8da-4be4-9bf7-9c8bac157021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" Oct 30 20:21:04.883983 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.883956 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46afd14f-e8da-4be4-9bf7-9c8bac157021-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-89c124-cdng6\" (UID: \"46afd14f-e8da-4be4-9bf7-9c8bac157021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" Oct 30 20:21:04.896755 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:04.896728 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtjcp\" (UniqueName: \"kubernetes.io/projected/46afd14f-e8da-4be4-9bf7-9c8bac157021-kube-api-access-dtjcp\") pod \"maas-api-revoke-keys-e2e-inf-a-89c124-cdng6\" (UID: \"46afd14f-e8da-4be4-9bf7-9c8bac157021\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" Oct 30 20:21:05.052195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:05.052118 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" Oct 30 20:21:05.349603 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:05.349567 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6"] Oct 30 20:21:05.350777 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:21:05.350745 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46afd14f_e8da_4be4_9bf7_9c8bac157021.slice/crio-cb7facf8385850631a1ef7874bf6cb1a30fdbb6e20065ee2b597dd8f87c191c1 WatchSource:0}: Error finding container cb7facf8385850631a1ef7874bf6cb1a30fdbb6e20065ee2b597dd8f87c191c1: Status 404 returned error can't find the container with id cb7facf8385850631a1ef7874bf6cb1a30fdbb6e20065ee2b597dd8f87c191c1 Oct 30 20:21:05.510169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:05.509793 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:05.664717 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:05.664690 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" event={"ID":"46afd14f-e8da-4be4-9bf7-9c8bac157021","Type":"ContainerStarted","Data":"cb7facf8385850631a1ef7874bf6cb1a30fdbb6e20065ee2b597dd8f87c191c1"} Oct 30 20:21:05.921918 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:05.921831 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6"] Oct 30 20:21:06.025212 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:06.025161 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:06.208165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:06.208108 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:06.469465 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:06.469383 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:06.675826 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:06.675788 5091 generic.go:356] "Generic (PLEG): container finished" podID="46afd14f-e8da-4be4-9bf7-9c8bac157021" containerID="3c26e45599c5276732ea2cd222c385687acc4b6dcf77218056af171baeee0c8d" exitCode=0 Oct 30 20:21:06.676028 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:06.675878 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" event={"ID":"46afd14f-e8da-4be4-9bf7-9c8bac157021","Type":"ContainerDied","Data":"3c26e45599c5276732ea2cd222c385687acc4b6dcf77218056af171baeee0c8d"} Oct 30 20:21:07.381183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:07.380846 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:07.858919 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:07.858891 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" Oct 30 20:21:07.942811 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:07.942730 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46afd14f-e8da-4be4-9bf7-9c8bac157021-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46afd14f-e8da-4be4-9bf7-9c8bac157021-maas-api-openshift-ca\") pod \"46afd14f-e8da-4be4-9bf7-9c8bac157021\" (UID: \"46afd14f-e8da-4be4-9bf7-9c8bac157021\") " Oct 30 20:21:07.942990 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:07.942809 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46afd14f-e8da-4be4-9bf7-9c8bac157021-kube-api-access-dtjcp\" (UniqueName: \"kubernetes.io/projected/46afd14f-e8da-4be4-9bf7-9c8bac157021-kube-api-access-dtjcp\") pod \"46afd14f-e8da-4be4-9bf7-9c8bac157021\" (UID: \"46afd14f-e8da-4be4-9bf7-9c8bac157021\") " Oct 30 20:21:07.943062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:07.942986 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46afd14f-e8da-4be4-9bf7-9c8bac157021-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46afd14f-e8da-4be4-9bf7-9c8bac157021-maas-api-xks-ca\") pod \"46afd14f-e8da-4be4-9bf7-9c8bac157021\" (UID: \"46afd14f-e8da-4be4-9bf7-9c8bac157021\") " Oct 30 20:21:07.943126 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:07.943086 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46afd14f-e8da-4be4-9bf7-9c8bac157021-maas-api-openshift-ca" pod "46afd14f-e8da-4be4-9bf7-9c8bac157021" (UID: "46afd14f-e8da-4be4-9bf7-9c8bac157021"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:07.943552 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:07.943528 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46afd14f-e8da-4be4-9bf7-9c8bac157021-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:07.944937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:07.944913 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46afd14f-e8da-4be4-9bf7-9c8bac157021-kube-api-access-dtjcp" pod "46afd14f-e8da-4be4-9bf7-9c8bac157021" (UID: "46afd14f-e8da-4be4-9bf7-9c8bac157021"). InnerVolumeSpecName "kube-api-access-dtjcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:07.945061 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:07.945024 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46afd14f-e8da-4be4-9bf7-9c8bac157021-maas-api-xks-ca" pod "46afd14f-e8da-4be4-9bf7-9c8bac157021" (UID: "46afd14f-e8da-4be4-9bf7-9c8bac157021"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:21:08.045302 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:08.045266 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtjcp\" (UniqueName: \"kubernetes.io/projected/46afd14f-e8da-4be4-9bf7-9c8bac157021-kube-api-access-dtjcp\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:08.045302 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:08.045299 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46afd14f-e8da-4be4-9bf7-9c8bac157021-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:08.321217 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:08.321176 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:08.693875 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:08.693787 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" event={"ID":"46afd14f-e8da-4be4-9bf7-9c8bac157021","Type":"ContainerDied","Data":"cb7facf8385850631a1ef7874bf6cb1a30fdbb6e20065ee2b597dd8f87c191c1"} Oct 30 20:21:08.693875 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:08.693812 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6" Oct 30 20:21:08.693875 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:08.693843 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb7facf8385850631a1ef7874bf6cb1a30fdbb6e20065ee2b597dd8f87c191c1" Oct 30 20:21:09.186113 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:09.186048 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:10.099174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:10.098230 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:10.922617 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:10.922575 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" podUID="5776e63e-a778-4589-9e2a-10548456ff14" containerName="payload-pre-processing" containerID="cri-o://e38c2e5f865210eb2106a39e84b987795b89e2eabb1499bd2ed5ba9c3cd28595" gracePeriod=25 Oct 30 20:21:11.010521 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:11.010480 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:11.317939 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:11.317900 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" podUID="5776e63e-a778-4589-9e2a-10548456ff14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:21:11.985171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:11.983690 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:12.875873 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:12.875832 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:13.769311 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:13.769269 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:14.662129 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:14.662087 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:14.847315 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:14.847279 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw"] Oct 30 20:21:14.863347 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:14.863314 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl"] Oct 30 20:21:14.973552 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:14.973457 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:15.157169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:15.156850 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:15.301640 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:15.301592 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:15.716981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:15.716899 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:15.978016 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:21:15.977933 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:21:16.371158 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:16.371109 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:16.597890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:16.597815 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:16.631903 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:16.631817 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj"] Oct 30 20:21:16.645578 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:16.645539 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c257e7-tcqxj"] Oct 30 20:21:16.759835 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:16.759797 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:17.238292 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:17.238257 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:17.739799 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:17.739762 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:18.442618 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:18.442581 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9149aa30-03f0-4385-bbd8-28a4e00c40e8" path="/var/lib/kubelet/pods/9149aa30-03f0-4385-bbd8-28a4e00c40e8/volumes" Oct 30 20:21:18.590451 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:18.590411 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:19.504683 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:19.504644 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:19.848048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:19.847956 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" podUID="506a22bb-16c2-4cc8-bbfe-6a9586f1d434" containerName="payload-processing" containerID="cri-o://337da125b3fb23bcd6cd8744e260f800e97a9d3e59b8df408ec5a4ca2436d2d8" gracePeriod=25 Oct 30 20:21:19.863897 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:19.863855 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" podUID="85571ef8-1def-4ecc-af2a-39168d653b2e" containerName="payload-pre-processing" containerID="cri-o://0239af929a749312bd3456b8c545afbe6f79f017faa29ab5a7a5fed9791d234a" gracePeriod=25 Oct 30 20:21:20.130362 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.130331 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" Oct 30 20:21:20.166590 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.166548 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/506a22bb-16c2-4cc8-bbfe-6a9586f1d434-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/506a22bb-16c2-4cc8-bbfe-6a9586f1d434-plugins-config-volume\") pod \"506a22bb-16c2-4cc8-bbfe-6a9586f1d434\" (UID: \"506a22bb-16c2-4cc8-bbfe-6a9586f1d434\") " Oct 30 20:21:20.166855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.166832 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506a22bb-16c2-4cc8-bbfe-6a9586f1d434-kube-api-access-6v47q\" (UniqueName: \"kubernetes.io/projected/506a22bb-16c2-4cc8-bbfe-6a9586f1d434-kube-api-access-6v47q\") pod \"506a22bb-16c2-4cc8-bbfe-6a9586f1d434\" (UID: \"506a22bb-16c2-4cc8-bbfe-6a9586f1d434\") " Oct 30 20:21:20.167070 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.167009 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506a22bb-16c2-4cc8-bbfe-6a9586f1d434-plugins-config-volume" pod "506a22bb-16c2-4cc8-bbfe-6a9586f1d434" (UID: "506a22bb-16c2-4cc8-bbfe-6a9586f1d434"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:20.167462 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.167437 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/506a22bb-16c2-4cc8-bbfe-6a9586f1d434-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:20.169467 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.169435 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506a22bb-16c2-4cc8-bbfe-6a9586f1d434-kube-api-access-6v47q" pod "506a22bb-16c2-4cc8-bbfe-6a9586f1d434" (UID: "506a22bb-16c2-4cc8-bbfe-6a9586f1d434"). InnerVolumeSpecName "kube-api-access-6v47q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:20.274682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.269017 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:20.276998 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.276398 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v47q\" (UniqueName: \"kubernetes.io/projected/506a22bb-16c2-4cc8-bbfe-6a9586f1d434-kube-api-access-6v47q\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:20.798836 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.798802 5091 generic.go:356] "Generic (PLEG): container finished" podID="506a22bb-16c2-4cc8-bbfe-6a9586f1d434" containerID="337da125b3fb23bcd6cd8744e260f800e97a9d3e59b8df408ec5a4ca2436d2d8" exitCode=0 Oct 30 20:21:20.799263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.798857 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" Oct 30 20:21:20.799263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.798891 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" event={"ID":"506a22bb-16c2-4cc8-bbfe-6a9586f1d434","Type":"ContainerDied","Data":"337da125b3fb23bcd6cd8744e260f800e97a9d3e59b8df408ec5a4ca2436d2d8"} Oct 30 20:21:20.799263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.798951 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw" event={"ID":"506a22bb-16c2-4cc8-bbfe-6a9586f1d434","Type":"ContainerDied","Data":"8ff4ba3308de656c30fe8789c51f4cfc3cf14818f51349822eae6281a1ec156b"} Oct 30 20:21:20.799263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.798979 5091 scope.go:122] "RemoveContainer" containerID="337da125b3fb23bcd6cd8744e260f800e97a9d3e59b8df408ec5a4ca2436d2d8" Oct 30 20:21:20.810246 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.810226 5091 scope.go:122] "RemoveContainer" containerID="337da125b3fb23bcd6cd8744e260f800e97a9d3e59b8df408ec5a4ca2436d2d8" Oct 30 20:21:20.810516 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:21:20.810496 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"337da125b3fb23bcd6cd8744e260f800e97a9d3e59b8df408ec5a4ca2436d2d8\": container with ID starting with 337da125b3fb23bcd6cd8744e260f800e97a9d3e59b8df408ec5a4ca2436d2d8 not found: ID does not exist" containerID="337da125b3fb23bcd6cd8744e260f800e97a9d3e59b8df408ec5a4ca2436d2d8" Oct 30 20:21:20.810577 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.810524 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"337da125b3fb23bcd6cd8744e260f800e97a9d3e59b8df408ec5a4ca2436d2d8"} err="failed to get container status \"337da125b3fb23bcd6cd8744e260f800e97a9d3e59b8df408ec5a4ca2436d2d8\": rpc error: code = NotFound desc = could not find container \"337da125b3fb23bcd6cd8744e260f800e97a9d3e59b8df408ec5a4ca2436d2d8\": container with ID starting with 337da125b3fb23bcd6cd8744e260f800e97a9d3e59b8df408ec5a4ca2436d2d8 not found: ID does not exist" Oct 30 20:21:20.823724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.823699 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw"] Oct 30 20:21:20.829503 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:20.829480 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-89c124-5694d4b8c4-dlbnw"] Oct 30 20:21:21.095110 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:21.095023 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:21.317688 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:21.317646 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" podUID="5776e63e-a778-4589-9e2a-10548456ff14" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:21:21.804501 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:21.804469 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 20:21:21.806036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:21.806013 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 20:21:21.904168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:21.898925 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:22.442126 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:22.442088 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="506a22bb-16c2-4cc8-bbfe-6a9586f1d434" path="/var/lib/kubelet/pods/506a22bb-16c2-4cc8-bbfe-6a9586f1d434/volumes" Oct 30 20:21:22.781831 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:22.781788 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:23.604181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:23.604111 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:24.647161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:24.647102 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:24.798597 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:24.798559 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:25.063169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.062761 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:25.476894 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.476764 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:25.480434 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.480407 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4"] Oct 30 20:21:25.481030 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.481013 5091 cpu_manager.go:414] "containerMap: removing container" podUID="506a22bb-16c2-4cc8-bbfe-6a9586f1d434" containerName="payload-processing" Oct 30 20:21:25.481030 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.481029 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="506a22bb-16c2-4cc8-bbfe-6a9586f1d434" containerName="payload-processing" Oct 30 20:21:25.481218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.481073 5091 cpu_manager.go:414] "containerMap: removing container" podUID="46afd14f-e8da-4be4-9bf7-9c8bac157021" containerName="revoke-keys" Oct 30 20:21:25.481218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.481079 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46afd14f-e8da-4be4-9bf7-9c8bac157021" containerName="revoke-keys" Oct 30 20:21:25.481333 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.481229 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="506a22bb-16c2-4cc8-bbfe-6a9586f1d434" containerName="payload-processing" Oct 30 20:21:25.481333 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.481245 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="46afd14f-e8da-4be4-9bf7-9c8bac157021" containerName="revoke-keys" Oct 30 20:21:25.486321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.486302 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" Oct 30 20:21:25.494223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.494085 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:21:25.494223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.494170 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:21:25.498935 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.497983 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4"] Oct 30 20:21:25.579017 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.578977 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g5b4\" (UniqueName: \"kubernetes.io/projected/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-kube-api-access-5g5b4\") pod \"maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4\" (UID: \"5d28c3c7-c79b-4312-b5a1-7df1abef3a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" Oct 30 20:21:25.579241 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.579213 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4\" (UID: \"5d28c3c7-c79b-4312-b5a1-7df1abef3a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" Oct 30 20:21:25.579408 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.579385 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4\" (UID: \"5d28c3c7-c79b-4312-b5a1-7df1abef3a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" Oct 30 20:21:25.685474 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.685415 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4\" (UID: \"5d28c3c7-c79b-4312-b5a1-7df1abef3a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" Oct 30 20:21:25.687077 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.686947 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4\" (UID: \"5d28c3c7-c79b-4312-b5a1-7df1abef3a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" Oct 30 20:21:25.687450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.687427 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g5b4\" (UniqueName: \"kubernetes.io/projected/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-kube-api-access-5g5b4\") pod \"maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4\" (UID: \"5d28c3c7-c79b-4312-b5a1-7df1abef3a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" Oct 30 20:21:25.688196 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.688174 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4\" (UID: \"5d28c3c7-c79b-4312-b5a1-7df1abef3a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" Oct 30 20:21:25.695947 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.695923 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4\" (UID: \"5d28c3c7-c79b-4312-b5a1-7df1abef3a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" Oct 30 20:21:25.709354 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.707688 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g5b4\" (UniqueName: \"kubernetes.io/projected/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-kube-api-access-5g5b4\") pod \"maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4\" (UID: \"5d28c3c7-c79b-4312-b5a1-7df1abef3a23\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" Oct 30 20:21:25.724505 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.724471 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:25.805483 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.805449 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6"] Oct 30 20:21:25.811927 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.811889 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-89c124-cdng6"] Oct 30 20:21:25.812080 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.812032 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" Oct 30 20:21:25.920169 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:25.904383 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:26.103243 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:21:26.103211 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d28c3c7_c79b_4312_b5a1_7df1abef3a23.slice/crio-e6a8d19294d776e1daaa4ddb941d7ee9509976195564eed848bc774d033b43b0 WatchSource:0}: Error finding container e6a8d19294d776e1daaa4ddb941d7ee9509976195564eed848bc774d033b43b0: Status 404 returned error can't find the container with id e6a8d19294d776e1daaa4ddb941d7ee9509976195564eed848bc774d033b43b0 Oct 30 20:21:26.105862 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:26.105835 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4"] Oct 30 20:21:26.152272 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:26.152243 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:26.408821 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:26.408783 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:26.443263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:26.443235 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46afd14f-e8da-4be4-9bf7-9c8bac157021" path="/var/lib/kubelet/pods/46afd14f-e8da-4be4-9bf7-9c8bac157021/volumes" Oct 30 20:21:26.857206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:26.857131 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" event={"ID":"5d28c3c7-c79b-4312-b5a1-7df1abef3a23","Type":"ContainerStarted","Data":"512ae3041ec519de1b1b44888773c7b6215ca6d683776f8d0aa02cf8fb05484f"} Oct 30 20:21:26.857681 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:26.857234 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" event={"ID":"5d28c3c7-c79b-4312-b5a1-7df1abef3a23","Type":"ContainerStarted","Data":"e6a8d19294d776e1daaa4ddb941d7ee9509976195564eed848bc774d033b43b0"} Oct 30 20:21:26.883928 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:26.883867 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" podStartSLOduration=1.8838480880000001 podStartE2EDuration="1.883848088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:26.879876763 +0000 UTC m=+3006.575084724" watchObservedRunningTime="2026-10-30 20:21:26.883848088 +0000 UTC m=+3006.579056048" Oct 30 20:21:26.959168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:26.958119 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:27.040226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:27.040193 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn"] Oct 30 20:21:27.043770 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:27.043746 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8d9e4d9d-zdtjn"] Oct 30 20:21:27.066561 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:27.066537 5091 scope.go:122] "RemoveContainer" containerID="9fe10b78c457562ffefcbbeb83cfe8024009c8a7eedc436ed4b5d3bae78df5b8" Oct 30 20:21:27.788173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:27.785066 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:27.869688 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:27.869651 5091 generic.go:356] "Generic (PLEG): container finished" podID="5d28c3c7-c79b-4312-b5a1-7df1abef3a23" containerID="512ae3041ec519de1b1b44888773c7b6215ca6d683776f8d0aa02cf8fb05484f" exitCode=0 Oct 30 20:21:27.870088 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:27.869726 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" event={"ID":"5d28c3c7-c79b-4312-b5a1-7df1abef3a23","Type":"ContainerDied","Data":"512ae3041ec519de1b1b44888773c7b6215ca6d683776f8d0aa02cf8fb05484f"} Oct 30 20:21:28.442937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:28.442900 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f8bc28a-95aa-4ceb-97f9-91a2f881d50d" path="/var/lib/kubelet/pods/2f8bc28a-95aa-4ceb-97f9-91a2f881d50d/volumes" Oct 30 20:21:28.685262 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:28.685216 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:28.765743 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:28.765699 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" podUID="85571ef8-1def-4ecc-af2a-39168d653b2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:21:29.050189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:29.050136 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" Oct 30 20:21:29.145300 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:29.145266 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-maas-api-openshift-ca\") pod \"5d28c3c7-c79b-4312-b5a1-7df1abef3a23\" (UID: \"5d28c3c7-c79b-4312-b5a1-7df1abef3a23\") " Oct 30 20:21:29.145593 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:29.145574 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-maas-api-xks-ca\") pod \"5d28c3c7-c79b-4312-b5a1-7df1abef3a23\" (UID: \"5d28c3c7-c79b-4312-b5a1-7df1abef3a23\") " Oct 30 20:21:29.145668 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:29.145584 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-maas-api-openshift-ca" pod "5d28c3c7-c79b-4312-b5a1-7df1abef3a23" (UID: "5d28c3c7-c79b-4312-b5a1-7df1abef3a23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:29.145668 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:29.145614 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-kube-api-access-5g5b4\" (UniqueName: \"kubernetes.io/projected/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-kube-api-access-5g5b4\") pod \"5d28c3c7-c79b-4312-b5a1-7df1abef3a23\" (UID: \"5d28c3c7-c79b-4312-b5a1-7df1abef3a23\") " Oct 30 20:21:29.146229 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:29.146204 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:29.147862 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:29.147752 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-maas-api-xks-ca" pod "5d28c3c7-c79b-4312-b5a1-7df1abef3a23" (UID: "5d28c3c7-c79b-4312-b5a1-7df1abef3a23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:21:29.148027 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:29.148005 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-kube-api-access-5g5b4" pod "5d28c3c7-c79b-4312-b5a1-7df1abef3a23" (UID: "5d28c3c7-c79b-4312-b5a1-7df1abef3a23"). InnerVolumeSpecName "kube-api-access-5g5b4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:29.247088 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:29.247056 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:29.247088 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:29.247082 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g5b4\" (UniqueName: \"kubernetes.io/projected/5d28c3c7-c79b-4312-b5a1-7df1abef3a23-kube-api-access-5g5b4\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:29.507560 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:29.507522 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:29.894521 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:29.894442 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" Oct 30 20:21:29.894521 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:29.894461 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4" event={"ID":"5d28c3c7-c79b-4312-b5a1-7df1abef3a23","Type":"ContainerDied","Data":"e6a8d19294d776e1daaa4ddb941d7ee9509976195564eed848bc774d033b43b0"} Oct 30 20:21:29.894719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:29.894534 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6a8d19294d776e1daaa4ddb941d7ee9509976195564eed848bc774d033b43b0" Oct 30 20:21:30.404280 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:30.401093 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:31.215211 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.215184 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" Oct 30 20:21:31.312554 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.312495 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:31.408538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.408502 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5776e63e-a778-4589-9e2a-10548456ff14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5776e63e-a778-4589-9e2a-10548456ff14-plugins-config-volume\") pod \"5776e63e-a778-4589-9e2a-10548456ff14\" (UID: \"5776e63e-a778-4589-9e2a-10548456ff14\") " Oct 30 20:21:31.409024 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.408857 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5776e63e-a778-4589-9e2a-10548456ff14-plugins-config-volume" pod "5776e63e-a778-4589-9e2a-10548456ff14" (UID: "5776e63e-a778-4589-9e2a-10548456ff14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:31.409024 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.408918 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5776e63e-a778-4589-9e2a-10548456ff14-kube-api-access-p6gmr\" (UniqueName: \"kubernetes.io/projected/5776e63e-a778-4589-9e2a-10548456ff14-kube-api-access-p6gmr\") pod \"5776e63e-a778-4589-9e2a-10548456ff14\" (UID: \"5776e63e-a778-4589-9e2a-10548456ff14\") " Oct 30 20:21:31.409415 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.409393 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5776e63e-a778-4589-9e2a-10548456ff14-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:31.410967 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.410949 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5776e63e-a778-4589-9e2a-10548456ff14-kube-api-access-p6gmr" pod "5776e63e-a778-4589-9e2a-10548456ff14" (UID: "5776e63e-a778-4589-9e2a-10548456ff14"). InnerVolumeSpecName "kube-api-access-p6gmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:31.511316 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.511280 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6gmr\" (UniqueName: \"kubernetes.io/projected/5776e63e-a778-4589-9e2a-10548456ff14-kube-api-access-p6gmr\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:31.912098 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.912019 5091 generic.go:356] "Generic (PLEG): container finished" podID="5776e63e-a778-4589-9e2a-10548456ff14" containerID="e38c2e5f865210eb2106a39e84b987795b89e2eabb1499bd2ed5ba9c3cd28595" exitCode=0 Oct 30 20:21:31.912098 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.912061 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" Oct 30 20:21:31.912344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.912106 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" event={"ID":"5776e63e-a778-4589-9e2a-10548456ff14","Type":"ContainerDied","Data":"e38c2e5f865210eb2106a39e84b987795b89e2eabb1499bd2ed5ba9c3cd28595"} Oct 30 20:21:31.912344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.912169 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6" event={"ID":"5776e63e-a778-4589-9e2a-10548456ff14","Type":"ContainerDied","Data":"34da5e0512c93d4b0c53d8736a5f76e654c1e7f0872a1f2a282f099c69c6b9a9"} Oct 30 20:21:31.912344 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.912201 5091 scope.go:122] "RemoveContainer" containerID="e38c2e5f865210eb2106a39e84b987795b89e2eabb1499bd2ed5ba9c3cd28595" Oct 30 20:21:31.924042 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.924019 5091 scope.go:122] "RemoveContainer" containerID="e38c2e5f865210eb2106a39e84b987795b89e2eabb1499bd2ed5ba9c3cd28595" Oct 30 20:21:31.924319 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:21:31.924298 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38c2e5f865210eb2106a39e84b987795b89e2eabb1499bd2ed5ba9c3cd28595\": container with ID starting with e38c2e5f865210eb2106a39e84b987795b89e2eabb1499bd2ed5ba9c3cd28595 not found: ID does not exist" containerID="e38c2e5f865210eb2106a39e84b987795b89e2eabb1499bd2ed5ba9c3cd28595" Oct 30 20:21:31.924399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.924333 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38c2e5f865210eb2106a39e84b987795b89e2eabb1499bd2ed5ba9c3cd28595"} err="failed to get container status \"e38c2e5f865210eb2106a39e84b987795b89e2eabb1499bd2ed5ba9c3cd28595\": rpc error: code = NotFound desc = could not find container \"e38c2e5f865210eb2106a39e84b987795b89e2eabb1499bd2ed5ba9c3cd28595\": container with ID starting with e38c2e5f865210eb2106a39e84b987795b89e2eabb1499bd2ed5ba9c3cd28595 not found: ID does not exist" Oct 30 20:21:31.941612 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.941588 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6"] Oct 30 20:21:31.945713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:31.945689 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c257e7-5c54cb9c47-tflt6"] Oct 30 20:21:32.377941 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:32.377844 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:32.442250 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:32.442218 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5776e63e-a778-4589-9e2a-10548456ff14" path="/var/lib/kubelet/pods/5776e63e-a778-4589-9e2a-10548456ff14/volumes" Oct 30 20:21:33.245580 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:33.245542 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:34.256347 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:34.256305 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:35.042823 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.042763 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:35.185556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.185507 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:35.356931 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.356854 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v"] Oct 30 20:21:35.357364 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.357132 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" podUID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" containerName="istio-proxy" containerID="cri-o://a32916c06e9e991d8c6eb389b20d3666185bfb7e766dc3a3a6511b7c2b8dc3d0" gracePeriod=30 Oct 30 20:21:35.463167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.462793 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:35.667882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.667795 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p"] Oct 30 20:21:35.675825 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.675790 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs"] Oct 30 20:21:35.788384 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.788314 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:35.927748 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.927671 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:35.990168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.989529 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6"] Oct 30 20:21:35.997168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.990956 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5776e63e-a778-4589-9e2a-10548456ff14" containerName="payload-pre-processing" Oct 30 20:21:35.997168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.990981 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5776e63e-a778-4589-9e2a-10548456ff14" containerName="payload-pre-processing" Oct 30 20:21:35.997168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.991052 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5d28c3c7-c79b-4312-b5a1-7df1abef3a23" containerName="revoke-keys" Oct 30 20:21:35.997168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.991061 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d28c3c7-c79b-4312-b5a1-7df1abef3a23" containerName="revoke-keys" Oct 30 20:21:35.997168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.991299 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d28c3c7-c79b-4312-b5a1-7df1abef3a23" containerName="revoke-keys" Oct 30 20:21:35.997168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:35.991346 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5776e63e-a778-4589-9e2a-10548456ff14" containerName="payload-pre-processing" Oct 30 20:21:36.060171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.054990 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6"] Oct 30 20:21:36.060171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.055327 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" Oct 30 20:21:36.075168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.061884 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:21:36.075168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.062583 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:21:36.106385 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.106345 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f4b503-4429-441a-b6fa-d547b87a08fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6\" (UID: \"86f4b503-4429-441a-b6fa-d547b87a08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" Oct 30 20:21:36.106618 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.106573 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcqlh\" (UniqueName: \"kubernetes.io/projected/86f4b503-4429-441a-b6fa-d547b87a08fa-kube-api-access-qcqlh\") pod \"maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6\" (UID: \"86f4b503-4429-441a-b6fa-d547b87a08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" Oct 30 20:21:36.106717 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.106674 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f4b503-4429-441a-b6fa-d547b87a08fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6\" (UID: \"86f4b503-4429-441a-b6fa-d547b87a08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" Oct 30 20:21:36.226332 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.219270 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:36.228480 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.227504 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f4b503-4429-441a-b6fa-d547b87a08fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6\" (UID: \"86f4b503-4429-441a-b6fa-d547b87a08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" Oct 30 20:21:36.228648 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.228610 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f4b503-4429-441a-b6fa-d547b87a08fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6\" (UID: \"86f4b503-4429-441a-b6fa-d547b87a08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" Oct 30 20:21:36.228751 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.228314 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f4b503-4429-441a-b6fa-d547b87a08fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6\" (UID: \"86f4b503-4429-441a-b6fa-d547b87a08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" Oct 30 20:21:36.230897 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.228802 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcqlh\" (UniqueName: \"kubernetes.io/projected/86f4b503-4429-441a-b6fa-d547b87a08fa-kube-api-access-qcqlh\") pod \"maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6\" (UID: \"86f4b503-4429-441a-b6fa-d547b87a08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" Oct 30 20:21:36.239520 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.238489 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f4b503-4429-441a-b6fa-d547b87a08fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6\" (UID: \"86f4b503-4429-441a-b6fa-d547b87a08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" Oct 30 20:21:36.245258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.245232 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcqlh\" (UniqueName: \"kubernetes.io/projected/86f4b503-4429-441a-b6fa-d547b87a08fa-kube-api-access-qcqlh\") pod \"maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6\" (UID: \"86f4b503-4429-441a-b6fa-d547b87a08fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" Oct 30 20:21:36.385747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.385719 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" Oct 30 20:21:36.681072 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:36.681042 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6"] Oct 30 20:21:36.681322 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:21:36.681297 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f4b503_4429_441a_b6fa_d547b87a08fa.slice/crio-3e69f7be4783193f577c72441207c6e5e4fcfe9f678e8618de20236b7dd9d92b WatchSource:0}: Error finding container 3e69f7be4783193f577c72441207c6e5e4fcfe9f678e8618de20236b7dd9d92b: Status 404 returned error can't find the container with id 3e69f7be4783193f577c72441207c6e5e4fcfe9f678e8618de20236b7dd9d92b Oct 30 20:21:37.030741 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:37.030706 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g"] Oct 30 20:21:37.035331 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:37.035305 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce9920af-qzp4g"] Oct 30 20:21:37.113796 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:37.112400 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:37.118758 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:37.118726 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" event={"ID":"86f4b503-4429-441a-b6fa-d547b87a08fa","Type":"ContainerStarted","Data":"9a75055fbb18c0efb4c61c0f345c2e21d1b3fa58d9647e853435293963a28e8e"} Oct 30 20:21:37.118944 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:37.118787 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" event={"ID":"86f4b503-4429-441a-b6fa-d547b87a08fa","Type":"ContainerStarted","Data":"3e69f7be4783193f577c72441207c6e5e4fcfe9f678e8618de20236b7dd9d92b"} Oct 30 20:21:37.160068 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:37.160013 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" podStartSLOduration=2.159984221 podStartE2EDuration="2.159984221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:37.154634948 +0000 UTC m=+3016.849842907" watchObservedRunningTime="2026-10-30 20:21:37.159984221 +0000 UTC m=+3016.855192180" Oct 30 20:21:37.440810 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:37.440774 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c"] Oct 30 20:21:37.441278 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:37.440971 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" podUID="92b34b40-9e41-4d49-b6b3-d8fb78227451" containerName="istio-proxy" containerID="cri-o://7139afc699f0abf99339334ee3f417fb15fbd38a1ec7240cfc082fad9e54d6c8" gracePeriod=30 Oct 30 20:21:37.903173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:37.875186 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:38.128992 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:38.128959 5091 generic.go:356] "Generic (PLEG): container finished" podID="86f4b503-4429-441a-b6fa-d547b87a08fa" containerID="9a75055fbb18c0efb4c61c0f345c2e21d1b3fa58d9647e853435293963a28e8e" exitCode=0 Oct 30 20:21:38.129239 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:38.129023 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" event={"ID":"86f4b503-4429-441a-b6fa-d547b87a08fa","Type":"ContainerDied","Data":"9a75055fbb18c0efb4c61c0f345c2e21d1b3fa58d9647e853435293963a28e8e"} Oct 30 20:21:38.448409 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:38.448373 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77e16466-369a-423c-82c1-cbb82fd66fca" path="/var/lib/kubelet/pods/77e16466-369a-423c-82c1-cbb82fd66fca/volumes" Oct 30 20:21:38.778724 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:38.778685 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:38.785165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:38.779593 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" podUID="85571ef8-1def-4ecc-af2a-39168d653b2e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:21:39.291303 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:39.291274 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" Oct 30 20:21:39.385648 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:39.385609 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86f4b503-4429-441a-b6fa-d547b87a08fa-kube-api-access-qcqlh\" (UniqueName: \"kubernetes.io/projected/86f4b503-4429-441a-b6fa-d547b87a08fa-kube-api-access-qcqlh\") pod \"86f4b503-4429-441a-b6fa-d547b87a08fa\" (UID: \"86f4b503-4429-441a-b6fa-d547b87a08fa\") " Oct 30 20:21:39.385844 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:39.385667 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86f4b503-4429-441a-b6fa-d547b87a08fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f4b503-4429-441a-b6fa-d547b87a08fa-maas-api-openshift-ca\") pod \"86f4b503-4429-441a-b6fa-d547b87a08fa\" (UID: \"86f4b503-4429-441a-b6fa-d547b87a08fa\") " Oct 30 20:21:39.385844 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:39.385769 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86f4b503-4429-441a-b6fa-d547b87a08fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f4b503-4429-441a-b6fa-d547b87a08fa-maas-api-xks-ca\") pod \"86f4b503-4429-441a-b6fa-d547b87a08fa\" (UID: \"86f4b503-4429-441a-b6fa-d547b87a08fa\") " Oct 30 20:21:39.386097 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:39.386073 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f4b503-4429-441a-b6fa-d547b87a08fa-maas-api-openshift-ca" pod "86f4b503-4429-441a-b6fa-d547b87a08fa" (UID: "86f4b503-4429-441a-b6fa-d547b87a08fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:39.386435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:39.386396 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86f4b503-4429-441a-b6fa-d547b87a08fa-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:39.387977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:39.387955 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f4b503-4429-441a-b6fa-d547b87a08fa-kube-api-access-qcqlh" pod "86f4b503-4429-441a-b6fa-d547b87a08fa" (UID: "86f4b503-4429-441a-b6fa-d547b87a08fa"). InnerVolumeSpecName "kube-api-access-qcqlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:39.388265 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:39.388240 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86f4b503-4429-441a-b6fa-d547b87a08fa-maas-api-xks-ca" pod "86f4b503-4429-441a-b6fa-d547b87a08fa" (UID: "86f4b503-4429-441a-b6fa-d547b87a08fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:21:39.487435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:39.487403 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86f4b503-4429-441a-b6fa-d547b87a08fa-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:39.487435 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:39.487433 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcqlh\" (UniqueName: \"kubernetes.io/projected/86f4b503-4429-441a-b6fa-d547b87a08fa-kube-api-access-qcqlh\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:39.534921 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:39.534447 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:40.126109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.126041 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" Oct 30 20:21:40.148708 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.148669 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" Oct 30 20:21:40.148890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.148673 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6" event={"ID":"86f4b503-4429-441a-b6fa-d547b87a08fa","Type":"ContainerDied","Data":"3e69f7be4783193f577c72441207c6e5e4fcfe9f678e8618de20236b7dd9d92b"} Oct 30 20:21:40.148890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.148829 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e69f7be4783193f577c72441207c6e5e4fcfe9f678e8618de20236b7dd9d92b" Oct 30 20:21:40.150066 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.150040 5091 generic.go:356] "Generic (PLEG): container finished" podID="85571ef8-1def-4ecc-af2a-39168d653b2e" containerID="0239af929a749312bd3456b8c545afbe6f79f017faa29ab5a7a5fed9791d234a" exitCode=0 Oct 30 20:21:40.150226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.150077 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" Oct 30 20:21:40.150226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.150115 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" event={"ID":"85571ef8-1def-4ecc-af2a-39168d653b2e","Type":"ContainerDied","Data":"0239af929a749312bd3456b8c545afbe6f79f017faa29ab5a7a5fed9791d234a"} Oct 30 20:21:40.150226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.150178 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl" event={"ID":"85571ef8-1def-4ecc-af2a-39168d653b2e","Type":"ContainerDied","Data":"d7acd9ea1ea4b4f09f687a03674681df45fe45d5837a6f74abf82898bea7bd63"} Oct 30 20:21:40.150226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.150202 5091 scope.go:122] "RemoveContainer" containerID="0239af929a749312bd3456b8c545afbe6f79f017faa29ab5a7a5fed9791d234a" Oct 30 20:21:40.166981 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.166955 5091 scope.go:122] "RemoveContainer" containerID="0239af929a749312bd3456b8c545afbe6f79f017faa29ab5a7a5fed9791d234a" Oct 30 20:21:40.167494 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:21:40.167463 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0239af929a749312bd3456b8c545afbe6f79f017faa29ab5a7a5fed9791d234a\": container with ID starting with 0239af929a749312bd3456b8c545afbe6f79f017faa29ab5a7a5fed9791d234a not found: ID does not exist" containerID="0239af929a749312bd3456b8c545afbe6f79f017faa29ab5a7a5fed9791d234a" Oct 30 20:21:40.167624 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.167504 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0239af929a749312bd3456b8c545afbe6f79f017faa29ab5a7a5fed9791d234a"} err="failed to get container status \"0239af929a749312bd3456b8c545afbe6f79f017faa29ab5a7a5fed9791d234a\": rpc error: code = NotFound desc = could not find container \"0239af929a749312bd3456b8c545afbe6f79f017faa29ab5a7a5fed9791d234a\": container with ID starting with 0239af929a749312bd3456b8c545afbe6f79f017faa29ab5a7a5fed9791d234a not found: ID does not exist" Oct 30 20:21:40.198180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.198122 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85571ef8-1def-4ecc-af2a-39168d653b2e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85571ef8-1def-4ecc-af2a-39168d653b2e-plugins-config-volume\") pod \"85571ef8-1def-4ecc-af2a-39168d653b2e\" (UID: \"85571ef8-1def-4ecc-af2a-39168d653b2e\") " Oct 30 20:21:40.198374 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.198213 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85571ef8-1def-4ecc-af2a-39168d653b2e-kube-api-access-pqlft\" (UniqueName: \"kubernetes.io/projected/85571ef8-1def-4ecc-af2a-39168d653b2e-kube-api-access-pqlft\") pod \"85571ef8-1def-4ecc-af2a-39168d653b2e\" (UID: \"85571ef8-1def-4ecc-af2a-39168d653b2e\") " Oct 30 20:21:40.198554 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.198530 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85571ef8-1def-4ecc-af2a-39168d653b2e-plugins-config-volume" pod "85571ef8-1def-4ecc-af2a-39168d653b2e" (UID: "85571ef8-1def-4ecc-af2a-39168d653b2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:40.198869 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.198845 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85571ef8-1def-4ecc-af2a-39168d653b2e-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:40.200891 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.200861 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85571ef8-1def-4ecc-af2a-39168d653b2e-kube-api-access-pqlft" pod "85571ef8-1def-4ecc-af2a-39168d653b2e" (UID: "85571ef8-1def-4ecc-af2a-39168d653b2e"). InnerVolumeSpecName "kube-api-access-pqlft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:40.300770 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.300732 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqlft\" (UniqueName: \"kubernetes.io/projected/85571ef8-1def-4ecc-af2a-39168d653b2e-kube-api-access-pqlft\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:40.471829 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.471789 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:40.511736 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.511683 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl"] Oct 30 20:21:40.516940 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.516914 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-89c124-b8cccb68c-8rlnl"] Oct 30 20:21:40.668365 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.668326 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" podUID="0b1b31d4-eafb-4f36-922a-c7fdcc8c007f" containerName="payload-processing" containerID="cri-o://8a56a4980659f1dce4e6a96661e21c43e4226e290ebdfd016b44af1132cf2ab4" gracePeriod=25 Oct 30 20:21:40.676371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.676343 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" podUID="fa98f754-341a-426e-91cf-1c914b3a2433" containerName="payload-pre-processing" containerID="cri-o://7301ad28de5127e792170f28099af85f9db7cff71f9a19a0a0ec9ddf24771a58" gracePeriod=25 Oct 30 20:21:40.680807 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.680785 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:21:40.808644 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.808612 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-credential-socket\") pod \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " Oct 30 20:21:40.808818 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.808672 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-workload-socket\") pod \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " Oct 30 20:21:40.808818 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.808730 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-podinfo\") pod \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " Oct 30 20:21:40.808818 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.808757 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-workload-certs\") pod \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " Oct 30 20:21:40.809022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.808850 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320dc7ae-3660-41d1-bc53-49d3427a0b6d-kube-api-access-vnnpb\" (UniqueName: \"kubernetes.io/projected/320dc7ae-3660-41d1-bc53-49d3427a0b6d-kube-api-access-vnnpb\") pod \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " Oct 30 20:21:40.809022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.808884 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-envoy\") pod \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " Oct 30 20:21:40.809022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.808894 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-credential-socket" pod "320dc7ae-3660-41d1-bc53-49d3427a0b6d" (UID: "320dc7ae-3660-41d1-bc53-49d3427a0b6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:40.809022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.808932 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-data\") pod \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " Oct 30 20:21:40.809022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.808973 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istiod-ca-cert\") pod \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " Oct 30 20:21:40.809022 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.809013 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-token\" (UniqueName: \"kubernetes.io/projected/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-token\") pod \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\" (UID: \"320dc7ae-3660-41d1-bc53-49d3427a0b6d\") " Oct 30 20:21:40.809438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.809072 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-workload-socket" pod "320dc7ae-3660-41d1-bc53-49d3427a0b6d" (UID: "320dc7ae-3660-41d1-bc53-49d3427a0b6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:40.809438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.809108 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-workload-certs" pod "320dc7ae-3660-41d1-bc53-49d3427a0b6d" (UID: "320dc7ae-3660-41d1-bc53-49d3427a0b6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:40.809438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.809312 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-data" pod "320dc7ae-3660-41d1-bc53-49d3427a0b6d" (UID: "320dc7ae-3660-41d1-bc53-49d3427a0b6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:40.809610 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.809457 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istiod-ca-cert" pod "320dc7ae-3660-41d1-bc53-49d3427a0b6d" (UID: "320dc7ae-3660-41d1-bc53-49d3427a0b6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:40.809737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.809717 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:40.809778 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.809744 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:40.809778 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.809762 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:40.809778 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.809776 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:40.809909 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.809789 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:40.811098 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.811072 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-podinfo" pod "320dc7ae-3660-41d1-bc53-49d3427a0b6d" (UID: "320dc7ae-3660-41d1-bc53-49d3427a0b6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:21:40.811492 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.811471 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-envoy" pod "320dc7ae-3660-41d1-bc53-49d3427a0b6d" (UID: "320dc7ae-3660-41d1-bc53-49d3427a0b6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:40.811587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.811514 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-token" pod "320dc7ae-3660-41d1-bc53-49d3427a0b6d" (UID: "320dc7ae-3660-41d1-bc53-49d3427a0b6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:40.811927 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.811905 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320dc7ae-3660-41d1-bc53-49d3427a0b6d-kube-api-access-vnnpb" pod "320dc7ae-3660-41d1-bc53-49d3427a0b6d" (UID: "320dc7ae-3660-41d1-bc53-49d3427a0b6d"). InnerVolumeSpecName "kube-api-access-vnnpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:40.911691 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.911615 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnnpb\" (UniqueName: \"kubernetes.io/projected/320dc7ae-3660-41d1-bc53-49d3427a0b6d-kube-api-access-vnnpb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:40.911691 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.911655 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:40.911691 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.911672 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:40.911691 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:40.911688 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/320dc7ae-3660-41d1-bc53-49d3427a0b6d-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:41.161170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:41.161106 5091 generic.go:356] "Generic (PLEG): container finished" podID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" containerID="a32916c06e9e991d8c6eb389b20d3666185bfb7e766dc3a3a6511b7c2b8dc3d0" exitCode=0 Oct 30 20:21:41.161356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:41.161183 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" Oct 30 20:21:41.161356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:41.161208 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" event={"ID":"320dc7ae-3660-41d1-bc53-49d3427a0b6d","Type":"ContainerDied","Data":"a32916c06e9e991d8c6eb389b20d3666185bfb7e766dc3a3a6511b7c2b8dc3d0"} Oct 30 20:21:41.161356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:41.161274 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v" event={"ID":"320dc7ae-3660-41d1-bc53-49d3427a0b6d","Type":"ContainerDied","Data":"82a4ce5c726095b8621b583e30cca42efa5dc801d63ec2d54ce4d2daad229d16"} Oct 30 20:21:41.161356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:41.161304 5091 scope.go:122] "RemoveContainer" containerID="a32916c06e9e991d8c6eb389b20d3666185bfb7e766dc3a3a6511b7c2b8dc3d0" Oct 30 20:21:41.176524 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:41.176492 5091 scope.go:122] "RemoveContainer" containerID="f834c3bc0ffbb915da2643b1434f92a79d4a8a71e0eb0914ad67c8e29cfab6a7" Oct 30 20:21:41.190178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:41.190123 5091 scope.go:122] "RemoveContainer" containerID="a32916c06e9e991d8c6eb389b20d3666185bfb7e766dc3a3a6511b7c2b8dc3d0" Oct 30 20:21:41.190498 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:21:41.190472 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32916c06e9e991d8c6eb389b20d3666185bfb7e766dc3a3a6511b7c2b8dc3d0\": container with ID starting with a32916c06e9e991d8c6eb389b20d3666185bfb7e766dc3a3a6511b7c2b8dc3d0 not found: ID does not exist" containerID="a32916c06e9e991d8c6eb389b20d3666185bfb7e766dc3a3a6511b7c2b8dc3d0" Oct 30 20:21:41.190587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:41.190511 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32916c06e9e991d8c6eb389b20d3666185bfb7e766dc3a3a6511b7c2b8dc3d0"} err="failed to get container status \"a32916c06e9e991d8c6eb389b20d3666185bfb7e766dc3a3a6511b7c2b8dc3d0\": rpc error: code = NotFound desc = could not find container \"a32916c06e9e991d8c6eb389b20d3666185bfb7e766dc3a3a6511b7c2b8dc3d0\": container with ID starting with a32916c06e9e991d8c6eb389b20d3666185bfb7e766dc3a3a6511b7c2b8dc3d0 not found: ID does not exist" Oct 30 20:21:41.190587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:41.190535 5091 scope.go:122] "RemoveContainer" containerID="f834c3bc0ffbb915da2643b1434f92a79d4a8a71e0eb0914ad67c8e29cfab6a7" Oct 30 20:21:41.190822 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:21:41.190781 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f834c3bc0ffbb915da2643b1434f92a79d4a8a71e0eb0914ad67c8e29cfab6a7\": container with ID starting with f834c3bc0ffbb915da2643b1434f92a79d4a8a71e0eb0914ad67c8e29cfab6a7 not found: ID does not exist" containerID="f834c3bc0ffbb915da2643b1434f92a79d4a8a71e0eb0914ad67c8e29cfab6a7" Oct 30 20:21:41.190891 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:41.190816 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f834c3bc0ffbb915da2643b1434f92a79d4a8a71e0eb0914ad67c8e29cfab6a7"} err="failed to get container status \"f834c3bc0ffbb915da2643b1434f92a79d4a8a71e0eb0914ad67c8e29cfab6a7\": rpc error: code = NotFound desc = could not find container \"f834c3bc0ffbb915da2643b1434f92a79d4a8a71e0eb0914ad67c8e29cfab6a7\": container with ID starting with f834c3bc0ffbb915da2643b1434f92a79d4a8a71e0eb0914ad67c8e29cfab6a7 not found: ID does not exist" Oct 30 20:21:41.193728 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:41.193705 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v"] Oct 30 20:21:41.200138 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:41.200113 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-89c124-openshift-default-dfcf98449-kv85v"] Oct 30 20:21:41.296763 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:41.290774 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:42.045031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.044985 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:42.320268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.320175 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" podUID="0b1b31d4-eafb-4f36-922a-c7fdcc8c007f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:21:42.320268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.320202 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" podUID="fa98f754-341a-426e-91cf-1c914b3a2433" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:21:42.444132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.444088 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" path="/var/lib/kubelet/pods/320dc7ae-3660-41d1-bc53-49d3427a0b6d/volumes" Oct 30 20:21:42.444777 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.444731 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85571ef8-1def-4ecc-af2a-39168d653b2e" path="/var/lib/kubelet/pods/85571ef8-1def-4ecc-af2a-39168d653b2e/volumes" Oct 30 20:21:42.721279 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.721249 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:42.750009 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.749981 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:21:42.835910 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.835877 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-workload-certs\") pod \"92b34b40-9e41-4d49-b6b3-d8fb78227451\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " Oct 30 20:21:42.836083 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.835936 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-envoy\") pod \"92b34b40-9e41-4d49-b6b3-d8fb78227451\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " Oct 30 20:21:42.836083 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.835989 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b34b40-9e41-4d49-b6b3-d8fb78227451-kube-api-access-7458q\" (UniqueName: \"kubernetes.io/projected/92b34b40-9e41-4d49-b6b3-d8fb78227451-kube-api-access-7458q\") pod \"92b34b40-9e41-4d49-b6b3-d8fb78227451\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " Oct 30 20:21:42.836083 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.836051 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-token\" (UniqueName: \"kubernetes.io/projected/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-token\") pod \"92b34b40-9e41-4d49-b6b3-d8fb78227451\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " Oct 30 20:21:42.836240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.836117 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-workload-socket\") pod \"92b34b40-9e41-4d49-b6b3-d8fb78227451\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " Oct 30 20:21:42.836240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.836222 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-credential-socket\") pod \"92b34b40-9e41-4d49-b6b3-d8fb78227451\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " Oct 30 20:21:42.836357 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.836238 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-workload-certs" pod "92b34b40-9e41-4d49-b6b3-d8fb78227451" (UID: "92b34b40-9e41-4d49-b6b3-d8fb78227451"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:42.836357 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.836264 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-podinfo\") pod \"92b34b40-9e41-4d49-b6b3-d8fb78227451\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " Oct 30 20:21:42.836357 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.836344 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-data\") pod \"92b34b40-9e41-4d49-b6b3-d8fb78227451\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " Oct 30 20:21:42.836529 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.836419 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92b34b40-9e41-4d49-b6b3-d8fb78227451-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92b34b40-9e41-4d49-b6b3-d8fb78227451-istiod-ca-cert\") pod \"92b34b40-9e41-4d49-b6b3-d8fb78227451\" (UID: \"92b34b40-9e41-4d49-b6b3-d8fb78227451\") " Oct 30 20:21:42.836529 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.836490 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-credential-socket" pod "92b34b40-9e41-4d49-b6b3-d8fb78227451" (UID: "92b34b40-9e41-4d49-b6b3-d8fb78227451"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:42.836737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.836707 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-data" pod "92b34b40-9e41-4d49-b6b3-d8fb78227451" (UID: "92b34b40-9e41-4d49-b6b3-d8fb78227451"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:42.836737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.836721 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-workload-socket" pod "92b34b40-9e41-4d49-b6b3-d8fb78227451" (UID: "92b34b40-9e41-4d49-b6b3-d8fb78227451"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:42.837067 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.837037 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b34b40-9e41-4d49-b6b3-d8fb78227451-istiod-ca-cert" pod "92b34b40-9e41-4d49-b6b3-d8fb78227451" (UID: "92b34b40-9e41-4d49-b6b3-d8fb78227451"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:42.837328 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.837307 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:42.837426 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.837335 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:42.837426 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.837352 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:42.837426 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.837365 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92b34b40-9e41-4d49-b6b3-d8fb78227451-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:42.837426 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.837380 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:42.838430 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.838401 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-envoy" pod "92b34b40-9e41-4d49-b6b3-d8fb78227451" (UID: "92b34b40-9e41-4d49-b6b3-d8fb78227451"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:42.838529 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.838497 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b34b40-9e41-4d49-b6b3-d8fb78227451-kube-api-access-7458q" pod "92b34b40-9e41-4d49-b6b3-d8fb78227451" (UID: "92b34b40-9e41-4d49-b6b3-d8fb78227451"). InnerVolumeSpecName "kube-api-access-7458q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:42.838630 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.838613 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-token" pod "92b34b40-9e41-4d49-b6b3-d8fb78227451" (UID: "92b34b40-9e41-4d49-b6b3-d8fb78227451"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:42.838859 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.838843 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-podinfo" pod "92b34b40-9e41-4d49-b6b3-d8fb78227451" (UID: "92b34b40-9e41-4d49-b6b3-d8fb78227451"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:21:42.924285 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.923087 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:42.949473 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.949356 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:42.949473 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.949396 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7458q\" (UniqueName: \"kubernetes.io/projected/92b34b40-9e41-4d49-b6b3-d8fb78227451-kube-api-access-7458q\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:42.949473 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.949414 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:42.949473 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:42.949433 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92b34b40-9e41-4d49-b6b3-d8fb78227451-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:43.028165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:43.026322 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4"] Oct 30 20:21:43.028584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:43.028558 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-517027-9m2r4"] Oct 30 20:21:43.073701 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:43.073660 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:43.189970 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:43.189890 5091 generic.go:356] "Generic (PLEG): container finished" podID="92b34b40-9e41-4d49-b6b3-d8fb78227451" containerID="7139afc699f0abf99339334ee3f417fb15fbd38a1ec7240cfc082fad9e54d6c8" exitCode=0 Oct 30 20:21:43.189970 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:43.189959 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" Oct 30 20:21:43.190251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:43.189974 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" event={"ID":"92b34b40-9e41-4d49-b6b3-d8fb78227451","Type":"ContainerDied","Data":"7139afc699f0abf99339334ee3f417fb15fbd38a1ec7240cfc082fad9e54d6c8"} Oct 30 20:21:43.190251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:43.190032 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c" event={"ID":"92b34b40-9e41-4d49-b6b3-d8fb78227451","Type":"ContainerDied","Data":"6d5e56a2b2315574ab44a520cce7839a22c8a823647f33af513ae265f76efa57"} Oct 30 20:21:43.190251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:43.190060 5091 scope.go:122] "RemoveContainer" containerID="7139afc699f0abf99339334ee3f417fb15fbd38a1ec7240cfc082fad9e54d6c8" Oct 30 20:21:43.203572 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:43.203551 5091 scope.go:122] "RemoveContainer" containerID="7139afc699f0abf99339334ee3f417fb15fbd38a1ec7240cfc082fad9e54d6c8" Oct 30 20:21:43.203903 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:21:43.203882 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7139afc699f0abf99339334ee3f417fb15fbd38a1ec7240cfc082fad9e54d6c8\": container with ID starting with 7139afc699f0abf99339334ee3f417fb15fbd38a1ec7240cfc082fad9e54d6c8 not found: ID does not exist" containerID="7139afc699f0abf99339334ee3f417fb15fbd38a1ec7240cfc082fad9e54d6c8" Oct 30 20:21:43.203980 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:43.203916 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7139afc699f0abf99339334ee3f417fb15fbd38a1ec7240cfc082fad9e54d6c8"} err="failed to get container status \"7139afc699f0abf99339334ee3f417fb15fbd38a1ec7240cfc082fad9e54d6c8\": rpc error: code = NotFound desc = could not find container \"7139afc699f0abf99339334ee3f417fb15fbd38a1ec7240cfc082fad9e54d6c8\": container with ID starting with 7139afc699f0abf99339334ee3f417fb15fbd38a1ec7240cfc082fad9e54d6c8 not found: ID does not exist" Oct 30 20:21:43.222766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:43.222721 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c"] Oct 30 20:21:43.223710 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:43.223672 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:43.227977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:43.227951 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-rgs5c"] Oct 30 20:21:43.503403 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:43.503368 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:43.974200 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:43.973978 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:44.441766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:44.441735 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d28c3c7-c79b-4312-b5a1-7df1abef3a23" path="/var/lib/kubelet/pods/5d28c3c7-c79b-4312-b5a1-7df1abef3a23/volumes" Oct 30 20:21:44.442316 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:44.442297 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92b34b40-9e41-4d49-b6b3-d8fb78227451" path="/var/lib/kubelet/pods/92b34b40-9e41-4d49-b6b3-d8fb78227451/volumes" Oct 30 20:21:44.809095 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:44.809011 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:45.670850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:45.668115 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:45.978214 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:21:45.978112 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:21:46.155888 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:46.155853 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn"] Oct 30 20:21:46.275361 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:46.275329 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:46.448035 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:46.447987 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:46.603047 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:46.602957 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:47.502167 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:47.495746 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:48.381170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:48.378194 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:49.290735 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:49.287496 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:50.092170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:50.090462 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:50.965756 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:50.964232 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:51.156587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:51.156543 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" podUID="71b363c1-4123-454f-86aa-daab26507f6e" containerName="payload-processing" containerID="cri-o://3f1ffb0d23ff1a69fa42df8d63ae842a7ea6768bdfb76f3746def3287920e7a2" gracePeriod=25 Oct 30 20:21:51.208058 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:51.208025 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:51.355130 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:51.355056 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:51.453205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:51.453180 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" Oct 30 20:21:51.498403 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:51.498363 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq"] Oct 30 20:21:51.498638 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:51.498610 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" podUID="43e01282-cbf5-43e3-9815-32c7d4439800" containerName="istio-proxy" containerID="cri-o://08cb1dcb463b7df7783bb8e40545b717c07a5b7956f0378b2158ee057d8967ec" gracePeriod=30 Oct 30 20:21:51.574898 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:51.574755 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:51.612682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:51.612624 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71b363c1-4123-454f-86aa-daab26507f6e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b363c1-4123-454f-86aa-daab26507f6e-plugins-config-volume\") pod \"71b363c1-4123-454f-86aa-daab26507f6e\" (UID: \"71b363c1-4123-454f-86aa-daab26507f6e\") " Oct 30 20:21:51.637319 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:51.619079 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71b363c1-4123-454f-86aa-daab26507f6e-plugins-config-volume" pod "71b363c1-4123-454f-86aa-daab26507f6e" (UID: "71b363c1-4123-454f-86aa-daab26507f6e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:51.638064 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:51.638038 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71b363c1-4123-454f-86aa-daab26507f6e-kube-api-access-bkj26\" (UniqueName: \"kubernetes.io/projected/71b363c1-4123-454f-86aa-daab26507f6e-kube-api-access-bkj26\") pod \"71b363c1-4123-454f-86aa-daab26507f6e\" (UID: \"71b363c1-4123-454f-86aa-daab26507f6e\") " Oct 30 20:21:51.643010 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:51.642966 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71b363c1-4123-454f-86aa-daab26507f6e-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:51.669577 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:51.669533 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71b363c1-4123-454f-86aa-daab26507f6e-kube-api-access-bkj26" pod "71b363c1-4123-454f-86aa-daab26507f6e" (UID: "71b363c1-4123-454f-86aa-daab26507f6e"). InnerVolumeSpecName "kube-api-access-bkj26". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:51.744584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:51.744548 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkj26\" (UniqueName: \"kubernetes.io/projected/71b363c1-4123-454f-86aa-daab26507f6e-kube-api-access-bkj26\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:51.777090 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:51.766540 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:52.018581 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.018547 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw"] Oct 30 20:21:52.019233 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019208 5091 cpu_manager.go:414] "containerMap: removing container" podUID="92b34b40-9e41-4d49-b6b3-d8fb78227451" containerName="istio-proxy" Oct 30 20:21:52.019233 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019232 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b34b40-9e41-4d49-b6b3-d8fb78227451" containerName="istio-proxy" Oct 30 20:21:52.019450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019254 5091 cpu_manager.go:414] "containerMap: removing container" podUID="71b363c1-4123-454f-86aa-daab26507f6e" containerName="payload-processing" Oct 30 20:21:52.019450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019263 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71b363c1-4123-454f-86aa-daab26507f6e" containerName="payload-processing" Oct 30 20:21:52.019450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019286 5091 cpu_manager.go:414] "containerMap: removing container" podUID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" containerName="istio-proxy" Oct 30 20:21:52.019450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019295 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" containerName="istio-proxy" Oct 30 20:21:52.019450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019310 5091 cpu_manager.go:414] "containerMap: removing container" podUID="86f4b503-4429-441a-b6fa-d547b87a08fa" containerName="revoke-keys" Oct 30 20:21:52.019450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019317 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86f4b503-4429-441a-b6fa-d547b87a08fa" containerName="revoke-keys" Oct 30 20:21:52.019450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019350 5091 cpu_manager.go:414] "containerMap: removing container" podUID="85571ef8-1def-4ecc-af2a-39168d653b2e" containerName="payload-pre-processing" Oct 30 20:21:52.019450 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019356 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85571ef8-1def-4ecc-af2a-39168d653b2e" containerName="payload-pre-processing" Oct 30 20:21:52.019880 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019538 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="71b363c1-4123-454f-86aa-daab26507f6e" containerName="payload-processing" Oct 30 20:21:52.019880 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019556 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" containerName="istio-proxy" Oct 30 20:21:52.019880 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019566 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="86f4b503-4429-441a-b6fa-d547b87a08fa" containerName="revoke-keys" Oct 30 20:21:52.019880 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019581 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="85571ef8-1def-4ecc-af2a-39168d653b2e" containerName="payload-pre-processing" Oct 30 20:21:52.019880 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019597 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" containerName="istio-proxy" Oct 30 20:21:52.019880 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.019611 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b34b40-9e41-4d49-b6b3-d8fb78227451" containerName="istio-proxy" Oct 30 20:21:52.023551 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.023529 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" Oct 30 20:21:52.033658 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.033187 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw"] Oct 30 20:21:52.040262 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.040230 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr"] Oct 30 20:21:52.047716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.047666 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4917e43-nwsvr"] Oct 30 20:21:52.120850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.120773 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:21:52.120850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.120819 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:21:52.149985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.149948 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69437371-bf5b-4b40-9b57-cae3356c388a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw\" (UID: \"69437371-bf5b-4b40-9b57-cae3356c388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" Oct 30 20:21:52.150209 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.150079 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69437371-bf5b-4b40-9b57-cae3356c388a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw\" (UID: \"69437371-bf5b-4b40-9b57-cae3356c388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" Oct 30 20:21:52.150286 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.150226 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8ss5\" (UniqueName: \"kubernetes.io/projected/69437371-bf5b-4b40-9b57-cae3356c388a-kube-api-access-r8ss5\") pod \"maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw\" (UID: \"69437371-bf5b-4b40-9b57-cae3356c388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" Oct 30 20:21:52.251227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.251178 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8ss5\" (UniqueName: \"kubernetes.io/projected/69437371-bf5b-4b40-9b57-cae3356c388a-kube-api-access-r8ss5\") pod \"maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw\" (UID: \"69437371-bf5b-4b40-9b57-cae3356c388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" Oct 30 20:21:52.251431 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.251370 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69437371-bf5b-4b40-9b57-cae3356c388a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw\" (UID: \"69437371-bf5b-4b40-9b57-cae3356c388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" Oct 30 20:21:52.251517 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.251487 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69437371-bf5b-4b40-9b57-cae3356c388a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw\" (UID: \"69437371-bf5b-4b40-9b57-cae3356c388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" Oct 30 20:21:52.252225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.252191 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69437371-bf5b-4b40-9b57-cae3356c388a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw\" (UID: \"69437371-bf5b-4b40-9b57-cae3356c388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" Oct 30 20:21:52.254159 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.254121 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69437371-bf5b-4b40-9b57-cae3356c388a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw\" (UID: \"69437371-bf5b-4b40-9b57-cae3356c388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" Oct 30 20:21:52.267892 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.267868 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8ss5\" (UniqueName: \"kubernetes.io/projected/69437371-bf5b-4b40-9b57-cae3356c388a-kube-api-access-r8ss5\") pod \"maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw\" (UID: \"69437371-bf5b-4b40-9b57-cae3356c388a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" Oct 30 20:21:52.281929 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.281901 5091 generic.go:356] "Generic (PLEG): container finished" podID="71b363c1-4123-454f-86aa-daab26507f6e" containerID="3f1ffb0d23ff1a69fa42df8d63ae842a7ea6768bdfb76f3746def3287920e7a2" exitCode=0 Oct 30 20:21:52.282062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.281950 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" Oct 30 20:21:52.282062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.281954 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" event={"ID":"71b363c1-4123-454f-86aa-daab26507f6e","Type":"ContainerDied","Data":"3f1ffb0d23ff1a69fa42df8d63ae842a7ea6768bdfb76f3746def3287920e7a2"} Oct 30 20:21:52.282062 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.282048 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn" event={"ID":"71b363c1-4123-454f-86aa-daab26507f6e","Type":"ContainerDied","Data":"edd163560e9081ff275c11345359ebfb399461e56f5d4772e7313ee1e17a7c9f"} Oct 30 20:21:52.282263 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.282087 5091 scope.go:122] "RemoveContainer" containerID="3f1ffb0d23ff1a69fa42df8d63ae842a7ea6768bdfb76f3746def3287920e7a2" Oct 30 20:21:52.294082 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.294062 5091 scope.go:122] "RemoveContainer" containerID="3f1ffb0d23ff1a69fa42df8d63ae842a7ea6768bdfb76f3746def3287920e7a2" Oct 30 20:21:52.294388 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:21:52.294366 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f1ffb0d23ff1a69fa42df8d63ae842a7ea6768bdfb76f3746def3287920e7a2\": container with ID starting with 3f1ffb0d23ff1a69fa42df8d63ae842a7ea6768bdfb76f3746def3287920e7a2 not found: ID does not exist" containerID="3f1ffb0d23ff1a69fa42df8d63ae842a7ea6768bdfb76f3746def3287920e7a2" Oct 30 20:21:52.294472 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.294398 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f1ffb0d23ff1a69fa42df8d63ae842a7ea6768bdfb76f3746def3287920e7a2"} err="failed to get container status \"3f1ffb0d23ff1a69fa42df8d63ae842a7ea6768bdfb76f3746def3287920e7a2\": rpc error: code = NotFound desc = could not find container \"3f1ffb0d23ff1a69fa42df8d63ae842a7ea6768bdfb76f3746def3287920e7a2\": container with ID starting with 3f1ffb0d23ff1a69fa42df8d63ae842a7ea6768bdfb76f3746def3287920e7a2 not found: ID does not exist" Oct 30 20:21:52.311868 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.311837 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn"] Oct 30 20:21:52.317296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.317254 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bdf9e6-7544c775f8-dcjpn"] Oct 30 20:21:52.320208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.320179 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" podUID="fa98f754-341a-426e-91cf-1c914b3a2433" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:21:52.320208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.320200 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" podUID="0b1b31d4-eafb-4f36-922a-c7fdcc8c007f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:21:52.345716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.345681 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" Oct 30 20:21:52.478166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.473432 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:52.478166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.474531 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd" path="/var/lib/kubelet/pods/1bcbdc53-3a4f-4f3f-8145-f0f5e02f41bd/volumes" Oct 30 20:21:52.478166 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.474959 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71b363c1-4123-454f-86aa-daab26507f6e" path="/var/lib/kubelet/pods/71b363c1-4123-454f-86aa-daab26507f6e/volumes" Oct 30 20:21:52.702057 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.702030 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw"] Oct 30 20:21:52.882117 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.882076 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6"] Oct 30 20:21:52.885626 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.885598 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bdf9e6-56vv6"] Oct 30 20:21:52.895248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:52.895220 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:53.043317 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:53.043284 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:53.292122 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:53.292081 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" event={"ID":"69437371-bf5b-4b40-9b57-cae3356c388a","Type":"ContainerStarted","Data":"4def0611513e1a6fcb6e8ad610ac9509f89fd9fd72b5980a50cdc0972341c500"} Oct 30 20:21:53.292322 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:53.292182 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" event={"ID":"69437371-bf5b-4b40-9b57-cae3356c388a","Type":"ContainerStarted","Data":"054d520383c7a72ed652f43d886e3f93f1f909093b6464392cdfa1da19d20990"} Oct 30 20:21:53.313607 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:53.313561 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" podStartSLOduration=2.313547758 podStartE2EDuration="2.313547758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:21:53.312820057 +0000 UTC m=+3033.008028017" watchObservedRunningTime="2026-10-30 20:21:53.313547758 +0000 UTC m=+3033.008755750" Oct 30 20:21:53.542461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:53.542366 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:53.889359 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:53.889050 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:54.304700 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:54.304662 5091 generic.go:356] "Generic (PLEG): container finished" podID="69437371-bf5b-4b40-9b57-cae3356c388a" containerID="4def0611513e1a6fcb6e8ad610ac9509f89fd9fd72b5980a50cdc0972341c500" exitCode=0 Oct 30 20:21:54.305238 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:54.304752 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" event={"ID":"69437371-bf5b-4b40-9b57-cae3356c388a","Type":"ContainerDied","Data":"4def0611513e1a6fcb6e8ad610ac9509f89fd9fd72b5980a50cdc0972341c500"} Oct 30 20:21:54.442729 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:54.442693 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86f4b503-4429-441a-b6fa-d547b87a08fa" path="/var/lib/kubelet/pods/86f4b503-4429-441a-b6fa-d547b87a08fa/volumes" Oct 30 20:21:54.777232 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:54.770546 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:55.496387 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:55.496358 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" Oct 30 20:21:55.612504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:55.612421 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69437371-bf5b-4b40-9b57-cae3356c388a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69437371-bf5b-4b40-9b57-cae3356c388a-maas-api-xks-ca\") pod \"69437371-bf5b-4b40-9b57-cae3356c388a\" (UID: \"69437371-bf5b-4b40-9b57-cae3356c388a\") " Oct 30 20:21:55.612673 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:55.612567 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69437371-bf5b-4b40-9b57-cae3356c388a-kube-api-access-r8ss5\" (UniqueName: \"kubernetes.io/projected/69437371-bf5b-4b40-9b57-cae3356c388a-kube-api-access-r8ss5\") pod \"69437371-bf5b-4b40-9b57-cae3356c388a\" (UID: \"69437371-bf5b-4b40-9b57-cae3356c388a\") " Oct 30 20:21:55.612673 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:55.612599 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69437371-bf5b-4b40-9b57-cae3356c388a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69437371-bf5b-4b40-9b57-cae3356c388a-maas-api-openshift-ca\") pod \"69437371-bf5b-4b40-9b57-cae3356c388a\" (UID: \"69437371-bf5b-4b40-9b57-cae3356c388a\") " Oct 30 20:21:55.613015 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:55.612992 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69437371-bf5b-4b40-9b57-cae3356c388a-maas-api-openshift-ca" pod "69437371-bf5b-4b40-9b57-cae3356c388a" (UID: "69437371-bf5b-4b40-9b57-cae3356c388a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:55.613206 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:55.613188 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69437371-bf5b-4b40-9b57-cae3356c388a-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:55.614581 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:55.614552 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69437371-bf5b-4b40-9b57-cae3356c388a-maas-api-xks-ca" pod "69437371-bf5b-4b40-9b57-cae3356c388a" (UID: "69437371-bf5b-4b40-9b57-cae3356c388a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:21:55.614776 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:55.614751 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69437371-bf5b-4b40-9b57-cae3356c388a-kube-api-access-r8ss5" pod "69437371-bf5b-4b40-9b57-cae3356c388a" (UID: "69437371-bf5b-4b40-9b57-cae3356c388a"). InnerVolumeSpecName "kube-api-access-r8ss5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:55.696084 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:55.695628 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:55.714571 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:55.714537 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8ss5\" (UniqueName: \"kubernetes.io/projected/69437371-bf5b-4b40-9b57-cae3356c388a-kube-api-access-r8ss5\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:55.714571 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:55.714571 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69437371-bf5b-4b40-9b57-cae3356c388a-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:56.322494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.322463 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" Oct 30 20:21:56.322685 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.322500 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw" event={"ID":"69437371-bf5b-4b40-9b57-cae3356c388a","Type":"ContainerDied","Data":"054d520383c7a72ed652f43d886e3f93f1f909093b6464392cdfa1da19d20990"} Oct 30 20:21:56.322685 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.322550 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="054d520383c7a72ed652f43d886e3f93f1f909093b6464392cdfa1da19d20990" Oct 30 20:21:56.592727 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.589184 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:56.811272 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.811247 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:21:56.929944 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.929857 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-istio-data\") pod \"43e01282-cbf5-43e3-9815-32c7d4439800\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " Oct 30 20:21:56.930098 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.929958 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-credential-socket\") pod \"43e01282-cbf5-43e3-9815-32c7d4439800\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " Oct 30 20:21:56.930098 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930013 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e01282-cbf5-43e3-9815-32c7d4439800-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e01282-cbf5-43e3-9815-32c7d4439800-istiod-ca-cert\") pod \"43e01282-cbf5-43e3-9815-32c7d4439800\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " Oct 30 20:21:56.930227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930093 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-workload-certs\") pod \"43e01282-cbf5-43e3-9815-32c7d4439800\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " Oct 30 20:21:56.930227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930140 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-workload-socket\") pod \"43e01282-cbf5-43e3-9815-32c7d4439800\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " Oct 30 20:21:56.930312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930211 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-istio-data" pod "43e01282-cbf5-43e3-9815-32c7d4439800" (UID: "43e01282-cbf5-43e3-9815-32c7d4439800"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:56.930312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930247 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e01282-cbf5-43e3-9815-32c7d4439800-istio-token\" (UniqueName: \"kubernetes.io/projected/43e01282-cbf5-43e3-9815-32c7d4439800-istio-token\") pod \"43e01282-cbf5-43e3-9815-32c7d4439800\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " Oct 30 20:21:56.930312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930281 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-credential-socket" pod "43e01282-cbf5-43e3-9815-32c7d4439800" (UID: "43e01282-cbf5-43e3-9815-32c7d4439800"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:56.930432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930345 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43e01282-cbf5-43e3-9815-32c7d4439800-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e01282-cbf5-43e3-9815-32c7d4439800-istio-podinfo\") pod \"43e01282-cbf5-43e3-9815-32c7d4439800\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " Oct 30 20:21:56.930432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930389 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-workload-certs" pod "43e01282-cbf5-43e3-9815-32c7d4439800" (UID: "43e01282-cbf5-43e3-9815-32c7d4439800"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:56.930432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930409 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e01282-cbf5-43e3-9815-32c7d4439800-kube-api-access-g7zfr\" (UniqueName: \"kubernetes.io/projected/43e01282-cbf5-43e3-9815-32c7d4439800-kube-api-access-g7zfr\") pod \"43e01282-cbf5-43e3-9815-32c7d4439800\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " Oct 30 20:21:56.930432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930422 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-workload-socket" pod "43e01282-cbf5-43e3-9815-32c7d4439800" (UID: "43e01282-cbf5-43e3-9815-32c7d4439800"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:56.930652 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930464 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-istio-envoy\") pod \"43e01282-cbf5-43e3-9815-32c7d4439800\" (UID: \"43e01282-cbf5-43e3-9815-32c7d4439800\") " Oct 30 20:21:56.930652 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930467 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e01282-cbf5-43e3-9815-32c7d4439800-istiod-ca-cert" pod "43e01282-cbf5-43e3-9815-32c7d4439800" (UID: "43e01282-cbf5-43e3-9815-32c7d4439800"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:21:56.930956 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930936 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:56.931047 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930963 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:56.931047 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930979 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43e01282-cbf5-43e3-9815-32c7d4439800-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:56.931047 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.930993 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:56.931047 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.931006 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:56.932671 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.932643 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43e01282-cbf5-43e3-9815-32c7d4439800-istio-podinfo" pod "43e01282-cbf5-43e3-9815-32c7d4439800" (UID: "43e01282-cbf5-43e3-9815-32c7d4439800"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:21:56.932798 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.932664 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e01282-cbf5-43e3-9815-32c7d4439800-istio-token" pod "43e01282-cbf5-43e3-9815-32c7d4439800" (UID: "43e01282-cbf5-43e3-9815-32c7d4439800"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:56.932798 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.932688 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-istio-envoy" pod "43e01282-cbf5-43e3-9815-32c7d4439800" (UID: "43e01282-cbf5-43e3-9815-32c7d4439800"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:21:56.932918 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:56.932847 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e01282-cbf5-43e3-9815-32c7d4439800-kube-api-access-g7zfr" pod "43e01282-cbf5-43e3-9815-32c7d4439800" (UID: "43e01282-cbf5-43e3-9815-32c7d4439800"). InnerVolumeSpecName "kube-api-access-g7zfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:21:57.031896 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.031860 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7zfr\" (UniqueName: \"kubernetes.io/projected/43e01282-cbf5-43e3-9815-32c7d4439800-kube-api-access-g7zfr\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:57.031896 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.031895 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43e01282-cbf5-43e3-9815-32c7d4439800-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:57.032125 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.031905 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43e01282-cbf5-43e3-9815-32c7d4439800-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:57.032125 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.031915 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43e01282-cbf5-43e3-9815-32c7d4439800-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:21:57.332455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.332420 5091 generic.go:356] "Generic (PLEG): container finished" podID="43e01282-cbf5-43e3-9815-32c7d4439800" containerID="08cb1dcb463b7df7783bb8e40545b717c07a5b7956f0378b2158ee057d8967ec" exitCode=0 Oct 30 20:21:57.332649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.332471 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" Oct 30 20:21:57.332649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.332501 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" event={"ID":"43e01282-cbf5-43e3-9815-32c7d4439800","Type":"ContainerDied","Data":"08cb1dcb463b7df7783bb8e40545b717c07a5b7956f0378b2158ee057d8967ec"} Oct 30 20:21:57.332649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.332556 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq" event={"ID":"43e01282-cbf5-43e3-9815-32c7d4439800","Type":"ContainerDied","Data":"cf8a8481bda03aec270ddfafede3d046ed2a242f8aa77f7d76a5d10d1888d254"} Oct 30 20:21:57.332649 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.332578 5091 scope.go:122] "RemoveContainer" containerID="08cb1dcb463b7df7783bb8e40545b717c07a5b7956f0378b2158ee057d8967ec" Oct 30 20:21:57.356433 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.356409 5091 scope.go:122] "RemoveContainer" containerID="51dd32bbba52c2223ad6dbbffa3b462b1068e4d77a4c2b50b458cd5b7a6ed92a" Oct 30 20:21:57.369626 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.369601 5091 scope.go:122] "RemoveContainer" containerID="08cb1dcb463b7df7783bb8e40545b717c07a5b7956f0378b2158ee057d8967ec" Oct 30 20:21:57.369924 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:21:57.369895 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08cb1dcb463b7df7783bb8e40545b717c07a5b7956f0378b2158ee057d8967ec\": container with ID starting with 08cb1dcb463b7df7783bb8e40545b717c07a5b7956f0378b2158ee057d8967ec not found: ID does not exist" containerID="08cb1dcb463b7df7783bb8e40545b717c07a5b7956f0378b2158ee057d8967ec" Oct 30 20:21:57.369997 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.369935 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08cb1dcb463b7df7783bb8e40545b717c07a5b7956f0378b2158ee057d8967ec"} err="failed to get container status \"08cb1dcb463b7df7783bb8e40545b717c07a5b7956f0378b2158ee057d8967ec\": rpc error: code = NotFound desc = could not find container \"08cb1dcb463b7df7783bb8e40545b717c07a5b7956f0378b2158ee057d8967ec\": container with ID starting with 08cb1dcb463b7df7783bb8e40545b717c07a5b7956f0378b2158ee057d8967ec not found: ID does not exist" Oct 30 20:21:57.369997 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.369959 5091 scope.go:122] "RemoveContainer" containerID="51dd32bbba52c2223ad6dbbffa3b462b1068e4d77a4c2b50b458cd5b7a6ed92a" Oct 30 20:21:57.370269 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:21:57.370247 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51dd32bbba52c2223ad6dbbffa3b462b1068e4d77a4c2b50b458cd5b7a6ed92a\": container with ID starting with 51dd32bbba52c2223ad6dbbffa3b462b1068e4d77a4c2b50b458cd5b7a6ed92a not found: ID does not exist" containerID="51dd32bbba52c2223ad6dbbffa3b462b1068e4d77a4c2b50b458cd5b7a6ed92a" Oct 30 20:21:57.370367 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.370276 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51dd32bbba52c2223ad6dbbffa3b462b1068e4d77a4c2b50b458cd5b7a6ed92a"} err="failed to get container status \"51dd32bbba52c2223ad6dbbffa3b462b1068e4d77a4c2b50b458cd5b7a6ed92a\": rpc error: code = NotFound desc = could not find container \"51dd32bbba52c2223ad6dbbffa3b462b1068e4d77a4c2b50b458cd5b7a6ed92a\": container with ID starting with 51dd32bbba52c2223ad6dbbffa3b462b1068e4d77a4c2b50b458cd5b7a6ed92a not found: ID does not exist" Oct 30 20:21:57.389998 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.389971 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq"] Oct 30 20:21:57.436530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.436498 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-517027-openshift-default-8456974bb5-6p6rq"] Oct 30 20:21:57.464210 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:57.464172 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:58.442618 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:58.442583 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e01282-cbf5-43e3-9815-32c7d4439800" path="/var/lib/kubelet/pods/43e01282-cbf5-43e3-9815-32c7d4439800/volumes" Oct 30 20:21:58.545915 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:58.545878 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:21:59.595176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:21:59.594092 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:00.146340 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.146279 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:00.291598 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.291557 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:00.420551 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.420465 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww"] Oct 30 20:22:00.420759 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.420724 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" podUID="a6f27a0f-15c8-4266-b1ea-ced8d23c985b" containerName="istio-proxy" containerID="cri-o://309a8d7107e3b7b2dc23f168ba3f1cb62096e1692403c34613c7c9959c67b878" gracePeriod=30 Oct 30 20:22:00.456862 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.456829 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:00.604728 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.604692 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:00.957439 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.957409 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" Oct 30 20:22:00.970876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.970810 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj"] Oct 30 20:22:00.971777 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.971755 5091 cpu_manager.go:414] "containerMap: removing container" podUID="43e01282-cbf5-43e3-9815-32c7d4439800" containerName="istio-proxy" Oct 30 20:22:00.971871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.971781 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e01282-cbf5-43e3-9815-32c7d4439800" containerName="istio-proxy" Oct 30 20:22:00.971871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.971801 5091 cpu_manager.go:414] "containerMap: removing container" podUID="69437371-bf5b-4b40-9b57-cae3356c388a" containerName="revoke-keys" Oct 30 20:22:00.971871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.971808 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69437371-bf5b-4b40-9b57-cae3356c388a" containerName="revoke-keys" Oct 30 20:22:00.971871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.971830 5091 cpu_manager.go:414] "containerMap: removing container" podUID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" containerName="istio-proxy" Oct 30 20:22:00.971871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.971837 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="320dc7ae-3660-41d1-bc53-49d3427a0b6d" containerName="istio-proxy" Oct 30 20:22:00.971871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.971853 5091 cpu_manager.go:414] "containerMap: removing container" podUID="0b1b31d4-eafb-4f36-922a-c7fdcc8c007f" containerName="payload-processing" Oct 30 20:22:00.971871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.971862 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b1b31d4-eafb-4f36-922a-c7fdcc8c007f" containerName="payload-processing" Oct 30 20:22:00.972103 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.972039 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e01282-cbf5-43e3-9815-32c7d4439800" containerName="istio-proxy" Oct 30 20:22:00.972103 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.972052 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="69437371-bf5b-4b40-9b57-cae3356c388a" containerName="revoke-keys" Oct 30 20:22:00.972103 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.972066 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b1b31d4-eafb-4f36-922a-c7fdcc8c007f" containerName="payload-processing" Oct 30 20:22:00.976509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.976487 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" Oct 30 20:22:00.984182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.984122 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:22:00.984448 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.984430 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj"] Oct 30 20:22:00.989225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:00.989202 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:22:01.017380 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.017351 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" Oct 30 20:22:01.112684 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.112651 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b1b31d4-eafb-4f36-922a-c7fdcc8c007f-kube-api-access-kh8ld\" (UniqueName: \"kubernetes.io/projected/0b1b31d4-eafb-4f36-922a-c7fdcc8c007f-kube-api-access-kh8ld\") pod \"0b1b31d4-eafb-4f36-922a-c7fdcc8c007f\" (UID: \"0b1b31d4-eafb-4f36-922a-c7fdcc8c007f\") " Oct 30 20:22:01.112872 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.112711 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b1b31d4-eafb-4f36-922a-c7fdcc8c007f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1b31d4-eafb-4f36-922a-c7fdcc8c007f-plugins-config-volume\") pod \"0b1b31d4-eafb-4f36-922a-c7fdcc8c007f\" (UID: \"0b1b31d4-eafb-4f36-922a-c7fdcc8c007f\") " Oct 30 20:22:01.113208 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.113186 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b1b31d4-eafb-4f36-922a-c7fdcc8c007f-plugins-config-volume" pod "0b1b31d4-eafb-4f36-922a-c7fdcc8c007f" (UID: "0b1b31d4-eafb-4f36-922a-c7fdcc8c007f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:01.113258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.113218 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvdwg\" (UniqueName: \"kubernetes.io/projected/d30db8d5-5bfd-4e2e-add3-aa61588dc165-kube-api-access-rvdwg\") pod \"maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj\" (UID: \"d30db8d5-5bfd-4e2e-add3-aa61588dc165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" Oct 30 20:22:01.113258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.113251 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30db8d5-5bfd-4e2e-add3-aa61588dc165-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj\" (UID: \"d30db8d5-5bfd-4e2e-add3-aa61588dc165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" Oct 30 20:22:01.113359 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.113285 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30db8d5-5bfd-4e2e-add3-aa61588dc165-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj\" (UID: \"d30db8d5-5bfd-4e2e-add3-aa61588dc165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" Oct 30 20:22:01.113504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.113477 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b1b31d4-eafb-4f36-922a-c7fdcc8c007f-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:01.114850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.114830 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b1b31d4-eafb-4f36-922a-c7fdcc8c007f-kube-api-access-kh8ld" pod "0b1b31d4-eafb-4f36-922a-c7fdcc8c007f" (UID: "0b1b31d4-eafb-4f36-922a-c7fdcc8c007f"). InnerVolumeSpecName "kube-api-access-kh8ld". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:01.214966 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.214885 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa98f754-341a-426e-91cf-1c914b3a2433-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa98f754-341a-426e-91cf-1c914b3a2433-plugins-config-volume\") pod \"fa98f754-341a-426e-91cf-1c914b3a2433\" (UID: \"fa98f754-341a-426e-91cf-1c914b3a2433\") " Oct 30 20:22:01.215106 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.215040 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa98f754-341a-426e-91cf-1c914b3a2433-kube-api-access-h86dr\" (UniqueName: \"kubernetes.io/projected/fa98f754-341a-426e-91cf-1c914b3a2433-kube-api-access-h86dr\") pod \"fa98f754-341a-426e-91cf-1c914b3a2433\" (UID: \"fa98f754-341a-426e-91cf-1c914b3a2433\") " Oct 30 20:22:01.215277 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.215254 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa98f754-341a-426e-91cf-1c914b3a2433-plugins-config-volume" pod "fa98f754-341a-426e-91cf-1c914b3a2433" (UID: "fa98f754-341a-426e-91cf-1c914b3a2433"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:01.215324 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.215294 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvdwg\" (UniqueName: \"kubernetes.io/projected/d30db8d5-5bfd-4e2e-add3-aa61588dc165-kube-api-access-rvdwg\") pod \"maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj\" (UID: \"d30db8d5-5bfd-4e2e-add3-aa61588dc165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" Oct 30 20:22:01.215360 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.215322 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30db8d5-5bfd-4e2e-add3-aa61588dc165-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj\" (UID: \"d30db8d5-5bfd-4e2e-add3-aa61588dc165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" Oct 30 20:22:01.215360 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.215350 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30db8d5-5bfd-4e2e-add3-aa61588dc165-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj\" (UID: \"d30db8d5-5bfd-4e2e-add3-aa61588dc165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" Oct 30 20:22:01.215469 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.215456 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh8ld\" (UniqueName: \"kubernetes.io/projected/0b1b31d4-eafb-4f36-922a-c7fdcc8c007f-kube-api-access-kh8ld\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:01.215505 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.215474 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa98f754-341a-426e-91cf-1c914b3a2433-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:01.216079 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.216052 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30db8d5-5bfd-4e2e-add3-aa61588dc165-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj\" (UID: \"d30db8d5-5bfd-4e2e-add3-aa61588dc165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" Oct 30 20:22:01.217329 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.217302 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa98f754-341a-426e-91cf-1c914b3a2433-kube-api-access-h86dr" pod "fa98f754-341a-426e-91cf-1c914b3a2433" (UID: "fa98f754-341a-426e-91cf-1c914b3a2433"). InnerVolumeSpecName "kube-api-access-h86dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:01.217639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.217617 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30db8d5-5bfd-4e2e-add3-aa61588dc165-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj\" (UID: \"d30db8d5-5bfd-4e2e-add3-aa61588dc165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" Oct 30 20:22:01.229398 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.229376 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvdwg\" (UniqueName: \"kubernetes.io/projected/d30db8d5-5bfd-4e2e-add3-aa61588dc165-kube-api-access-rvdwg\") pod \"maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj\" (UID: \"d30db8d5-5bfd-4e2e-add3-aa61588dc165\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" Oct 30 20:22:01.316516 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.316467 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" Oct 30 20:22:01.317021 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.316996 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h86dr\" (UniqueName: \"kubernetes.io/projected/fa98f754-341a-426e-91cf-1c914b3a2433-kube-api-access-h86dr\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:01.408173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.407835 5091 generic.go:356] "Generic (PLEG): container finished" podID="fa98f754-341a-426e-91cf-1c914b3a2433" containerID="7301ad28de5127e792170f28099af85f9db7cff71f9a19a0a0ec9ddf24771a58" exitCode=0 Oct 30 20:22:01.408173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.407995 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" event={"ID":"fa98f754-341a-426e-91cf-1c914b3a2433","Type":"ContainerDied","Data":"7301ad28de5127e792170f28099af85f9db7cff71f9a19a0a0ec9ddf24771a58"} Oct 30 20:22:01.408173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.408041 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" event={"ID":"fa98f754-341a-426e-91cf-1c914b3a2433","Type":"ContainerDied","Data":"140fae30d542744c3f0d8332b4da592b2ea8d7bf62b0f626aa3311b307f4950c"} Oct 30 20:22:01.408173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.408072 5091 scope.go:122] "RemoveContainer" containerID="7301ad28de5127e792170f28099af85f9db7cff71f9a19a0a0ec9ddf24771a58" Oct 30 20:22:01.408173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.408182 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs" Oct 30 20:22:01.413325 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.412232 5091 generic.go:356] "Generic (PLEG): container finished" podID="0b1b31d4-eafb-4f36-922a-c7fdcc8c007f" containerID="8a56a4980659f1dce4e6a96661e21c43e4226e290ebdfd016b44af1132cf2ab4" exitCode=0 Oct 30 20:22:01.413325 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.412412 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" event={"ID":"0b1b31d4-eafb-4f36-922a-c7fdcc8c007f","Type":"ContainerDied","Data":"8a56a4980659f1dce4e6a96661e21c43e4226e290ebdfd016b44af1132cf2ab4"} Oct 30 20:22:01.413325 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.412461 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" event={"ID":"0b1b31d4-eafb-4f36-922a-c7fdcc8c007f","Type":"ContainerDied","Data":"7a15fe607751af3f148f41c71a9b4c6358ea4dad37a476a7034591f1b38b65ae"} Oct 30 20:22:01.413325 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.412517 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p" Oct 30 20:22:01.430170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.429999 5091 scope.go:122] "RemoveContainer" containerID="7301ad28de5127e792170f28099af85f9db7cff71f9a19a0a0ec9ddf24771a58" Oct 30 20:22:01.430504 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:22:01.430451 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7301ad28de5127e792170f28099af85f9db7cff71f9a19a0a0ec9ddf24771a58\": container with ID starting with 7301ad28de5127e792170f28099af85f9db7cff71f9a19a0a0ec9ddf24771a58 not found: ID does not exist" containerID="7301ad28de5127e792170f28099af85f9db7cff71f9a19a0a0ec9ddf24771a58" Oct 30 20:22:01.430627 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.430513 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7301ad28de5127e792170f28099af85f9db7cff71f9a19a0a0ec9ddf24771a58"} err="failed to get container status \"7301ad28de5127e792170f28099af85f9db7cff71f9a19a0a0ec9ddf24771a58\": rpc error: code = NotFound desc = could not find container \"7301ad28de5127e792170f28099af85f9db7cff71f9a19a0a0ec9ddf24771a58\": container with ID starting with 7301ad28de5127e792170f28099af85f9db7cff71f9a19a0a0ec9ddf24771a58 not found: ID does not exist" Oct 30 20:22:01.430627 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.430539 5091 scope.go:122] "RemoveContainer" containerID="8a56a4980659f1dce4e6a96661e21c43e4226e290ebdfd016b44af1132cf2ab4" Oct 30 20:22:01.447002 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.446738 5091 scope.go:122] "RemoveContainer" containerID="8a56a4980659f1dce4e6a96661e21c43e4226e290ebdfd016b44af1132cf2ab4" Oct 30 20:22:01.447340 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:22:01.447316 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a56a4980659f1dce4e6a96661e21c43e4226e290ebdfd016b44af1132cf2ab4\": container with ID starting with 8a56a4980659f1dce4e6a96661e21c43e4226e290ebdfd016b44af1132cf2ab4 not found: ID does not exist" containerID="8a56a4980659f1dce4e6a96661e21c43e4226e290ebdfd016b44af1132cf2ab4" Oct 30 20:22:01.447441 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.447351 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a56a4980659f1dce4e6a96661e21c43e4226e290ebdfd016b44af1132cf2ab4"} err="failed to get container status \"8a56a4980659f1dce4e6a96661e21c43e4226e290ebdfd016b44af1132cf2ab4\": rpc error: code = NotFound desc = could not find container \"8a56a4980659f1dce4e6a96661e21c43e4226e290ebdfd016b44af1132cf2ab4\": container with ID starting with 8a56a4980659f1dce4e6a96661e21c43e4226e290ebdfd016b44af1132cf2ab4 not found: ID does not exist" Oct 30 20:22:01.450784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.450759 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p"] Oct 30 20:22:01.456444 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.456421 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-517027-869b54f744-6sn2p"] Oct 30 20:22:01.471825 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.471757 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs"] Oct 30 20:22:01.477425 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.477394 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-517027-587c5945f5-qgfvs"] Oct 30 20:22:01.515007 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.514319 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:01.612964 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:22:01.612932 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd30db8d5_5bfd_4e2e_add3_aa61588dc165.slice/crio-dd3538898ba3f5ac996fed280915a943242bfb434295dd5bda579f2188df5c88 WatchSource:0}: Error finding container dd3538898ba3f5ac996fed280915a943242bfb434295dd5bda579f2188df5c88: Status 404 returned error can't find the container with id dd3538898ba3f5ac996fed280915a943242bfb434295dd5bda579f2188df5c88 Oct 30 20:22:01.613360 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:01.613131 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj"] Oct 30 20:22:02.033330 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:02.032284 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf"] Oct 30 20:22:02.036243 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:02.036216 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2235aa4a-d2zsf"] Oct 30 20:22:02.173399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:02.173366 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth"] Oct 30 20:22:02.283454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:02.283326 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:02.492221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:02.481434 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:02.508880 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:02.503031 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b1b31d4-eafb-4f36-922a-c7fdcc8c007f" path="/var/lib/kubelet/pods/0b1b31d4-eafb-4f36-922a-c7fdcc8c007f/volumes" Oct 30 20:22:02.508880 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:02.503702 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="865fe459-6cb9-40c2-9afd-835e90619526" path="/var/lib/kubelet/pods/865fe459-6cb9-40c2-9afd-835e90619526/volumes" Oct 30 20:22:02.508880 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:02.508321 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa98f754-341a-426e-91cf-1c914b3a2433" path="/var/lib/kubelet/pods/fa98f754-341a-426e-91cf-1c914b3a2433/volumes" Oct 30 20:22:02.515540 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:02.514530 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" event={"ID":"d30db8d5-5bfd-4e2e-add3-aa61588dc165","Type":"ContainerStarted","Data":"0b86894c012a44ad2c251960fc78bb2caebf7148d80fb8782fec2ae4928b10c3"} Oct 30 20:22:02.515540 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:02.514603 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" event={"ID":"d30db8d5-5bfd-4e2e-add3-aa61588dc165","Type":"ContainerStarted","Data":"dd3538898ba3f5ac996fed280915a943242bfb434295dd5bda579f2188df5c88"} Oct 30 20:22:02.536793 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:02.536700 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" podStartSLOduration=2.5366875760000003 podStartE2EDuration="2.536687576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:02.535335104 +0000 UTC m=+3042.230543064" watchObservedRunningTime="2026-10-30 20:22:02.536687576 +0000 UTC m=+3042.231895540" Oct 30 20:22:02.695194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:02.677164 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:03.526159 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:03.526111 5091 generic.go:356] "Generic (PLEG): container finished" podID="d30db8d5-5bfd-4e2e-add3-aa61588dc165" containerID="0b86894c012a44ad2c251960fc78bb2caebf7148d80fb8782fec2ae4928b10c3" exitCode=0 Oct 30 20:22:03.526368 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:03.526185 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" event={"ID":"d30db8d5-5bfd-4e2e-add3-aa61588dc165","Type":"ContainerDied","Data":"0b86894c012a44ad2c251960fc78bb2caebf7148d80fb8782fec2ae4928b10c3"} Oct 30 20:22:03.590625 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:03.590578 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:04.443528 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:04.443484 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:04.645269 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:04.645223 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:04.709937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:04.709846 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw"] Oct 30 20:22:04.715207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:04.715173 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-5f9b3c-zq9rw"] Oct 30 20:22:04.787436 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:04.787408 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" Oct 30 20:22:04.826888 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:04.826834 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:04.886507 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:04.886476 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d30db8d5-5bfd-4e2e-add3-aa61588dc165-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30db8d5-5bfd-4e2e-add3-aa61588dc165-maas-api-openshift-ca\") pod \"d30db8d5-5bfd-4e2e-add3-aa61588dc165\" (UID: \"d30db8d5-5bfd-4e2e-add3-aa61588dc165\") " Oct 30 20:22:04.886726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:04.886535 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d30db8d5-5bfd-4e2e-add3-aa61588dc165-kube-api-access-rvdwg\" (UniqueName: \"kubernetes.io/projected/d30db8d5-5bfd-4e2e-add3-aa61588dc165-kube-api-access-rvdwg\") pod \"d30db8d5-5bfd-4e2e-add3-aa61588dc165\" (UID: \"d30db8d5-5bfd-4e2e-add3-aa61588dc165\") " Oct 30 20:22:04.886726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:04.886604 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d30db8d5-5bfd-4e2e-add3-aa61588dc165-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30db8d5-5bfd-4e2e-add3-aa61588dc165-maas-api-xks-ca\") pod \"d30db8d5-5bfd-4e2e-add3-aa61588dc165\" (UID: \"d30db8d5-5bfd-4e2e-add3-aa61588dc165\") " Oct 30 20:22:04.886930 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:04.886904 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d30db8d5-5bfd-4e2e-add3-aa61588dc165-maas-api-openshift-ca" pod "d30db8d5-5bfd-4e2e-add3-aa61588dc165" (UID: "d30db8d5-5bfd-4e2e-add3-aa61588dc165"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:04.887224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:04.887203 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d30db8d5-5bfd-4e2e-add3-aa61588dc165-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:04.888815 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:04.888786 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d30db8d5-5bfd-4e2e-add3-aa61588dc165-maas-api-xks-ca" pod "d30db8d5-5bfd-4e2e-add3-aa61588dc165" (UID: "d30db8d5-5bfd-4e2e-add3-aa61588dc165"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:22:04.888940 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:04.888898 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d30db8d5-5bfd-4e2e-add3-aa61588dc165-kube-api-access-rvdwg" pod "d30db8d5-5bfd-4e2e-add3-aa61588dc165" (UID: "d30db8d5-5bfd-4e2e-add3-aa61588dc165"). InnerVolumeSpecName "kube-api-access-rvdwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:04.988389 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:04.988311 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvdwg\" (UniqueName: \"kubernetes.io/projected/d30db8d5-5bfd-4e2e-add3-aa61588dc165-kube-api-access-rvdwg\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:04.988389 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:04.988340 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d30db8d5-5bfd-4e2e-add3-aa61588dc165-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:05.238288 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.238249 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:05.546851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.546821 5091 generic.go:356] "Generic (PLEG): container finished" podID="a6f27a0f-15c8-4266-b1ea-ced8d23c985b" containerID="309a8d7107e3b7b2dc23f168ba3f1cb62096e1692403c34613c7c9959c67b878" exitCode=0 Oct 30 20:22:05.547231 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.546898 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" event={"ID":"a6f27a0f-15c8-4266-b1ea-ced8d23c985b","Type":"ContainerDied","Data":"309a8d7107e3b7b2dc23f168ba3f1cb62096e1692403c34613c7c9959c67b878"} Oct 30 20:22:05.547231 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.546968 5091 scope.go:122] "RemoveContainer" containerID="d985f317ad1a4318547f90ccecf7923b1d7a892a12ff5e2bc36ecabb6b47b9ce" Oct 30 20:22:05.556178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.548703 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" Oct 30 20:22:05.556178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.548728 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj" event={"ID":"d30db8d5-5bfd-4e2e-add3-aa61588dc165","Type":"ContainerDied","Data":"dd3538898ba3f5ac996fed280915a943242bfb434295dd5bda579f2188df5c88"} Oct 30 20:22:05.556178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.548777 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd3538898ba3f5ac996fed280915a943242bfb434295dd5bda579f2188df5c88" Oct 30 20:22:05.686823 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.686745 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:22:05.805009 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.804979 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-podinfo\") pod \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " Oct 30 20:22:05.805177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.805024 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-workload-certs\") pod \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " Oct 30 20:22:05.805177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.805097 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-token\" (UniqueName: \"kubernetes.io/projected/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-token\") pod \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " Oct 30 20:22:05.805282 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.805269 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-credential-socket\") pod \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " Oct 30 20:22:05.805322 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.805281 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-workload-certs" pod "a6f27a0f-15c8-4266-b1ea-ced8d23c985b" (UID: "a6f27a0f-15c8-4266-b1ea-ced8d23c985b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:05.805389 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.805335 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-workload-socket\") pod \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " Oct 30 20:22:05.805454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.805419 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-kube-api-access-t4hjr\" (UniqueName: \"kubernetes.io/projected/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-kube-api-access-t4hjr\") pod \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " Oct 30 20:22:05.805520 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.805498 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-envoy\") pod \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " Oct 30 20:22:05.805578 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.805555 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istiod-ca-cert\") pod \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " Oct 30 20:22:05.805641 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.805579 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-credential-socket" pod "a6f27a0f-15c8-4266-b1ea-ced8d23c985b" (UID: "a6f27a0f-15c8-4266-b1ea-ced8d23c985b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:05.805641 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.805590 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-workload-socket" pod "a6f27a0f-15c8-4266-b1ea-ced8d23c985b" (UID: "a6f27a0f-15c8-4266-b1ea-ced8d23c985b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:05.805641 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.805600 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-data\") pod \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\" (UID: \"a6f27a0f-15c8-4266-b1ea-ced8d23c985b\") " Oct 30 20:22:05.805869 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.805848 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-data" pod "a6f27a0f-15c8-4266-b1ea-ced8d23c985b" (UID: "a6f27a0f-15c8-4266-b1ea-ced8d23c985b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:05.805978 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.805945 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istiod-ca-cert" pod "a6f27a0f-15c8-4266-b1ea-ced8d23c985b" (UID: "a6f27a0f-15c8-4266-b1ea-ced8d23c985b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:05.806255 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.806200 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:05.806255 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.806228 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:05.806255 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.806246 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:05.806499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.806262 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:05.806499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.806276 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:05.807717 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.807690 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-envoy" pod "a6f27a0f-15c8-4266-b1ea-ced8d23c985b" (UID: "a6f27a0f-15c8-4266-b1ea-ced8d23c985b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:05.807825 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.807730 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-token" pod "a6f27a0f-15c8-4266-b1ea-ced8d23c985b" (UID: "a6f27a0f-15c8-4266-b1ea-ced8d23c985b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:05.807894 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.807862 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-podinfo" pod "a6f27a0f-15c8-4266-b1ea-ced8d23c985b" (UID: "a6f27a0f-15c8-4266-b1ea-ced8d23c985b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:22:05.807894 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.807881 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-kube-api-access-t4hjr" pod "a6f27a0f-15c8-4266-b1ea-ced8d23c985b" (UID: "a6f27a0f-15c8-4266-b1ea-ced8d23c985b"). InnerVolumeSpecName "kube-api-access-t4hjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:05.863416 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.862013 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:05.907854 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.907815 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4hjr\" (UniqueName: \"kubernetes.io/projected/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-kube-api-access-t4hjr\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:05.908016 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.907856 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:05.908016 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.907890 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:05.908016 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:05.907906 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6f27a0f-15c8-4266-b1ea-ced8d23c985b-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:06.442324 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:06.442288 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69437371-bf5b-4b40-9b57-cae3356c388a" path="/var/lib/kubelet/pods/69437371-bf5b-4b40-9b57-cae3356c388a/volumes" Oct 30 20:22:06.561039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:06.561008 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" Oct 30 20:22:06.561586 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:06.561002 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww" event={"ID":"a6f27a0f-15c8-4266-b1ea-ced8d23c985b","Type":"ContainerDied","Data":"bb8dbf65a3aaf3df8998e57adf45ab303ee97d0d48f75a2e5f5010b345867977"} Oct 30 20:22:06.561586 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:06.561118 5091 scope.go:122] "RemoveContainer" containerID="309a8d7107e3b7b2dc23f168ba3f1cb62096e1692403c34613c7c9959c67b878" Oct 30 20:22:06.599280 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:06.599251 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww"] Oct 30 20:22:06.612856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:06.612822 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-bdf9e6-openshift-default-5bd8499f7-j6pww"] Oct 30 20:22:06.946429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:06.946385 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:07.174865 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.174824 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" podUID="f94b5b35-f3af-48c4-a865-c278844435ed" containerName="payload-pre-processing" containerID="cri-o://c4f9efa69db4ee330fdb10c325532668b2864c9840c29b24faa5dc967b1dcfee" gracePeriod=25 Oct 30 20:22:07.460737 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.460708 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" Oct 30 20:22:07.526972 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.526942 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94b5b35-f3af-48c4-a865-c278844435ed-kube-api-access-z4cpq\" (UniqueName: \"kubernetes.io/projected/f94b5b35-f3af-48c4-a865-c278844435ed-kube-api-access-z4cpq\") pod \"f94b5b35-f3af-48c4-a865-c278844435ed\" (UID: \"f94b5b35-f3af-48c4-a865-c278844435ed\") " Oct 30 20:22:07.527248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.527227 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f94b5b35-f3af-48c4-a865-c278844435ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94b5b35-f3af-48c4-a865-c278844435ed-plugins-config-volume\") pod \"f94b5b35-f3af-48c4-a865-c278844435ed\" (UID: \"f94b5b35-f3af-48c4-a865-c278844435ed\") " Oct 30 20:22:07.527711 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.527683 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94b5b35-f3af-48c4-a865-c278844435ed-plugins-config-volume" pod "f94b5b35-f3af-48c4-a865-c278844435ed" (UID: "f94b5b35-f3af-48c4-a865-c278844435ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:07.528046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.528022 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f94b5b35-f3af-48c4-a865-c278844435ed-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:07.530012 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.529982 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94b5b35-f3af-48c4-a865-c278844435ed-kube-api-access-z4cpq" pod "f94b5b35-f3af-48c4-a865-c278844435ed" (UID: "f94b5b35-f3af-48c4-a865-c278844435ed"). InnerVolumeSpecName "kube-api-access-z4cpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:07.574004 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.573926 5091 generic.go:356] "Generic (PLEG): container finished" podID="f94b5b35-f3af-48c4-a865-c278844435ed" containerID="c4f9efa69db4ee330fdb10c325532668b2864c9840c29b24faa5dc967b1dcfee" exitCode=0 Oct 30 20:22:07.574004 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.573982 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" Oct 30 20:22:07.574004 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.573988 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" event={"ID":"f94b5b35-f3af-48c4-a865-c278844435ed","Type":"ContainerDied","Data":"c4f9efa69db4ee330fdb10c325532668b2864c9840c29b24faa5dc967b1dcfee"} Oct 30 20:22:07.574504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.574053 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth" event={"ID":"f94b5b35-f3af-48c4-a865-c278844435ed","Type":"ContainerDied","Data":"bb0417b775dfd886fabc98e992b56eebff166e4eaae52deb8eb319e8ed7a922b"} Oct 30 20:22:07.574504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.574085 5091 scope.go:122] "RemoveContainer" containerID="c4f9efa69db4ee330fdb10c325532668b2864c9840c29b24faa5dc967b1dcfee" Oct 30 20:22:07.589036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.589009 5091 scope.go:122] "RemoveContainer" containerID="c4f9efa69db4ee330fdb10c325532668b2864c9840c29b24faa5dc967b1dcfee" Oct 30 20:22:07.589582 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:22:07.589547 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f9efa69db4ee330fdb10c325532668b2864c9840c29b24faa5dc967b1dcfee\": container with ID starting with c4f9efa69db4ee330fdb10c325532668b2864c9840c29b24faa5dc967b1dcfee not found: ID does not exist" containerID="c4f9efa69db4ee330fdb10c325532668b2864c9840c29b24faa5dc967b1dcfee" Oct 30 20:22:07.589713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.589596 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f9efa69db4ee330fdb10c325532668b2864c9840c29b24faa5dc967b1dcfee"} err="failed to get container status \"c4f9efa69db4ee330fdb10c325532668b2864c9840c29b24faa5dc967b1dcfee\": rpc error: code = NotFound desc = could not find container \"c4f9efa69db4ee330fdb10c325532668b2864c9840c29b24faa5dc967b1dcfee\": container with ID starting with c4f9efa69db4ee330fdb10c325532668b2864c9840c29b24faa5dc967b1dcfee not found: ID does not exist" Oct 30 20:22:07.610878 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.610846 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth"] Oct 30 20:22:07.613900 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.613875 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-5f9b3c-5f484d6cff-swzth"] Oct 30 20:22:07.629310 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.629280 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4cpq\" (UniqueName: \"kubernetes.io/projected/f94b5b35-f3af-48c4-a865-c278844435ed-kube-api-access-z4cpq\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:07.868223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:07.859184 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:08.478491 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:08.476125 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6f27a0f-15c8-4266-b1ea-ced8d23c985b" path="/var/lib/kubelet/pods/a6f27a0f-15c8-4266-b1ea-ced8d23c985b/volumes" Oct 30 20:22:08.478491 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:08.476818 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f94b5b35-f3af-48c4-a865-c278844435ed" path="/var/lib/kubelet/pods/f94b5b35-f3af-48c4-a865-c278844435ed/volumes" Oct 30 20:22:08.747469 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:08.747421 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:09.575056 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:09.575008 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:10.365710 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:10.365670 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:11.152282 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:11.152244 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx"] Oct 30 20:22:11.167949 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:11.167915 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql"] Oct 30 20:22:11.277307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:11.277274 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:11.428989 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:11.428892 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:12.172676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:12.172629 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:13.084960 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:13.084910 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:13.330671 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:13.330632 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:13.470691 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:13.470597 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:13.665462 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:13.665432 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72"] Oct 30 20:22:13.665663 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:13.665637 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" containerName="istio-proxy" containerID="cri-o://c5db5edb55b9cbbd2fed3e99b4dd54eebbbc50d738d2bf0f882067ff447533c4" gracePeriod=30 Oct 30 20:22:13.793171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:13.792621 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:14.180844 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.180759 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.196136 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg"] Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197468 5091 cpu_manager.go:414] "containerMap: removing container" podUID="fa98f754-341a-426e-91cf-1c914b3a2433" containerName="payload-pre-processing" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197491 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa98f754-341a-426e-91cf-1c914b3a2433" containerName="payload-pre-processing" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197508 5091 cpu_manager.go:414] "containerMap: removing container" podUID="43e01282-cbf5-43e3-9815-32c7d4439800" containerName="istio-proxy" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197513 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e01282-cbf5-43e3-9815-32c7d4439800" containerName="istio-proxy" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197575 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f94b5b35-f3af-48c4-a865-c278844435ed" containerName="payload-pre-processing" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197582 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94b5b35-f3af-48c4-a865-c278844435ed" containerName="payload-pre-processing" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197612 5091 cpu_manager.go:414] "containerMap: removing container" podUID="d30db8d5-5bfd-4e2e-add3-aa61588dc165" containerName="revoke-keys" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197618 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d30db8d5-5bfd-4e2e-add3-aa61588dc165" containerName="revoke-keys" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197638 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a6f27a0f-15c8-4266-b1ea-ced8d23c985b" containerName="istio-proxy" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197643 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f27a0f-15c8-4266-b1ea-ced8d23c985b" containerName="istio-proxy" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197656 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a6f27a0f-15c8-4266-b1ea-ced8d23c985b" containerName="istio-proxy" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197662 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6f27a0f-15c8-4266-b1ea-ced8d23c985b" containerName="istio-proxy" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197837 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94b5b35-f3af-48c4-a865-c278844435ed" containerName="payload-pre-processing" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197853 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="d30db8d5-5bfd-4e2e-add3-aa61588dc165" containerName="revoke-keys" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197861 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f27a0f-15c8-4266-b1ea-ced8d23c985b" containerName="istio-proxy" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197875 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6f27a0f-15c8-4266-b1ea-ced8d23c985b" containerName="istio-proxy" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197887 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa98f754-341a-426e-91cf-1c914b3a2433" containerName="payload-pre-processing" Oct 30 20:22:14.209192 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.197902 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e01282-cbf5-43e3-9815-32c7d4439800" containerName="istio-proxy" Oct 30 20:22:14.211426 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.211384 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" Oct 30 20:22:14.212773 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.212744 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg"] Oct 30 20:22:14.224985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.224961 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:22:14.225104 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.225021 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:22:14.345563 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.345525 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcdvr\" (UniqueName: \"kubernetes.io/projected/ee85beb2-2c97-4885-bc8d-08a90b2a85de-kube-api-access-mcdvr\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg\" (UID: \"ee85beb2-2c97-4885-bc8d-08a90b2a85de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" Oct 30 20:22:14.345744 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.345586 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee85beb2-2c97-4885-bc8d-08a90b2a85de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg\" (UID: \"ee85beb2-2c97-4885-bc8d-08a90b2a85de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" Oct 30 20:22:14.345744 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.345675 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee85beb2-2c97-4885-bc8d-08a90b2a85de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg\" (UID: \"ee85beb2-2c97-4885-bc8d-08a90b2a85de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" Oct 30 20:22:14.446430 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.446334 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcdvr\" (UniqueName: \"kubernetes.io/projected/ee85beb2-2c97-4885-bc8d-08a90b2a85de-kube-api-access-mcdvr\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg\" (UID: \"ee85beb2-2c97-4885-bc8d-08a90b2a85de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" Oct 30 20:22:14.446430 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.446383 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee85beb2-2c97-4885-bc8d-08a90b2a85de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg\" (UID: \"ee85beb2-2c97-4885-bc8d-08a90b2a85de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" Oct 30 20:22:14.446430 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.446411 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee85beb2-2c97-4885-bc8d-08a90b2a85de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg\" (UID: \"ee85beb2-2c97-4885-bc8d-08a90b2a85de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" Oct 30 20:22:14.447069 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.447047 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee85beb2-2c97-4885-bc8d-08a90b2a85de-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg\" (UID: \"ee85beb2-2c97-4885-bc8d-08a90b2a85de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" Oct 30 20:22:14.449470 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.449438 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee85beb2-2c97-4885-bc8d-08a90b2a85de-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg\" (UID: \"ee85beb2-2c97-4885-bc8d-08a90b2a85de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" Oct 30 20:22:14.461719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.461684 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcdvr\" (UniqueName: \"kubernetes.io/projected/ee85beb2-2c97-4885-bc8d-08a90b2a85de-kube-api-access-mcdvr\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg\" (UID: \"ee85beb2-2c97-4885-bc8d-08a90b2a85de\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" Oct 30 20:22:14.527841 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.527786 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" Oct 30 20:22:14.815495 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:14.815453 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg"] Oct 30 20:22:14.815704 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:22:14.815674 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee85beb2_2c97_4885_bc8d_08a90b2a85de.slice/crio-23a827d1bc708e4bf53255607ef6bd83912d47b9e0135c9aefd5ed50a2e414ef WatchSource:0}: Error finding container 23a827d1bc708e4bf53255607ef6bd83912d47b9e0135c9aefd5ed50a2e414ef: Status 404 returned error can't find the container with id 23a827d1bc708e4bf53255607ef6bd83912d47b9e0135c9aefd5ed50a2e414ef Oct 30 20:22:15.033637 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:15.033601 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb"] Oct 30 20:22:15.040985 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:15.040956 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e02073a0-stsbb"] Oct 30 20:22:15.099216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:15.098990 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:15.253556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:15.253520 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:15.656717 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:15.656673 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" event={"ID":"ee85beb2-2c97-4885-bc8d-08a90b2a85de","Type":"ContainerStarted","Data":"0cc00837b29e177dd57fbcdc644e9425318ff030404902c81edc48c613d952af"} Oct 30 20:22:15.656902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:15.656740 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" event={"ID":"ee85beb2-2c97-4885-bc8d-08a90b2a85de","Type":"ContainerStarted","Data":"23a827d1bc708e4bf53255607ef6bd83912d47b9e0135c9aefd5ed50a2e414ef"} Oct 30 20:22:15.685376 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:15.685329 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" podStartSLOduration=1.685316073 podStartE2EDuration="1.685316073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:15.682530303 +0000 UTC m=+3055.377738259" watchObservedRunningTime="2026-10-30 20:22:15.685316073 +0000 UTC m=+3055.380524031" Oct 30 20:22:15.903183 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:15.897756 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:15.977714 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:22:15.977672 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:22:16.152555 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:16.152511 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" podUID="b423dc9e-e093-4107-b703-b94fa7c96911" containerName="payload-processing" containerID="cri-o://c0f59e92b80b57445f885ec82a976b7f2e98e7e5e42feecc15fa1ba42380e70b" gracePeriod=25 Oct 30 20:22:16.168328 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:16.168290 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" podUID="ca54001a-b870-4f69-a748-8e71e28c8241" containerName="payload-pre-processing" containerID="cri-o://7cfb65325a0de0326e94225f9eadd6338d2f77edd215ac52c2496a3554b97771" gracePeriod=25 Oct 30 20:22:16.441642 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:16.441607 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceea20c5-24d4-4269-8f0a-a022f2a34e76" path="/var/lib/kubelet/pods/ceea20c5-24d4-4269-8f0a-a022f2a34e76/volumes" Oct 30 20:22:16.674659 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:16.674626 5091 generic.go:356] "Generic (PLEG): container finished" podID="ee85beb2-2c97-4885-bc8d-08a90b2a85de" containerID="0cc00837b29e177dd57fbcdc644e9425318ff030404902c81edc48c613d952af" exitCode=0 Oct 30 20:22:16.674848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:16.674667 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" event={"ID":"ee85beb2-2c97-4885-bc8d-08a90b2a85de","Type":"ContainerDied","Data":"0cc00837b29e177dd57fbcdc644e9425318ff030404902c81edc48c613d952af"} Oct 30 20:22:16.735534 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:16.733435 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:17.551811 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:17.551770 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" podUID="ca54001a-b870-4f69-a748-8e71e28c8241" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:22:17.551811 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:17.551800 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" podUID="b423dc9e-e093-4107-b703-b94fa7c96911" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:22:17.808685 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:17.808320 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:17.868749 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:17.868715 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" Oct 30 20:22:18.008972 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.008934 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee85beb2-2c97-4885-bc8d-08a90b2a85de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee85beb2-2c97-4885-bc8d-08a90b2a85de-maas-api-xks-ca\") pod \"ee85beb2-2c97-4885-bc8d-08a90b2a85de\" (UID: \"ee85beb2-2c97-4885-bc8d-08a90b2a85de\") " Oct 30 20:22:18.009204 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.009190 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee85beb2-2c97-4885-bc8d-08a90b2a85de-kube-api-access-mcdvr\" (UniqueName: \"kubernetes.io/projected/ee85beb2-2c97-4885-bc8d-08a90b2a85de-kube-api-access-mcdvr\") pod \"ee85beb2-2c97-4885-bc8d-08a90b2a85de\" (UID: \"ee85beb2-2c97-4885-bc8d-08a90b2a85de\") " Oct 30 20:22:18.009248 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.009220 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee85beb2-2c97-4885-bc8d-08a90b2a85de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee85beb2-2c97-4885-bc8d-08a90b2a85de-maas-api-openshift-ca\") pod \"ee85beb2-2c97-4885-bc8d-08a90b2a85de\" (UID: \"ee85beb2-2c97-4885-bc8d-08a90b2a85de\") " Oct 30 20:22:18.009695 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.009660 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee85beb2-2c97-4885-bc8d-08a90b2a85de-maas-api-openshift-ca" pod "ee85beb2-2c97-4885-bc8d-08a90b2a85de" (UID: "ee85beb2-2c97-4885-bc8d-08a90b2a85de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:18.011160 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.011127 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee85beb2-2c97-4885-bc8d-08a90b2a85de-maas-api-xks-ca" pod "ee85beb2-2c97-4885-bc8d-08a90b2a85de" (UID: "ee85beb2-2c97-4885-bc8d-08a90b2a85de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:22:18.011338 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.011312 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee85beb2-2c97-4885-bc8d-08a90b2a85de-kube-api-access-mcdvr" pod "ee85beb2-2c97-4885-bc8d-08a90b2a85de" (UID: "ee85beb2-2c97-4885-bc8d-08a90b2a85de"). InnerVolumeSpecName "kube-api-access-mcdvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:18.110639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.110564 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee85beb2-2c97-4885-bc8d-08a90b2a85de-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:18.110639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.110594 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcdvr\" (UniqueName: \"kubernetes.io/projected/ee85beb2-2c97-4885-bc8d-08a90b2a85de-kube-api-access-mcdvr\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:18.110639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.110606 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee85beb2-2c97-4885-bc8d-08a90b2a85de-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:18.607294 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.607250 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:18.695638 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.695608 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" event={"ID":"ee85beb2-2c97-4885-bc8d-08a90b2a85de","Type":"ContainerDied","Data":"23a827d1bc708e4bf53255607ef6bd83912d47b9e0135c9aefd5ed50a2e414ef"} Oct 30 20:22:18.695638 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.695625 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg" Oct 30 20:22:18.695787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.695666 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23a827d1bc708e4bf53255607ef6bd83912d47b9e0135c9aefd5ed50a2e414ef" Oct 30 20:22:18.842054 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.842013 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:18.905827 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.905769 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj"] Oct 30 20:22:18.909374 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.909348 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-222480-nj7jj"] Oct 30 20:22:18.958086 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.958062 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:22:18.993784 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:18.993749 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:19.179307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.179221 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:19.186234 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.186212 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5109436f-1c85-4413-a7e7-f62dac25203f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5109436f-1c85-4413-a7e7-f62dac25203f-istiod-ca-cert\") pod \"5109436f-1c85-4413-a7e7-f62dac25203f\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " Oct 30 20:22:19.186332 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.186281 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-istio-data\") pod \"5109436f-1c85-4413-a7e7-f62dac25203f\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " Oct 30 20:22:19.186332 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.186321 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-workload-socket\") pod \"5109436f-1c85-4413-a7e7-f62dac25203f\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " Oct 30 20:22:19.186437 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.186352 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5109436f-1c85-4413-a7e7-f62dac25203f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5109436f-1c85-4413-a7e7-f62dac25203f-istio-podinfo\") pod \"5109436f-1c85-4413-a7e7-f62dac25203f\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " Oct 30 20:22:19.186437 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.186391 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-workload-certs\") pod \"5109436f-1c85-4413-a7e7-f62dac25203f\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " Oct 30 20:22:19.186437 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.186420 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-istio-envoy\") pod \"5109436f-1c85-4413-a7e7-f62dac25203f\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " Oct 30 20:22:19.186581 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.186489 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5109436f-1c85-4413-a7e7-f62dac25203f-kube-api-access-8w62t\" (UniqueName: \"kubernetes.io/projected/5109436f-1c85-4413-a7e7-f62dac25203f-kube-api-access-8w62t\") pod \"5109436f-1c85-4413-a7e7-f62dac25203f\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " Oct 30 20:22:19.186581 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.186545 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-istio-data" pod "5109436f-1c85-4413-a7e7-f62dac25203f" (UID: "5109436f-1c85-4413-a7e7-f62dac25203f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:19.186581 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.186555 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-credential-socket\") pod \"5109436f-1c85-4413-a7e7-f62dac25203f\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " Oct 30 20:22:19.186733 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.186637 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5109436f-1c85-4413-a7e7-f62dac25203f-istio-token\" (UniqueName: \"kubernetes.io/projected/5109436f-1c85-4413-a7e7-f62dac25203f-istio-token\") pod \"5109436f-1c85-4413-a7e7-f62dac25203f\" (UID: \"5109436f-1c85-4413-a7e7-f62dac25203f\") " Oct 30 20:22:19.186733 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.186701 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-workload-socket" pod "5109436f-1c85-4413-a7e7-f62dac25203f" (UID: "5109436f-1c85-4413-a7e7-f62dac25203f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:19.186861 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.186731 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5109436f-1c85-4413-a7e7-f62dac25203f-istiod-ca-cert" pod "5109436f-1c85-4413-a7e7-f62dac25203f" (UID: "5109436f-1c85-4413-a7e7-f62dac25203f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:19.186919 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.186885 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-credential-socket" pod "5109436f-1c85-4413-a7e7-f62dac25203f" (UID: "5109436f-1c85-4413-a7e7-f62dac25203f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:19.187011 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.186982 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-workload-certs" pod "5109436f-1c85-4413-a7e7-f62dac25203f" (UID: "5109436f-1c85-4413-a7e7-f62dac25203f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:19.187219 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.187195 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5109436f-1c85-4413-a7e7-f62dac25203f-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:19.187312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.187221 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:19.187312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.187237 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:19.187312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.187250 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:19.187312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.187262 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:19.188850 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.188820 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-istio-envoy" pod "5109436f-1c85-4413-a7e7-f62dac25203f" (UID: "5109436f-1c85-4413-a7e7-f62dac25203f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:19.189222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.189202 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5109436f-1c85-4413-a7e7-f62dac25203f-istio-podinfo" pod "5109436f-1c85-4413-a7e7-f62dac25203f" (UID: "5109436f-1c85-4413-a7e7-f62dac25203f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:22:19.189355 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.189333 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5109436f-1c85-4413-a7e7-f62dac25203f-istio-token" pod "5109436f-1c85-4413-a7e7-f62dac25203f" (UID: "5109436f-1c85-4413-a7e7-f62dac25203f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:19.189428 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.189352 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5109436f-1c85-4413-a7e7-f62dac25203f-kube-api-access-8w62t" pod "5109436f-1c85-4413-a7e7-f62dac25203f" (UID: "5109436f-1c85-4413-a7e7-f62dac25203f"). InnerVolumeSpecName "kube-api-access-8w62t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:19.288677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.288645 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5109436f-1c85-4413-a7e7-f62dac25203f-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:19.288677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.288672 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5109436f-1c85-4413-a7e7-f62dac25203f-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:19.288677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.288682 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w62t\" (UniqueName: \"kubernetes.io/projected/5109436f-1c85-4413-a7e7-f62dac25203f-kube-api-access-8w62t\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:19.288889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.288693 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5109436f-1c85-4413-a7e7-f62dac25203f-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:19.463191 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.463068 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:19.706650 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.706607 5091 generic.go:356] "Generic (PLEG): container finished" podID="5109436f-1c85-4413-a7e7-f62dac25203f" containerID="c5db5edb55b9cbbd2fed3e99b4dd54eebbbc50d738d2bf0f882067ff447533c4" exitCode=0 Oct 30 20:22:19.707161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.706657 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" event={"ID":"5109436f-1c85-4413-a7e7-f62dac25203f","Type":"ContainerDied","Data":"c5db5edb55b9cbbd2fed3e99b4dd54eebbbc50d738d2bf0f882067ff447533c4"} Oct 30 20:22:19.707161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.706664 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" Oct 30 20:22:19.707161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.706717 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72" event={"ID":"5109436f-1c85-4413-a7e7-f62dac25203f","Type":"ContainerDied","Data":"c6dfc7d70ee86c32d3b0132c9b267114ee7c29683798db903bfc5fd6d6f285fc"} Oct 30 20:22:19.707161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.706749 5091 scope.go:122] "RemoveContainer" containerID="c5db5edb55b9cbbd2fed3e99b4dd54eebbbc50d738d2bf0f882067ff447533c4" Oct 30 20:22:19.720052 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.720023 5091 scope.go:122] "RemoveContainer" containerID="0f4009c3de1f21e6df9055dbab35fb4077d58766c61711a5fbce24a99dcb56e7" Oct 30 20:22:19.732202 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.732184 5091 scope.go:122] "RemoveContainer" containerID="c5db5edb55b9cbbd2fed3e99b4dd54eebbbc50d738d2bf0f882067ff447533c4" Oct 30 20:22:19.732485 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:22:19.732467 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5db5edb55b9cbbd2fed3e99b4dd54eebbbc50d738d2bf0f882067ff447533c4\": container with ID starting with c5db5edb55b9cbbd2fed3e99b4dd54eebbbc50d738d2bf0f882067ff447533c4 not found: ID does not exist" containerID="c5db5edb55b9cbbd2fed3e99b4dd54eebbbc50d738d2bf0f882067ff447533c4" Oct 30 20:22:19.732530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.732494 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5db5edb55b9cbbd2fed3e99b4dd54eebbbc50d738d2bf0f882067ff447533c4"} err="failed to get container status \"c5db5edb55b9cbbd2fed3e99b4dd54eebbbc50d738d2bf0f882067ff447533c4\": rpc error: code = NotFound desc = could not find container \"c5db5edb55b9cbbd2fed3e99b4dd54eebbbc50d738d2bf0f882067ff447533c4\": container with ID starting with c5db5edb55b9cbbd2fed3e99b4dd54eebbbc50d738d2bf0f882067ff447533c4 not found: ID does not exist" Oct 30 20:22:19.732530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.732512 5091 scope.go:122] "RemoveContainer" containerID="0f4009c3de1f21e6df9055dbab35fb4077d58766c61711a5fbce24a99dcb56e7" Oct 30 20:22:19.732781 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:22:19.732762 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f4009c3de1f21e6df9055dbab35fb4077d58766c61711a5fbce24a99dcb56e7\": container with ID starting with 0f4009c3de1f21e6df9055dbab35fb4077d58766c61711a5fbce24a99dcb56e7 not found: ID does not exist" containerID="0f4009c3de1f21e6df9055dbab35fb4077d58766c61711a5fbce24a99dcb56e7" Oct 30 20:22:19.732827 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.732787 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f4009c3de1f21e6df9055dbab35fb4077d58766c61711a5fbce24a99dcb56e7"} err="failed to get container status \"0f4009c3de1f21e6df9055dbab35fb4077d58766c61711a5fbce24a99dcb56e7\": rpc error: code = NotFound desc = could not find container \"0f4009c3de1f21e6df9055dbab35fb4077d58766c61711a5fbce24a99dcb56e7\": container with ID starting with 0f4009c3de1f21e6df9055dbab35fb4077d58766c61711a5fbce24a99dcb56e7 not found: ID does not exist" Oct 30 20:22:19.738085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.738061 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72"] Oct 30 20:22:19.743479 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.743455 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-5f9b3c-openshift-default-779985d877-ngv72"] Oct 30 20:22:19.968655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:19.968501 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:20.442398 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:20.442363 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" path="/var/lib/kubelet/pods/5109436f-1c85-4413-a7e7-f62dac25203f/volumes" Oct 30 20:22:20.442837 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:20.442822 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d30db8d5-5bfd-4e2e-add3-aa61588dc165" path="/var/lib/kubelet/pods/d30db8d5-5bfd-4e2e-add3-aa61588dc165/volumes" Oct 30 20:22:20.867592 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:20.867549 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:21.894771 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:21.894732 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:22.743632 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:22.743594 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:23.561979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:23.561939 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:24.485564 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:24.485525 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:24.631821 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:24.631782 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:25.028036 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:25.028001 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp"] Oct 30 20:22:25.034384 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:25.034344 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50ff8fa5-tgzqp"] Oct 30 20:22:25.255193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:25.255154 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:26.174822 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:26.174784 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:26.444205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:26.444102 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e66a26f3-5b02-4bb5-b3b2-f33cffb93bec" path="/var/lib/kubelet/pods/e66a26f3-5b02-4bb5-b3b2-f33cffb93bec/volumes" Oct 30 20:22:26.970672 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:26.970635 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:27.125179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.122717 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:27.166769 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.165804 5091 scope.go:122] "RemoveContainer" containerID="cc9181cb9d8515085f21a54ce5fe846609ba287c01c04aaa2576e114756c8dc4" Oct 30 20:22:27.184204 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.184117 5091 scope.go:122] "RemoveContainer" containerID="f5500571352ec05d957229ba49b52690a8bcf87429b3fd7d69e75c6809606fb6" Oct 30 20:22:27.199620 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.199593 5091 scope.go:122] "RemoveContainer" containerID="30afe6887606747bea451a55adb89fcd95a557fa76ee6bc71c9ebe46f7df1caf" Oct 30 20:22:27.211523 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.211501 5091 scope.go:122] "RemoveContainer" containerID="84ebb82bac85eee2526f21b3fa24b02aa4ec4258c4136a67c625f2301694ca09" Oct 30 20:22:27.221371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.221342 5091 scope.go:122] "RemoveContainer" containerID="5c6e70682bb09bae6664f9e5835acfe4c62e4cbadd23687f5c608de27b9ad9c9" Oct 30 20:22:27.230519 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.230497 5091 scope.go:122] "RemoveContainer" containerID="65923c0ed3f4a05f36d31e7b80f60b41cf1124cc3445b79c3531ac312b5da3a2" Oct 30 20:22:27.241581 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.241561 5091 scope.go:122] "RemoveContainer" containerID="65ef639f736dfb39823d8fe6dd72ebfe8e9afbd4bb4c8e26eba904e62988cb33" Oct 30 20:22:27.242054 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.242030 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg"] Oct 30 20:22:27.250736 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.250708 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-7eaeed-mhzvg"] Oct 30 20:22:27.253503 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.253483 5091 scope.go:122] "RemoveContainer" containerID="c87e19f4c679b5acdcc9827ef816fe7cba3ea115d99dff5f1c940af63b1dfa7f" Oct 30 20:22:27.264370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.264350 5091 scope.go:122] "RemoveContainer" containerID="408a6022ae16bb52d52ad9e8664abc35750ca8556f82bd388b740f916584bad8" Oct 30 20:22:27.296201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.289868 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:27.438426 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.438351 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:27.552329 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.552292 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" podUID="b423dc9e-e093-4107-b703-b94fa7c96911" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:22:27.552329 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.552318 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" podUID="ca54001a-b870-4f69-a748-8e71e28c8241" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:22:27.785277 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.785243 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:27.856367 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.856276 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth"] Oct 30 20:22:27.856890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.856870 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ee85beb2-2c97-4885-bc8d-08a90b2a85de" containerName="revoke-keys" Oct 30 20:22:27.856890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.856889 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee85beb2-2c97-4885-bc8d-08a90b2a85de" containerName="revoke-keys" Oct 30 20:22:27.857066 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.856905 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" containerName="istio-proxy" Oct 30 20:22:27.857066 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.856911 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" containerName="istio-proxy" Oct 30 20:22:27.857066 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.856919 5091 cpu_manager.go:414] "containerMap: removing container" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" containerName="istio-proxy" Oct 30 20:22:27.857066 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.856924 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" containerName="istio-proxy" Oct 30 20:22:27.857066 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.857039 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee85beb2-2c97-4885-bc8d-08a90b2a85de" containerName="revoke-keys" Oct 30 20:22:27.857066 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.857047 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" containerName="istio-proxy" Oct 30 20:22:27.857066 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.857056 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="5109436f-1c85-4413-a7e7-f62dac25203f" containerName="istio-proxy" Oct 30 20:22:27.862284 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.862261 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" Oct 30 20:22:27.869482 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.869455 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth"] Oct 30 20:22:27.915059 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.915032 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:22:27.915246 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:27.915228 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:22:28.024105 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.024073 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d48f525-1075-43ef-9cad-226a2b75a73a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth\" (UID: \"9d48f525-1075-43ef-9cad-226a2b75a73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" Oct 30 20:22:28.024312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.024252 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d48f525-1075-43ef-9cad-226a2b75a73a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth\" (UID: \"9d48f525-1075-43ef-9cad-226a2b75a73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" Oct 30 20:22:28.024359 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.024323 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ctgt\" (UniqueName: \"kubernetes.io/projected/9d48f525-1075-43ef-9cad-226a2b75a73a-kube-api-access-5ctgt\") pod \"maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth\" (UID: \"9d48f525-1075-43ef-9cad-226a2b75a73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" Oct 30 20:22:28.125425 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.125338 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d48f525-1075-43ef-9cad-226a2b75a73a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth\" (UID: \"9d48f525-1075-43ef-9cad-226a2b75a73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" Oct 30 20:22:28.125425 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.125406 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ctgt\" (UniqueName: \"kubernetes.io/projected/9d48f525-1075-43ef-9cad-226a2b75a73a-kube-api-access-5ctgt\") pod \"maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth\" (UID: \"9d48f525-1075-43ef-9cad-226a2b75a73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" Oct 30 20:22:28.125627 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.125560 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d48f525-1075-43ef-9cad-226a2b75a73a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth\" (UID: \"9d48f525-1075-43ef-9cad-226a2b75a73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" Oct 30 20:22:28.126088 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.126065 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d48f525-1075-43ef-9cad-226a2b75a73a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth\" (UID: \"9d48f525-1075-43ef-9cad-226a2b75a73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" Oct 30 20:22:28.127897 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.127874 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d48f525-1075-43ef-9cad-226a2b75a73a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth\" (UID: \"9d48f525-1075-43ef-9cad-226a2b75a73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" Oct 30 20:22:28.175866 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.175839 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ctgt\" (UniqueName: \"kubernetes.io/projected/9d48f525-1075-43ef-9cad-226a2b75a73a-kube-api-access-5ctgt\") pod \"maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth\" (UID: \"9d48f525-1075-43ef-9cad-226a2b75a73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" Oct 30 20:22:28.184474 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.184451 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" Oct 30 20:22:28.424279 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.423050 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:28.446093 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.446057 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee85beb2-2c97-4885-bc8d-08a90b2a85de" path="/var/lib/kubelet/pods/ee85beb2-2c97-4885-bc8d-08a90b2a85de/volumes" Oct 30 20:22:28.475379 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.475350 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth"] Oct 30 20:22:28.476849 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:22:28.476824 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d48f525_1075_43ef_9cad_226a2b75a73a.slice/crio-995f37473f0bc080af6053615e1348b9434e4c9b3eaad41d580c1b45152b78b3 WatchSource:0}: Error finding container 995f37473f0bc080af6053615e1348b9434e4c9b3eaad41d580c1b45152b78b3: Status 404 returned error can't find the container with id 995f37473f0bc080af6053615e1348b9434e4c9b3eaad41d580c1b45152b78b3 Oct 30 20:22:28.807427 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.807392 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" event={"ID":"9d48f525-1075-43ef-9cad-226a2b75a73a","Type":"ContainerStarted","Data":"b8049b5c4f4857a929f30cc2d9c1fe7409ebd09f473738ee5eeaf2161181757d"} Oct 30 20:22:28.807581 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.807451 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" event={"ID":"9d48f525-1075-43ef-9cad-226a2b75a73a","Type":"ContainerStarted","Data":"995f37473f0bc080af6053615e1348b9434e4c9b3eaad41d580c1b45152b78b3"} Oct 30 20:22:28.831111 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:28.831060 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" podStartSLOduration=1.8310165999999999 podStartE2EDuration="1.8310166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:28.830835461 +0000 UTC m=+3068.526043419" watchObservedRunningTime="2026-10-30 20:22:28.8310166 +0000 UTC m=+3068.526224561" Oct 30 20:22:29.447783 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:29.447747 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:29.818438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:29.818402 5091 generic.go:356] "Generic (PLEG): container finished" podID="9d48f525-1075-43ef-9cad-226a2b75a73a" containerID="b8049b5c4f4857a929f30cc2d9c1fe7409ebd09f473738ee5eeaf2161181757d" exitCode=0 Oct 30 20:22:29.818596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:29.818484 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" event={"ID":"9d48f525-1075-43ef-9cad-226a2b75a73a","Type":"ContainerDied","Data":"b8049b5c4f4857a929f30cc2d9c1fe7409ebd09f473738ee5eeaf2161181757d"} Oct 30 20:22:30.376573 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:30.376527 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:30.982711 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:30.982683 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" Oct 30 20:22:31.059987 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:31.059953 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d48f525-1075-43ef-9cad-226a2b75a73a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d48f525-1075-43ef-9cad-226a2b75a73a-maas-api-xks-ca\") pod \"9d48f525-1075-43ef-9cad-226a2b75a73a\" (UID: \"9d48f525-1075-43ef-9cad-226a2b75a73a\") " Oct 30 20:22:31.060251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:31.060122 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d48f525-1075-43ef-9cad-226a2b75a73a-kube-api-access-5ctgt\" (UniqueName: \"kubernetes.io/projected/9d48f525-1075-43ef-9cad-226a2b75a73a-kube-api-access-5ctgt\") pod \"9d48f525-1075-43ef-9cad-226a2b75a73a\" (UID: \"9d48f525-1075-43ef-9cad-226a2b75a73a\") " Oct 30 20:22:31.060251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:31.060223 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d48f525-1075-43ef-9cad-226a2b75a73a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d48f525-1075-43ef-9cad-226a2b75a73a-maas-api-openshift-ca\") pod \"9d48f525-1075-43ef-9cad-226a2b75a73a\" (UID: \"9d48f525-1075-43ef-9cad-226a2b75a73a\") " Oct 30 20:22:31.060650 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:31.060617 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d48f525-1075-43ef-9cad-226a2b75a73a-maas-api-openshift-ca" pod "9d48f525-1075-43ef-9cad-226a2b75a73a" (UID: "9d48f525-1075-43ef-9cad-226a2b75a73a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:31.060747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:31.060710 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d48f525-1075-43ef-9cad-226a2b75a73a-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:31.062412 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:31.062383 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d48f525-1075-43ef-9cad-226a2b75a73a-maas-api-xks-ca" pod "9d48f525-1075-43ef-9cad-226a2b75a73a" (UID: "9d48f525-1075-43ef-9cad-226a2b75a73a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:22:31.062660 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:31.062646 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d48f525-1075-43ef-9cad-226a2b75a73a-kube-api-access-5ctgt" pod "9d48f525-1075-43ef-9cad-226a2b75a73a" (UID: "9d48f525-1075-43ef-9cad-226a2b75a73a"). InnerVolumeSpecName "kube-api-access-5ctgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:31.172226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:31.172043 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ctgt\" (UniqueName: \"kubernetes.io/projected/9d48f525-1075-43ef-9cad-226a2b75a73a-kube-api-access-5ctgt\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:31.172226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:31.172091 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d48f525-1075-43ef-9cad-226a2b75a73a-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:31.176180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:31.174708 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:31.836306 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:31.836274 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" Oct 30 20:22:31.836306 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:31.836306 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth" event={"ID":"9d48f525-1075-43ef-9cad-226a2b75a73a","Type":"ContainerDied","Data":"995f37473f0bc080af6053615e1348b9434e4c9b3eaad41d580c1b45152b78b3"} Oct 30 20:22:31.836591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:31.836354 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="995f37473f0bc080af6053615e1348b9434e4c9b3eaad41d580c1b45152b78b3" Oct 30 20:22:32.165296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:32.158233 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:32.958898 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:32.958847 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:33.870178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:33.868862 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:34.737608 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:34.737571 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:34.901039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:34.900994 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:35.051677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.051641 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:35.269513 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.269477 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:35.569334 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.569296 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:35.706173 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.706112 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk"] Oct 30 20:22:35.706684 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.706669 5091 cpu_manager.go:414] "containerMap: removing container" podUID="9d48f525-1075-43ef-9cad-226a2b75a73a" containerName="revoke-keys" Oct 30 20:22:35.706735 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.706685 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d48f525-1075-43ef-9cad-226a2b75a73a" containerName="revoke-keys" Oct 30 20:22:35.706803 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.706793 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d48f525-1075-43ef-9cad-226a2b75a73a" containerName="revoke-keys" Oct 30 20:22:35.721491 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.721451 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" Oct 30 20:22:35.725109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.725078 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk"] Oct 30 20:22:35.739914 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.739878 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76xnn\" (UniqueName: \"kubernetes.io/projected/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-kube-api-access-76xnn\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk\" (UID: \"b2d0f616-0b81-4c73-9e93-63b46a9d02b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" Oct 30 20:22:35.740072 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.739998 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk\" (UID: \"b2d0f616-0b81-4c73-9e93-63b46a9d02b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" Oct 30 20:22:35.740286 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.740215 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk\" (UID: \"b2d0f616-0b81-4c73-9e93-63b46a9d02b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" Oct 30 20:22:35.808756 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.808721 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:22:35.808977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.808911 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:22:35.841158 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.841054 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk\" (UID: \"b2d0f616-0b81-4c73-9e93-63b46a9d02b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" Oct 30 20:22:35.841345 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.841214 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk\" (UID: \"b2d0f616-0b81-4c73-9e93-63b46a9d02b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" Oct 30 20:22:35.841429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.841344 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76xnn\" (UniqueName: \"kubernetes.io/projected/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-kube-api-access-76xnn\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk\" (UID: \"b2d0f616-0b81-4c73-9e93-63b46a9d02b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" Oct 30 20:22:35.841930 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.841905 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk\" (UID: \"b2d0f616-0b81-4c73-9e93-63b46a9d02b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" Oct 30 20:22:35.843653 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.843626 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk\" (UID: \"b2d0f616-0b81-4c73-9e93-63b46a9d02b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" Oct 30 20:22:35.856650 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:35.856617 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76xnn\" (UniqueName: \"kubernetes.io/projected/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-kube-api-access-76xnn\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk\" (UID: \"b2d0f616-0b81-4c73-9e93-63b46a9d02b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" Oct 30 20:22:36.039656 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.039623 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" Oct 30 20:22:36.325791 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.325754 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk"] Oct 30 20:22:36.329809 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:22:36.329774 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d0f616_0b81_4c73_9e93_63b46a9d02b1.slice/crio-15e126c1a1dde10110fc1709031442d23d12d04158482660583f5fedd81630d4 WatchSource:0}: Error finding container 15e126c1a1dde10110fc1709031442d23d12d04158482660583f5fedd81630d4: Status 404 returned error can't find the container with id 15e126c1a1dde10110fc1709031442d23d12d04158482660583f5fedd81630d4 Oct 30 20:22:36.448376 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.447751 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" Oct 30 20:22:36.560860 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.560834 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b423dc9e-e093-4107-b703-b94fa7c96911-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b423dc9e-e093-4107-b703-b94fa7c96911-plugins-config-volume\") pod \"b423dc9e-e093-4107-b703-b94fa7c96911\" (UID: \"b423dc9e-e093-4107-b703-b94fa7c96911\") " Oct 30 20:22:36.561005 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.560874 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b423dc9e-e093-4107-b703-b94fa7c96911-kube-api-access-tg8ft\" (UniqueName: \"kubernetes.io/projected/b423dc9e-e093-4107-b703-b94fa7c96911-kube-api-access-tg8ft\") pod \"b423dc9e-e093-4107-b703-b94fa7c96911\" (UID: \"b423dc9e-e093-4107-b703-b94fa7c96911\") " Oct 30 20:22:36.561220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.561194 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b423dc9e-e093-4107-b703-b94fa7c96911-plugins-config-volume" pod "b423dc9e-e093-4107-b703-b94fa7c96911" (UID: "b423dc9e-e093-4107-b703-b94fa7c96911"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:36.562900 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.562879 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b423dc9e-e093-4107-b703-b94fa7c96911-kube-api-access-tg8ft" pod "b423dc9e-e093-4107-b703-b94fa7c96911" (UID: "b423dc9e-e093-4107-b703-b94fa7c96911"). InnerVolumeSpecName "kube-api-access-tg8ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:36.578222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.578201 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" Oct 30 20:22:36.662454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.662373 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca54001a-b870-4f69-a748-8e71e28c8241-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca54001a-b870-4f69-a748-8e71e28c8241-plugins-config-volume\") pod \"ca54001a-b870-4f69-a748-8e71e28c8241\" (UID: \"ca54001a-b870-4f69-a748-8e71e28c8241\") " Oct 30 20:22:36.662454 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.662445 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca54001a-b870-4f69-a748-8e71e28c8241-kube-api-access-qgxmn\" (UniqueName: \"kubernetes.io/projected/ca54001a-b870-4f69-a748-8e71e28c8241-kube-api-access-qgxmn\") pod \"ca54001a-b870-4f69-a748-8e71e28c8241\" (UID: \"ca54001a-b870-4f69-a748-8e71e28c8241\") " Oct 30 20:22:36.662738 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.662714 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca54001a-b870-4f69-a748-8e71e28c8241-plugins-config-volume" pod "ca54001a-b870-4f69-a748-8e71e28c8241" (UID: "ca54001a-b870-4f69-a748-8e71e28c8241"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:36.662815 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.662774 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b423dc9e-e093-4107-b703-b94fa7c96911-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:36.662815 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.662788 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg8ft\" (UniqueName: \"kubernetes.io/projected/b423dc9e-e093-4107-b703-b94fa7c96911-kube-api-access-tg8ft\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:36.754171 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.753991 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:36.767188 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.766970 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca54001a-b870-4f69-a748-8e71e28c8241-kube-api-access-qgxmn" pod "ca54001a-b870-4f69-a748-8e71e28c8241" (UID: "ca54001a-b870-4f69-a748-8e71e28c8241"). InnerVolumeSpecName "kube-api-access-qgxmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:36.769290 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.768360 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgxmn\" (UniqueName: \"kubernetes.io/projected/ca54001a-b870-4f69-a748-8e71e28c8241-kube-api-access-qgxmn\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:36.769290 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.768393 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca54001a-b870-4f69-a748-8e71e28c8241-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:36.903888 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.903857 5091 generic.go:356] "Generic (PLEG): container finished" podID="ca54001a-b870-4f69-a748-8e71e28c8241" containerID="7cfb65325a0de0326e94225f9eadd6338d2f77edd215ac52c2496a3554b97771" exitCode=0 Oct 30 20:22:36.904048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.903902 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" Oct 30 20:22:36.904048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.903929 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" event={"ID":"ca54001a-b870-4f69-a748-8e71e28c8241","Type":"ContainerDied","Data":"7cfb65325a0de0326e94225f9eadd6338d2f77edd215ac52c2496a3554b97771"} Oct 30 20:22:36.904048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.903981 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql" event={"ID":"ca54001a-b870-4f69-a748-8e71e28c8241","Type":"ContainerDied","Data":"a738eb61632e2d0b290fa2d2ae638c250e1df02376dd4563894a55ed7e176262"} Oct 30 20:22:36.904048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.904010 5091 scope.go:122] "RemoveContainer" containerID="7cfb65325a0de0326e94225f9eadd6338d2f77edd215ac52c2496a3554b97771" Oct 30 20:22:36.905421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.905398 5091 generic.go:356] "Generic (PLEG): container finished" podID="b423dc9e-e093-4107-b703-b94fa7c96911" containerID="c0f59e92b80b57445f885ec82a976b7f2e98e7e5e42feecc15fa1ba42380e70b" exitCode=0 Oct 30 20:22:36.905534 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.905447 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" Oct 30 20:22:36.905639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.905613 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" event={"ID":"b423dc9e-e093-4107-b703-b94fa7c96911","Type":"ContainerDied","Data":"c0f59e92b80b57445f885ec82a976b7f2e98e7e5e42feecc15fa1ba42380e70b"} Oct 30 20:22:36.905767 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.905668 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx" event={"ID":"b423dc9e-e093-4107-b703-b94fa7c96911","Type":"ContainerDied","Data":"73ca098dc064be2d6e2da82143358bc48b16bb65a9886f422a2449614a8bb124"} Oct 30 20:22:36.907530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.907510 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" event={"ID":"b2d0f616-0b81-4c73-9e93-63b46a9d02b1","Type":"ContainerStarted","Data":"89e67d146fffcbed3e443eae986360f1063fdc7056a9f541027249e581d5c7e7"} Oct 30 20:22:36.907616 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.907551 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" event={"ID":"b2d0f616-0b81-4c73-9e93-63b46a9d02b1","Type":"ContainerStarted","Data":"15e126c1a1dde10110fc1709031442d23d12d04158482660583f5fedd81630d4"} Oct 30 20:22:36.921465 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.921441 5091 scope.go:122] "RemoveContainer" containerID="7cfb65325a0de0326e94225f9eadd6338d2f77edd215ac52c2496a3554b97771" Oct 30 20:22:36.921780 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:22:36.921758 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cfb65325a0de0326e94225f9eadd6338d2f77edd215ac52c2496a3554b97771\": container with ID starting with 7cfb65325a0de0326e94225f9eadd6338d2f77edd215ac52c2496a3554b97771 not found: ID does not exist" containerID="7cfb65325a0de0326e94225f9eadd6338d2f77edd215ac52c2496a3554b97771" Oct 30 20:22:36.921851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.921789 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cfb65325a0de0326e94225f9eadd6338d2f77edd215ac52c2496a3554b97771"} err="failed to get container status \"7cfb65325a0de0326e94225f9eadd6338d2f77edd215ac52c2496a3554b97771\": rpc error: code = NotFound desc = could not find container \"7cfb65325a0de0326e94225f9eadd6338d2f77edd215ac52c2496a3554b97771\": container with ID starting with 7cfb65325a0de0326e94225f9eadd6338d2f77edd215ac52c2496a3554b97771 not found: ID does not exist" Oct 30 20:22:36.921851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.921813 5091 scope.go:122] "RemoveContainer" containerID="c0f59e92b80b57445f885ec82a976b7f2e98e7e5e42feecc15fa1ba42380e70b" Oct 30 20:22:36.931710 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.931654 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" podStartSLOduration=1.9316357659999999 podStartE2EDuration="1.931635766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:22:36.930696566 +0000 UTC m=+3076.625904524" watchObservedRunningTime="2026-10-30 20:22:36.931635766 +0000 UTC m=+3076.626843730" Oct 30 20:22:36.932642 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.932621 5091 scope.go:122] "RemoveContainer" containerID="c0f59e92b80b57445f885ec82a976b7f2e98e7e5e42feecc15fa1ba42380e70b" Oct 30 20:22:36.932941 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:22:36.932918 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0f59e92b80b57445f885ec82a976b7f2e98e7e5e42feecc15fa1ba42380e70b\": container with ID starting with c0f59e92b80b57445f885ec82a976b7f2e98e7e5e42feecc15fa1ba42380e70b not found: ID does not exist" containerID="c0f59e92b80b57445f885ec82a976b7f2e98e7e5e42feecc15fa1ba42380e70b" Oct 30 20:22:36.933043 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.932945 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0f59e92b80b57445f885ec82a976b7f2e98e7e5e42feecc15fa1ba42380e70b"} err="failed to get container status \"c0f59e92b80b57445f885ec82a976b7f2e98e7e5e42feecc15fa1ba42380e70b\": rpc error: code = NotFound desc = could not find container \"c0f59e92b80b57445f885ec82a976b7f2e98e7e5e42feecc15fa1ba42380e70b\": container with ID starting with c0f59e92b80b57445f885ec82a976b7f2e98e7e5e42feecc15fa1ba42380e70b not found: ID does not exist" Oct 30 20:22:36.954621 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.954588 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql"] Oct 30 20:22:36.963878 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:36.963855 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-222480-65976658b7-m2tql"] Oct 30 20:22:37.004493 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:37.004466 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx"] Oct 30 20:22:37.007402 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:37.007379 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-222480-655cd8df6-rvlwx"] Oct 30 20:22:37.662257 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:37.661996 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:37.918574 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:37.918480 5091 generic.go:356] "Generic (PLEG): container finished" podID="b2d0f616-0b81-4c73-9e93-63b46a9d02b1" containerID="89e67d146fffcbed3e443eae986360f1063fdc7056a9f541027249e581d5c7e7" exitCode=0 Oct 30 20:22:37.918760 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:37.918569 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" event={"ID":"b2d0f616-0b81-4c73-9e93-63b46a9d02b1","Type":"ContainerDied","Data":"89e67d146fffcbed3e443eae986360f1063fdc7056a9f541027249e581d5c7e7"} Oct 30 20:22:37.997957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:37.997926 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6"] Oct 30 20:22:38.117720 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:38.117661 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:38.299168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:38.292638 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:38.441849 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:38.441818 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b423dc9e-e093-4107-b703-b94fa7c96911" path="/var/lib/kubelet/pods/b423dc9e-e093-4107-b703-b94fa7c96911/volumes" Oct 30 20:22:38.442266 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:38.442249 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca54001a-b870-4f69-a748-8e71e28c8241" path="/var/lib/kubelet/pods/ca54001a-b870-4f69-a748-8e71e28c8241/volumes" Oct 30 20:22:38.665225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:38.661687 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:39.086705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:39.086677 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" Oct 30 20:22:39.204556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:39.204467 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-maas-api-openshift-ca\") pod \"b2d0f616-0b81-4c73-9e93-63b46a9d02b1\" (UID: \"b2d0f616-0b81-4c73-9e93-63b46a9d02b1\") " Oct 30 20:22:39.204748 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:39.204647 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-maas-api-xks-ca\") pod \"b2d0f616-0b81-4c73-9e93-63b46a9d02b1\" (UID: \"b2d0f616-0b81-4c73-9e93-63b46a9d02b1\") " Oct 30 20:22:39.204748 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:39.204695 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-kube-api-access-76xnn\" (UniqueName: \"kubernetes.io/projected/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-kube-api-access-76xnn\") pod \"b2d0f616-0b81-4c73-9e93-63b46a9d02b1\" (UID: \"b2d0f616-0b81-4c73-9e93-63b46a9d02b1\") " Oct 30 20:22:39.204881 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:39.204849 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-maas-api-openshift-ca" pod "b2d0f616-0b81-4c73-9e93-63b46a9d02b1" (UID: "b2d0f616-0b81-4c73-9e93-63b46a9d02b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:39.205182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:39.205131 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:39.206751 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:39.206727 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-maas-api-xks-ca" pod "b2d0f616-0b81-4c73-9e93-63b46a9d02b1" (UID: "b2d0f616-0b81-4c73-9e93-63b46a9d02b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:22:39.206848 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:39.206827 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-kube-api-access-76xnn" pod "b2d0f616-0b81-4c73-9e93-63b46a9d02b1" (UID: "b2d0f616-0b81-4c73-9e93-63b46a9d02b1"). InnerVolumeSpecName "kube-api-access-76xnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:39.306878 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:39.306839 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:39.306878 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:39.306874 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76xnn\" (UniqueName: \"kubernetes.io/projected/b2d0f616-0b81-4c73-9e93-63b46a9d02b1-kube-api-access-76xnn\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:39.580175 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:39.576354 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:39.934677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:39.934603 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" Oct 30 20:22:39.934677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:39.934652 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk" event={"ID":"b2d0f616-0b81-4c73-9e93-63b46a9d02b1","Type":"ContainerDied","Data":"15e126c1a1dde10110fc1709031442d23d12d04158482660583f5fedd81630d4"} Oct 30 20:22:39.935107 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:39.934705 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15e126c1a1dde10110fc1709031442d23d12d04158482660583f5fedd81630d4" Oct 30 20:22:40.477675 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:40.477639 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:40.713161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:40.713108 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:40.748343 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:40.748313 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth"] Oct 30 20:22:40.752430 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:40.752406 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-4d5758-zrzth"] Oct 30 20:22:40.869179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:40.866857 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:41.352444 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:41.352396 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:41.780974 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:41.780896 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:42.441564 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:42.441530 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d48f525-1075-43ef-9cad-226a2b75a73a" path="/var/lib/kubelet/pods/9d48f525-1075-43ef-9cad-226a2b75a73a/volumes" Oct 30 20:22:42.671529 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:42.671474 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:42.998252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:42.998209 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" podUID="a60177f4-e29b-47ea-b5cd-3068808eb97f" containerName="payload-processing" containerID="cri-o://a592671d365bea5105d7ffbe294a02675cf26c6c717bdf0eac13025f7254dd91" gracePeriod=25 Oct 30 20:22:43.557352 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:43.553964 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:44.477752 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:44.477167 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:44.546165 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:44.546111 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" podUID="a60177f4-e29b-47ea-b5cd-3068808eb97f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:22:45.356372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:45.356337 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:45.848118 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:45.848085 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v"] Oct 30 20:22:45.981351 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:22:45.981307 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:22:46.005583 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:46.005528 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:46.238216 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:46.236074 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:47.194487 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:47.194427 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:48.020078 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:48.020041 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:48.219234 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:48.219141 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:48.266456 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:48.266418 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk"] Oct 30 20:22:48.269557 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:48.269520 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-981ee5-rqdhk"] Oct 30 20:22:48.373370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:48.369301 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:48.447341 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:48.447295 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2d0f616-0b81-4c73-9e93-63b46a9d02b1" path="/var/lib/kubelet/pods/b2d0f616-0b81-4c73-9e93-63b46a9d02b1/volumes" Oct 30 20:22:48.840102 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:48.840061 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:49.498232 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:49.498196 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:49.669538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:49.669480 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:49.866772 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:49.866680 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx"] Oct 30 20:22:49.867269 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:49.867235 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" podUID="ef039607-7c00-4a64-a63b-5e4a709ae26c" containerName="istio-proxy" containerID="cri-o://ffa299f2cbebfda0637d31301684a8a6fdfbacaf38f60ed498fa29be758b22ab" gracePeriod=30 Oct 30 20:22:50.016976 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.016885 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:50.394063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.394027 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44"] Oct 30 20:22:50.394871 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.394851 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b2d0f616-0b81-4c73-9e93-63b46a9d02b1" containerName="revoke-keys" Oct 30 20:22:50.394928 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.394876 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2d0f616-0b81-4c73-9e93-63b46a9d02b1" containerName="revoke-keys" Oct 30 20:22:50.394928 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.394902 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ca54001a-b870-4f69-a748-8e71e28c8241" containerName="payload-pre-processing" Oct 30 20:22:50.394928 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.394910 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca54001a-b870-4f69-a748-8e71e28c8241" containerName="payload-pre-processing" Oct 30 20:22:50.395038 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.394931 5091 cpu_manager.go:414] "containerMap: removing container" podUID="b423dc9e-e093-4107-b703-b94fa7c96911" containerName="payload-processing" Oct 30 20:22:50.395038 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.394939 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b423dc9e-e093-4107-b703-b94fa7c96911" containerName="payload-processing" Oct 30 20:22:50.395108 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.395059 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2d0f616-0b81-4c73-9e93-63b46a9d02b1" containerName="revoke-keys" Oct 30 20:22:50.395108 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.395075 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca54001a-b870-4f69-a748-8e71e28c8241" containerName="payload-pre-processing" Oct 30 20:22:50.395108 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.395087 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="b423dc9e-e093-4107-b703-b94fa7c96911" containerName="payload-processing" Oct 30 20:22:50.398545 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.398525 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" Oct 30 20:22:50.404573 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.404541 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:22:50.404774 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.404748 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:22:50.406473 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.406446 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44"] Oct 30 20:22:50.480382 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.480327 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:50.497659 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.497410 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6whq8\" (UniqueName: \"kubernetes.io/projected/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-kube-api-access-6whq8\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44\" (UID: \"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" Oct 30 20:22:50.497659 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.497643 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44\" (UID: \"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" Oct 30 20:22:50.497957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.497879 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44\" (UID: \"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" Oct 30 20:22:50.599713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.599675 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44\" (UID: \"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" Oct 30 20:22:50.600245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.599777 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6whq8\" (UniqueName: \"kubernetes.io/projected/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-kube-api-access-6whq8\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44\" (UID: \"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" Oct 30 20:22:50.600245 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.599813 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44\" (UID: \"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" Oct 30 20:22:50.600573 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.600549 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44\" (UID: \"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" Oct 30 20:22:50.602115 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.602096 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44\" (UID: \"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" Oct 30 20:22:50.613986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.613958 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6whq8\" (UniqueName: \"kubernetes.io/projected/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-kube-api-access-6whq8\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44\" (UID: \"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" Oct 30 20:22:50.713891 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.713809 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" Oct 30 20:22:50.848713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:50.848638 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" podUID="18693447-e4c1-4975-be81-368f7b3d54ed" containerName="payload-processing" containerID="cri-o://9db49a4c7a5c19f47fc89422c25e8defb60a39aa355f44caef1d5ccbc006b608" gracePeriod=25 Oct 30 20:22:51.006428 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:22:51.006393 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fc10a4_2b0d_49f0_81e1_6b3fe39cc131.slice/crio-55ab7c28b52bf66ff115961820af5922955aaa043543240a279a3c958ebccfd0 WatchSource:0}: Error finding container 55ab7c28b52bf66ff115961820af5922955aaa043543240a279a3c958ebccfd0: Status 404 returned error can't find the container with id 55ab7c28b52bf66ff115961820af5922955aaa043543240a279a3c958ebccfd0 Oct 30 20:22:51.006564 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:51.006489 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44"] Oct 30 20:22:51.043896 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:51.043861 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" event={"ID":"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131","Type":"ContainerStarted","Data":"55ab7c28b52bf66ff115961820af5922955aaa043543240a279a3c958ebccfd0"} Oct 30 20:22:51.476670 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:51.476613 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:52.055955 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:52.055914 5091 generic.go:356] "Generic (PLEG): container finished" podID="98fc10a4-2b0d-49f0-81e1-6b3fe39cc131" containerID="6c0c0a37967925c47a8eda43de23d6431ca7d9a48a1780e4dd59b958b18b2cc9" exitCode=0 Oct 30 20:22:52.056398 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:52.055995 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" event={"ID":"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131","Type":"ContainerDied","Data":"6c0c0a37967925c47a8eda43de23d6431ca7d9a48a1780e4dd59b958b18b2cc9"} Oct 30 20:22:52.274975 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:52.274939 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:53.079584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:53.079544 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:53.238447 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:53.238422 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" Oct 30 20:22:53.341421 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:53.341334 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-maas-api-xks-ca\") pod \"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131\" (UID: \"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131\") " Oct 30 20:22:53.341595 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:53.341474 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-kube-api-access-6whq8\" (UniqueName: \"kubernetes.io/projected/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-kube-api-access-6whq8\") pod \"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131\" (UID: \"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131\") " Oct 30 20:22:53.341674 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:53.341620 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-maas-api-openshift-ca\") pod \"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131\" (UID: \"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131\") " Oct 30 20:22:53.342039 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:53.342011 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-maas-api-openshift-ca" pod "98fc10a4-2b0d-49f0-81e1-6b3fe39cc131" (UID: "98fc10a4-2b0d-49f0-81e1-6b3fe39cc131"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:53.343591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:53.343562 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-maas-api-xks-ca" pod "98fc10a4-2b0d-49f0-81e1-6b3fe39cc131" (UID: "98fc10a4-2b0d-49f0-81e1-6b3fe39cc131"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:22:53.343591 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:53.343583 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-kube-api-access-6whq8" pod "98fc10a4-2b0d-49f0-81e1-6b3fe39cc131" (UID: "98fc10a4-2b0d-49f0-81e1-6b3fe39cc131"). InnerVolumeSpecName "kube-api-access-6whq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:53.443548 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:53.443510 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:53.443548 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:53.443546 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:53.443548 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:53.443557 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6whq8\" (UniqueName: \"kubernetes.io/projected/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131-kube-api-access-6whq8\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:53.899197 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:53.894727 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:54.099259 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:54.099227 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" Oct 30 20:22:54.099652 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:54.099257 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44" event={"ID":"98fc10a4-2b0d-49f0-81e1-6b3fe39cc131","Type":"ContainerDied","Data":"55ab7c28b52bf66ff115961820af5922955aaa043543240a279a3c958ebccfd0"} Oct 30 20:22:54.099652 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:54.099307 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55ab7c28b52bf66ff115961820af5922955aaa043543240a279a3c958ebccfd0" Oct 30 20:22:54.545716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:54.545679 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" podUID="a60177f4-e29b-47ea-b5cd-3068808eb97f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:22:54.877102 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:54.877002 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:55.111317 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.111278 5091 generic.go:356] "Generic (PLEG): container finished" podID="ef039607-7c00-4a64-a63b-5e4a709ae26c" containerID="ffa299f2cbebfda0637d31301684a8a6fdfbacaf38f60ed498fa29be758b22ab" exitCode=0 Oct 30 20:22:55.111705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.111501 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" event={"ID":"ef039607-7c00-4a64-a63b-5e4a709ae26c","Type":"ContainerDied","Data":"ffa299f2cbebfda0637d31301684a8a6fdfbacaf38f60ed498fa29be758b22ab"} Oct 30 20:22:55.111705 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.111560 5091 scope.go:122] "RemoveContainer" containerID="9a9a889a72a06c51fdd605d87b56053127dfc54a5538c1125c7d684012e9b7fe" Oct 30 20:22:55.155052 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.155018 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:22:55.171391 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.171367 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-data\") pod \"ef039607-7c00-4a64-a63b-5e4a709ae26c\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " Oct 30 20:22:55.171544 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.171433 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-envoy\") pod \"ef039607-7c00-4a64-a63b-5e4a709ae26c\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " Oct 30 20:22:55.171544 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.171479 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-podinfo\") pod \"ef039607-7c00-4a64-a63b-5e4a709ae26c\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " Oct 30 20:22:55.171544 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.171522 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef039607-7c00-4a64-a63b-5e4a709ae26c-kube-api-access-xmzzl\" (UniqueName: \"kubernetes.io/projected/ef039607-7c00-4a64-a63b-5e4a709ae26c-kube-api-access-xmzzl\") pod \"ef039607-7c00-4a64-a63b-5e4a709ae26c\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " Oct 30 20:22:55.171731 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.171649 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-data" pod "ef039607-7c00-4a64-a63b-5e4a709ae26c" (UID: "ef039607-7c00-4a64-a63b-5e4a709ae26c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:55.171731 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.171672 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-workload-certs\") pod \"ef039607-7c00-4a64-a63b-5e4a709ae26c\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " Oct 30 20:22:55.171731 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.171712 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-credential-socket\") pod \"ef039607-7c00-4a64-a63b-5e4a709ae26c\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " Oct 30 20:22:55.171902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.171747 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef039607-7c00-4a64-a63b-5e4a709ae26c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef039607-7c00-4a64-a63b-5e4a709ae26c-istiod-ca-cert\") pod \"ef039607-7c00-4a64-a63b-5e4a709ae26c\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " Oct 30 20:22:55.171902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.171780 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-token\" (UniqueName: \"kubernetes.io/projected/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-token\") pod \"ef039607-7c00-4a64-a63b-5e4a709ae26c\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " Oct 30 20:22:55.171902 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.171855 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-workload-socket\") pod \"ef039607-7c00-4a64-a63b-5e4a709ae26c\" (UID: \"ef039607-7c00-4a64-a63b-5e4a709ae26c\") " Oct 30 20:22:55.172067 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.171952 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-workload-certs" pod "ef039607-7c00-4a64-a63b-5e4a709ae26c" (UID: "ef039607-7c00-4a64-a63b-5e4a709ae26c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:55.172204 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.172185 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-credential-socket" pod "ef039607-7c00-4a64-a63b-5e4a709ae26c" (UID: "ef039607-7c00-4a64-a63b-5e4a709ae26c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:55.172396 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.172378 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-workload-socket" pod "ef039607-7c00-4a64-a63b-5e4a709ae26c" (UID: "ef039607-7c00-4a64-a63b-5e4a709ae26c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:55.172452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.172395 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:55.172452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.172415 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:55.172452 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.172429 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:55.172621 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.172497 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef039607-7c00-4a64-a63b-5e4a709ae26c-istiod-ca-cert" pod "ef039607-7c00-4a64-a63b-5e4a709ae26c" (UID: "ef039607-7c00-4a64-a63b-5e4a709ae26c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:22:55.174038 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.174002 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef039607-7c00-4a64-a63b-5e4a709ae26c-kube-api-access-xmzzl" pod "ef039607-7c00-4a64-a63b-5e4a709ae26c" (UID: "ef039607-7c00-4a64-a63b-5e4a709ae26c"). InnerVolumeSpecName "kube-api-access-xmzzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:55.174182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.174126 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-podinfo" pod "ef039607-7c00-4a64-a63b-5e4a709ae26c" (UID: "ef039607-7c00-4a64-a63b-5e4a709ae26c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:22:55.174621 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.174591 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-envoy" pod "ef039607-7c00-4a64-a63b-5e4a709ae26c" (UID: "ef039607-7c00-4a64-a63b-5e4a709ae26c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:22:55.174621 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.174610 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-token" pod "ef039607-7c00-4a64-a63b-5e4a709ae26c" (UID: "ef039607-7c00-4a64-a63b-5e4a709ae26c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:22:55.273766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.273728 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef039607-7c00-4a64-a63b-5e4a709ae26c-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:55.273766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.273760 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:55.273979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.273774 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:55.273979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.273787 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:55.273979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.273799 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef039607-7c00-4a64-a63b-5e4a709ae26c-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:55.273979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.273811 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmzzl\" (UniqueName: \"kubernetes.io/projected/ef039607-7c00-4a64-a63b-5e4a709ae26c-kube-api-access-xmzzl\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:22:55.667181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:55.665754 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:56.123665 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:56.123637 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" Oct 30 20:22:56.124162 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:56.123679 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx" event={"ID":"ef039607-7c00-4a64-a63b-5e4a709ae26c","Type":"ContainerDied","Data":"3a13f9385940f2543ac6b56bf34f009dddffa6bac706238e5ed756281f245b36"} Oct 30 20:22:56.124162 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:56.123736 5091 scope.go:122] "RemoveContainer" containerID="ffa299f2cbebfda0637d31301684a8a6fdfbacaf38f60ed498fa29be758b22ab" Oct 30 20:22:56.157661 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:56.157630 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx"] Oct 30 20:22:56.162954 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:56.162928 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-4d5758-openshift-default-5d4977d97c-wrjdx"] Oct 30 20:22:56.300181 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:56.292929 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:56.443496 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:56.443409 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef039607-7c00-4a64-a63b-5e4a709ae26c" path="/var/lib/kubelet/pods/ef039607-7c00-4a64-a63b-5e4a709ae26c/volumes" Oct 30 20:22:56.462930 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:56.462896 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:56.556980 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:56.556949 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f"] Oct 30 20:22:56.557214 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:56.557189 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" podUID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerName="istio-proxy" containerID="cri-o://06d5f5db822feef54d4220bc708d3a9e052d9680ac974faa5775cca2b51a8c2a" gracePeriod=30 Oct 30 20:22:56.643498 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:56.643461 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:57.130980 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.130942 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:57.282632 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.282599 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq"] Oct 30 20:22:57.283141 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.283116 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ef039607-7c00-4a64-a63b-5e4a709ae26c" containerName="istio-proxy" Oct 30 20:22:57.283141 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.283154 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef039607-7c00-4a64-a63b-5e4a709ae26c" containerName="istio-proxy" Oct 30 20:22:57.283268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.283200 5091 cpu_manager.go:414] "containerMap: removing container" podUID="98fc10a4-2b0d-49f0-81e1-6b3fe39cc131" containerName="revoke-keys" Oct 30 20:22:57.283268 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.283207 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98fc10a4-2b0d-49f0-81e1-6b3fe39cc131" containerName="revoke-keys" Oct 30 20:22:57.283339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.283287 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef039607-7c00-4a64-a63b-5e4a709ae26c" containerName="istio-proxy" Oct 30 20:22:57.283339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.283299 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="98fc10a4-2b0d-49f0-81e1-6b3fe39cc131" containerName="revoke-keys" Oct 30 20:22:57.283339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.283307 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef039607-7c00-4a64-a63b-5e4a709ae26c" containerName="istio-proxy" Oct 30 20:22:57.288366 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.288346 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" Oct 30 20:22:57.294487 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.294460 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:22:57.294624 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.294524 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:22:57.296442 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.296420 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq"] Oct 30 20:22:57.298860 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.298838 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15ea609-33fa-42cc-9bc6-6b3217433d70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq\" (UID: \"a15ea609-33fa-42cc-9bc6-6b3217433d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" Oct 30 20:22:57.298946 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.298886 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpdqj\" (UniqueName: \"kubernetes.io/projected/a15ea609-33fa-42cc-9bc6-6b3217433d70-kube-api-access-qpdqj\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq\" (UID: \"a15ea609-33fa-42cc-9bc6-6b3217433d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" Oct 30 20:22:57.299001 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.298971 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15ea609-33fa-42cc-9bc6-6b3217433d70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq\" (UID: \"a15ea609-33fa-42cc-9bc6-6b3217433d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" Oct 30 20:22:57.400530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.400449 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15ea609-33fa-42cc-9bc6-6b3217433d70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq\" (UID: \"a15ea609-33fa-42cc-9bc6-6b3217433d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" Oct 30 20:22:57.400696 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.400608 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15ea609-33fa-42cc-9bc6-6b3217433d70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq\" (UID: \"a15ea609-33fa-42cc-9bc6-6b3217433d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" Oct 30 20:22:57.400696 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.400637 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpdqj\" (UniqueName: \"kubernetes.io/projected/a15ea609-33fa-42cc-9bc6-6b3217433d70-kube-api-access-qpdqj\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq\" (UID: \"a15ea609-33fa-42cc-9bc6-6b3217433d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" Oct 30 20:22:57.401254 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.401226 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15ea609-33fa-42cc-9bc6-6b3217433d70-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq\" (UID: \"a15ea609-33fa-42cc-9bc6-6b3217433d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" Oct 30 20:22:57.403019 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.402994 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15ea609-33fa-42cc-9bc6-6b3217433d70-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq\" (UID: \"a15ea609-33fa-42cc-9bc6-6b3217433d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" Oct 30 20:22:57.414964 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.414937 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpdqj\" (UniqueName: \"kubernetes.io/projected/a15ea609-33fa-42cc-9bc6-6b3217433d70-kube-api-access-qpdqj\") pod \"maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq\" (UID: \"a15ea609-33fa-42cc-9bc6-6b3217433d70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" Oct 30 20:22:57.605237 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.605199 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" Oct 30 20:22:57.680285 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.680156 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:57.915674 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:57.915646 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq"] Oct 30 20:22:57.916434 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:22:57.916406 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15ea609_33fa_42cc_9bc6_6b3217433d70.slice/crio-ad825df107293b0384daeaa17788a36f58b21257954f21f29cc354eada93ca05 WatchSource:0}: Error finding container ad825df107293b0384daeaa17788a36f58b21257954f21f29cc354eada93ca05: Status 404 returned error can't find the container with id ad825df107293b0384daeaa17788a36f58b21257954f21f29cc354eada93ca05 Oct 30 20:22:58.142574 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:58.142540 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" event={"ID":"a15ea609-33fa-42cc-9bc6-6b3217433d70","Type":"ContainerStarted","Data":"ad825df107293b0384daeaa17788a36f58b21257954f21f29cc354eada93ca05"} Oct 30 20:22:58.597221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:58.596204 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:22:58.656790 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:58.656741 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" podUID="18693447-e4c1-4975-be81-368f7b3d54ed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:22:59.153429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:59.153334 5091 generic.go:356] "Generic (PLEG): container finished" podID="a15ea609-33fa-42cc-9bc6-6b3217433d70" containerID="e90b4957720bf12427c66272b31ced68dee8a783e7cc6c8ec49910d133cf38bd" exitCode=0 Oct 30 20:22:59.153429 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:59.153414 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" event={"ID":"a15ea609-33fa-42cc-9bc6-6b3217433d70","Type":"ContainerDied","Data":"e90b4957720bf12427c66272b31ced68dee8a783e7cc6c8ec49910d133cf38bd"} Oct 30 20:22:59.480223 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:22:59.478996 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:00.327718 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:00.327688 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" Oct 30 20:23:00.384287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:00.384223 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:00.436549 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:00.436461 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15ea609-33fa-42cc-9bc6-6b3217433d70-kube-api-access-qpdqj\" (UniqueName: \"kubernetes.io/projected/a15ea609-33fa-42cc-9bc6-6b3217433d70-kube-api-access-qpdqj\") pod \"a15ea609-33fa-42cc-9bc6-6b3217433d70\" (UID: \"a15ea609-33fa-42cc-9bc6-6b3217433d70\") " Oct 30 20:23:00.436747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:00.436693 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15ea609-33fa-42cc-9bc6-6b3217433d70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15ea609-33fa-42cc-9bc6-6b3217433d70-maas-api-openshift-ca\") pod \"a15ea609-33fa-42cc-9bc6-6b3217433d70\" (UID: \"a15ea609-33fa-42cc-9bc6-6b3217433d70\") " Oct 30 20:23:00.436747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:00.436730 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a15ea609-33fa-42cc-9bc6-6b3217433d70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15ea609-33fa-42cc-9bc6-6b3217433d70-maas-api-xks-ca\") pod \"a15ea609-33fa-42cc-9bc6-6b3217433d70\" (UID: \"a15ea609-33fa-42cc-9bc6-6b3217433d70\") " Oct 30 20:23:00.437102 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:00.437071 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15ea609-33fa-42cc-9bc6-6b3217433d70-maas-api-openshift-ca" pod "a15ea609-33fa-42cc-9bc6-6b3217433d70" (UID: "a15ea609-33fa-42cc-9bc6-6b3217433d70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:00.437320 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:00.437276 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a15ea609-33fa-42cc-9bc6-6b3217433d70-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:00.438961 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:00.438936 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15ea609-33fa-42cc-9bc6-6b3217433d70-kube-api-access-qpdqj" pod "a15ea609-33fa-42cc-9bc6-6b3217433d70" (UID: "a15ea609-33fa-42cc-9bc6-6b3217433d70"). InnerVolumeSpecName "kube-api-access-qpdqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:00.439180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:00.439133 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a15ea609-33fa-42cc-9bc6-6b3217433d70-maas-api-xks-ca" pod "a15ea609-33fa-42cc-9bc6-6b3217433d70" (UID: "a15ea609-33fa-42cc-9bc6-6b3217433d70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:23:00.538059 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:00.538021 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a15ea609-33fa-42cc-9bc6-6b3217433d70-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:00.538059 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:00.538055 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpdqj\" (UniqueName: \"kubernetes.io/projected/a15ea609-33fa-42cc-9bc6-6b3217433d70-kube-api-access-qpdqj\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:00.561783 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:00.561751 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd"] Oct 30 20:23:00.674722 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:00.674681 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:00.822849 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:00.820000 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:01.174026 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:01.173925 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" event={"ID":"a15ea609-33fa-42cc-9bc6-6b3217433d70","Type":"ContainerDied","Data":"ad825df107293b0384daeaa17788a36f58b21257954f21f29cc354eada93ca05"} Oct 30 20:23:01.174026 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:01.174011 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad825df107293b0384daeaa17788a36f58b21257954f21f29cc354eada93ca05" Oct 30 20:23:01.174026 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:01.173945 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq" Oct 30 20:23:01.268401 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:01.268359 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:02.026085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.026041 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:02.233604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.232955 5091 generic.go:356] "Generic (PLEG): container finished" podID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerID="06d5f5db822feef54d4220bc708d3a9e052d9680ac974faa5775cca2b51a8c2a" exitCode=0 Oct 30 20:23:02.233604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.233130 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" event={"ID":"3d5e6444-233c-4c4e-b82f-559b46e46a38","Type":"ContainerDied","Data":"06d5f5db822feef54d4220bc708d3a9e052d9680ac974faa5775cca2b51a8c2a"} Oct 30 20:23:02.233604 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.233294 5091 scope.go:122] "RemoveContainer" containerID="4ca9093b95260f9f6ff042b6e39d3dac8c32c65ef715112cebd541e805140ce5" Oct 30 20:23:02.235373 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.235341 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:02.312234 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.312091 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44"] Oct 30 20:23:02.315569 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.315537 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-4d2317-8vg44"] Oct 30 20:23:02.346538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.346501 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 20:23:02.389121 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.389077 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:02.452397 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.452355 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98fc10a4-2b0d-49f0-81e1-6b3fe39cc131" path="/var/lib/kubelet/pods/98fc10a4-2b0d-49f0-81e1-6b3fe39cc131/volumes" Oct 30 20:23:02.500696 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.500641 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-token\" (UniqueName: \"kubernetes.io/projected/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-token\") pod \"3d5e6444-233c-4c4e-b82f-559b46e46a38\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " Oct 30 20:23:02.500944 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.500776 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d5e6444-233c-4c4e-b82f-559b46e46a38-kube-api-access-96mkg\" (UniqueName: \"kubernetes.io/projected/3d5e6444-233c-4c4e-b82f-559b46e46a38-kube-api-access-96mkg\") pod \"3d5e6444-233c-4c4e-b82f-559b46e46a38\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " Oct 30 20:23:02.500944 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.500876 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-workload-certs\") pod \"3d5e6444-233c-4c4e-b82f-559b46e46a38\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " Oct 30 20:23:02.501089 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.500961 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-credential-socket\") pod \"3d5e6444-233c-4c4e-b82f-559b46e46a38\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " Oct 30 20:23:02.501089 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.501019 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-podinfo\") pod \"3d5e6444-233c-4c4e-b82f-559b46e46a38\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " Oct 30 20:23:02.501271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.501182 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-data\") pod \"3d5e6444-233c-4c4e-b82f-559b46e46a38\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " Oct 30 20:23:02.501271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.501238 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-envoy\") pod \"3d5e6444-233c-4c4e-b82f-559b46e46a38\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " Oct 30 20:23:02.501271 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.501255 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-workload-certs" pod "3d5e6444-233c-4c4e-b82f-559b46e46a38" (UID: "3d5e6444-233c-4c4e-b82f-559b46e46a38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:02.501499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.501281 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-workload-socket\") pod \"3d5e6444-233c-4c4e-b82f-559b46e46a38\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " Oct 30 20:23:02.501499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.501311 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d5e6444-233c-4c4e-b82f-559b46e46a38-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5e6444-233c-4c4e-b82f-559b46e46a38-istiod-ca-cert\") pod \"3d5e6444-233c-4c4e-b82f-559b46e46a38\" (UID: \"3d5e6444-233c-4c4e-b82f-559b46e46a38\") " Oct 30 20:23:02.501801 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.501766 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:02.502122 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.502077 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-data" pod "3d5e6444-233c-4c4e-b82f-559b46e46a38" (UID: "3d5e6444-233c-4c4e-b82f-559b46e46a38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:02.502374 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.502306 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-credential-socket" pod "3d5e6444-233c-4c4e-b82f-559b46e46a38" (UID: "3d5e6444-233c-4c4e-b82f-559b46e46a38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:02.502622 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.502502 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-workload-socket" pod "3d5e6444-233c-4c4e-b82f-559b46e46a38" (UID: "3d5e6444-233c-4c4e-b82f-559b46e46a38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:02.502968 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.502926 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d5e6444-233c-4c4e-b82f-559b46e46a38-istiod-ca-cert" pod "3d5e6444-233c-4c4e-b82f-559b46e46a38" (UID: "3d5e6444-233c-4c4e-b82f-559b46e46a38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:02.504006 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.503978 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-token" pod "3d5e6444-233c-4c4e-b82f-559b46e46a38" (UID: "3d5e6444-233c-4c4e-b82f-559b46e46a38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:02.504278 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.504251 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d5e6444-233c-4c4e-b82f-559b46e46a38-kube-api-access-96mkg" pod "3d5e6444-233c-4c4e-b82f-559b46e46a38" (UID: "3d5e6444-233c-4c4e-b82f-559b46e46a38"). InnerVolumeSpecName "kube-api-access-96mkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:02.504729 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.504701 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-podinfo" pod "3d5e6444-233c-4c4e-b82f-559b46e46a38" (UID: "3d5e6444-233c-4c4e-b82f-559b46e46a38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:23:02.504890 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.504871 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-envoy" pod "3d5e6444-233c-4c4e-b82f-559b46e46a38" (UID: "3d5e6444-233c-4c4e-b82f-559b46e46a38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:02.540713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.540684 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:02.603626 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.603543 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:02.603626 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.603573 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:02.603626 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.603590 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:02.603626 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.603603 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:02.603626 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.603616 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3d5e6444-233c-4c4e-b82f-559b46e46a38-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:02.603913 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.603630 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3d5e6444-233c-4c4e-b82f-559b46e46a38-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:02.603913 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.603645 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96mkg\" (UniqueName: \"kubernetes.io/projected/3d5e6444-233c-4c4e-b82f-559b46e46a38-kube-api-access-96mkg\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:02.603913 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.603662 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3d5e6444-233c-4c4e-b82f-559b46e46a38-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:02.862761 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:02.862632 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:03.243571 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:03.243535 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" event={"ID":"3d5e6444-233c-4c4e-b82f-559b46e46a38","Type":"ContainerDied","Data":"98d84e65a7f29b73c729921679f7e179a81a5ce9192267620324f7317626e6be"} Oct 30 20:23:03.244105 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:03.243606 5091 scope.go:122] "RemoveContainer" containerID="06d5f5db822feef54d4220bc708d3a9e052d9680ac974faa5775cca2b51a8c2a" Oct 30 20:23:03.244105 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:03.243782 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f" Oct 30 20:23:03.249509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:03.249478 5091 generic.go:356] "Generic (PLEG): container finished" podID="a60177f4-e29b-47ea-b5cd-3068808eb97f" containerID="a592671d365bea5105d7ffbe294a02675cf26c6c717bdf0eac13025f7254dd91" exitCode=0 Oct 30 20:23:03.249643 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:03.249554 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" event={"ID":"a60177f4-e29b-47ea-b5cd-3068808eb97f","Type":"ContainerDied","Data":"a592671d365bea5105d7ffbe294a02675cf26c6c717bdf0eac13025f7254dd91"} Oct 30 20:23:03.270700 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:03.270675 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" Oct 30 20:23:03.284215 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:03.284181 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f"] Oct 30 20:23:03.289796 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:03.289767 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-981ee5-openshift-default-cb9d68d7c-ljs5f"] Oct 30 20:23:03.414180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:03.414065 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a60177f4-e29b-47ea-b5cd-3068808eb97f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60177f4-e29b-47ea-b5cd-3068808eb97f-plugins-config-volume\") pod \"a60177f4-e29b-47ea-b5cd-3068808eb97f\" (UID: \"a60177f4-e29b-47ea-b5cd-3068808eb97f\") " Oct 30 20:23:03.414337 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:03.414293 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a60177f4-e29b-47ea-b5cd-3068808eb97f-kube-api-access-p4pcd\" (UniqueName: \"kubernetes.io/projected/a60177f4-e29b-47ea-b5cd-3068808eb97f-kube-api-access-p4pcd\") pod \"a60177f4-e29b-47ea-b5cd-3068808eb97f\" (UID: \"a60177f4-e29b-47ea-b5cd-3068808eb97f\") " Oct 30 20:23:03.414549 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:03.414521 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a60177f4-e29b-47ea-b5cd-3068808eb97f-plugins-config-volume" pod "a60177f4-e29b-47ea-b5cd-3068808eb97f" (UID: "a60177f4-e29b-47ea-b5cd-3068808eb97f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:03.414821 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:03.414786 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a60177f4-e29b-47ea-b5cd-3068808eb97f-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:03.416640 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:03.416613 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a60177f4-e29b-47ea-b5cd-3068808eb97f-kube-api-access-p4pcd" pod "a60177f4-e29b-47ea-b5cd-3068808eb97f" (UID: "a60177f4-e29b-47ea-b5cd-3068808eb97f"). InnerVolumeSpecName "kube-api-access-p4pcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:03.492104 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:03.492062 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:03.516518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:03.516484 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4pcd\" (UniqueName: \"kubernetes.io/projected/a60177f4-e29b-47ea-b5cd-3068808eb97f-kube-api-access-p4pcd\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:04.259235 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:04.259195 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" event={"ID":"a60177f4-e29b-47ea-b5cd-3068808eb97f","Type":"ContainerDied","Data":"e0891d4f44aa9b716011bbe727463f0d6cf2f2b625d2fcd24ceb7af433dc9b05"} Oct 30 20:23:04.259700 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:04.259273 5091 scope.go:122] "RemoveContainer" containerID="a592671d365bea5105d7ffbe294a02675cf26c6c717bdf0eac13025f7254dd91" Oct 30 20:23:04.259700 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:04.259208 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6" Oct 30 20:23:04.294939 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:04.294908 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6"] Oct 30 20:23:04.297921 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:04.297895 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-4d5758-66cf9d6b6d-cntb6"] Oct 30 20:23:04.367719 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:04.367594 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:04.440509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:04.440477 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d5e6444-233c-4c4e-b82f-559b46e46a38" path="/var/lib/kubelet/pods/3d5e6444-233c-4c4e-b82f-559b46e46a38/volumes" Oct 30 20:23:04.441073 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:04.441052 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a60177f4-e29b-47ea-b5cd-3068808eb97f" path="/var/lib/kubelet/pods/a60177f4-e29b-47ea-b5cd-3068808eb97f/volumes" Oct 30 20:23:05.171851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:05.171191 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:05.562973 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:05.562917 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" podUID="26191153-8910-4ca1-ad13-54f97b744a37" containerName="payload-pre-processing" containerID="cri-o://8bd9c240cd6031686fb903e8d906579554e1afce39786023b048b028a1f05ec7" gracePeriod=25 Oct 30 20:23:05.840790 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:05.840762 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" Oct 30 20:23:05.943747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:05.943709 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26191153-8910-4ca1-ad13-54f97b744a37-kube-api-access-ckcss\" (UniqueName: \"kubernetes.io/projected/26191153-8910-4ca1-ad13-54f97b744a37-kube-api-access-ckcss\") pod \"26191153-8910-4ca1-ad13-54f97b744a37\" (UID: \"26191153-8910-4ca1-ad13-54f97b744a37\") " Oct 30 20:23:05.943978 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:05.943947 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26191153-8910-4ca1-ad13-54f97b744a37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26191153-8910-4ca1-ad13-54f97b744a37-plugins-config-volume\") pod \"26191153-8910-4ca1-ad13-54f97b744a37\" (UID: \"26191153-8910-4ca1-ad13-54f97b744a37\") " Oct 30 20:23:05.944805 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:05.944717 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26191153-8910-4ca1-ad13-54f97b744a37-plugins-config-volume" pod "26191153-8910-4ca1-ad13-54f97b744a37" (UID: "26191153-8910-4ca1-ad13-54f97b744a37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:05.947732 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:05.947706 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26191153-8910-4ca1-ad13-54f97b744a37-kube-api-access-ckcss" pod "26191153-8910-4ca1-ad13-54f97b744a37" (UID: "26191153-8910-4ca1-ad13-54f97b744a37"). InnerVolumeSpecName "kube-api-access-ckcss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:06.045633 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:06.045588 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26191153-8910-4ca1-ad13-54f97b744a37-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:06.045633 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:06.045628 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckcss\" (UniqueName: \"kubernetes.io/projected/26191153-8910-4ca1-ad13-54f97b744a37-kube-api-access-ckcss\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:06.162876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:06.161926 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:06.278869 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:06.278833 5091 generic.go:356] "Generic (PLEG): container finished" podID="26191153-8910-4ca1-ad13-54f97b744a37" containerID="8bd9c240cd6031686fb903e8d906579554e1afce39786023b048b028a1f05ec7" exitCode=0 Oct 30 20:23:06.279049 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:06.278886 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" Oct 30 20:23:06.279049 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:06.278915 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" event={"ID":"26191153-8910-4ca1-ad13-54f97b744a37","Type":"ContainerDied","Data":"8bd9c240cd6031686fb903e8d906579554e1afce39786023b048b028a1f05ec7"} Oct 30 20:23:06.279049 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:06.278972 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd" event={"ID":"26191153-8910-4ca1-ad13-54f97b744a37","Type":"ContainerDied","Data":"6ecd75ba9151459e6d4a29a4c443d0ca394df93c93b2ce969e72a981a1b48b9d"} Oct 30 20:23:06.279049 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:06.278997 5091 scope.go:122] "RemoveContainer" containerID="8bd9c240cd6031686fb903e8d906579554e1afce39786023b048b028a1f05ec7" Oct 30 20:23:06.290882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:06.290856 5091 scope.go:122] "RemoveContainer" containerID="8bd9c240cd6031686fb903e8d906579554e1afce39786023b048b028a1f05ec7" Oct 30 20:23:06.291189 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:06.291158 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bd9c240cd6031686fb903e8d906579554e1afce39786023b048b028a1f05ec7\": container with ID starting with 8bd9c240cd6031686fb903e8d906579554e1afce39786023b048b028a1f05ec7 not found: ID does not exist" containerID="8bd9c240cd6031686fb903e8d906579554e1afce39786023b048b028a1f05ec7" Oct 30 20:23:06.291305 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:06.291196 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bd9c240cd6031686fb903e8d906579554e1afce39786023b048b028a1f05ec7"} err="failed to get container status \"8bd9c240cd6031686fb903e8d906579554e1afce39786023b048b028a1f05ec7\": rpc error: code = NotFound desc = could not find container \"8bd9c240cd6031686fb903e8d906579554e1afce39786023b048b028a1f05ec7\": container with ID starting with 8bd9c240cd6031686fb903e8d906579554e1afce39786023b048b028a1f05ec7 not found: ID does not exist" Oct 30 20:23:06.307499 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:06.307473 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd"] Oct 30 20:23:06.311723 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:06.311701 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-4d2317-7c657ff4f5-mzsbd"] Oct 30 20:23:06.441629 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:06.441541 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26191153-8910-4ca1-ad13-54f97b744a37" path="/var/lib/kubelet/pods/26191153-8910-4ca1-ad13-54f97b744a37/volumes" Oct 30 20:23:06.986362 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:06.982320 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:07.467277 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:07.467242 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk"] Oct 30 20:23:07.487118 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:07.487082 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc"] Oct 30 20:23:07.565437 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:07.565400 5091 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-worker-w1-7a745c: configmap "payload-processing-plugins-e2e-worker-w1-7a745c" not found Oct 30 20:23:07.565617 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:07.565489 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume podName:f8550b23-cb62-4ddb-90b9-8d2b0706dd20 nodeName:}" failed. No retries permitted until 2026-10-30 20:23:08.065464954 +0000 UTC m=+3107.760672906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume") pod "payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" (UID: "f8550b23-cb62-4ddb-90b9-8d2b0706dd20") : configmap "payload-processing-plugins-e2e-worker-w1-7a745c" not found Oct 30 20:23:07.590108 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:07.590068 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:07.770354 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:07.770319 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:08.038823 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:08.038736 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:08.081697 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:08.081659 5091 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-worker-w1-7a745c: configmap "payload-processing-plugins-e2e-worker-w1-7a745c" not found Oct 30 20:23:08.081872 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:08.081739 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume podName:f8550b23-cb62-4ddb-90b9-8d2b0706dd20 nodeName:}" failed. No retries permitted until 2026-10-30 20:23:09.081725698 +0000 UTC m=+3108.776933634 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume") pod "payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" (UID: "f8550b23-cb62-4ddb-90b9-8d2b0706dd20") : configmap "payload-processing-plugins-e2e-worker-w1-7a745c" not found Oct 30 20:23:08.656954 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:08.656911 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" podUID="18693447-e4c1-4975-be81-368f7b3d54ed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:23:08.782341 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:08.782302 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:09.036881 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:09.036843 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv"] Oct 30 20:23:09.039639 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:09.039614 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-64d5e97e-zszcv"] Oct 30 20:23:09.042099 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:09.042075 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp"] Oct 30 20:23:09.044840 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:09.044821 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d11adfc-lc7bp"] Oct 30 20:23:09.094566 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:09.094528 5091 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-worker-w1-7a745c: configmap "payload-processing-plugins-e2e-worker-w1-7a745c" not found Oct 30 20:23:09.094767 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:09.094605 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume podName:f8550b23-cb62-4ddb-90b9-8d2b0706dd20 nodeName:}" failed. No retries permitted until 2026-10-30 20:23:11.094586703 +0000 UTC m=+3110.789794661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume") pod "payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" (UID: "f8550b23-cb62-4ddb-90b9-8d2b0706dd20") : configmap "payload-processing-plugins-e2e-worker-w1-7a745c" not found Oct 30 20:23:09.684856 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:09.684820 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:10.368939 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.368905 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m"] Oct 30 20:23:10.369414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369399 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a60177f4-e29b-47ea-b5cd-3068808eb97f" containerName="payload-processing" Oct 30 20:23:10.369461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369415 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a60177f4-e29b-47ea-b5cd-3068808eb97f" containerName="payload-processing" Oct 30 20:23:10.369461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369436 5091 cpu_manager.go:414] "containerMap: removing container" podUID="26191153-8910-4ca1-ad13-54f97b744a37" containerName="payload-pre-processing" Oct 30 20:23:10.369461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369441 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26191153-8910-4ca1-ad13-54f97b744a37" containerName="payload-pre-processing" Oct 30 20:23:10.369461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369458 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerName="istio-proxy" Oct 30 20:23:10.369605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369463 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerName="istio-proxy" Oct 30 20:23:10.369605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369472 5091 cpu_manager.go:414] "containerMap: removing container" podUID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerName="istio-proxy" Oct 30 20:23:10.369605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369477 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerName="istio-proxy" Oct 30 20:23:10.369605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369486 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ef039607-7c00-4a64-a63b-5e4a709ae26c" containerName="istio-proxy" Oct 30 20:23:10.369605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369491 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef039607-7c00-4a64-a63b-5e4a709ae26c" containerName="istio-proxy" Oct 30 20:23:10.369605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369502 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a15ea609-33fa-42cc-9bc6-6b3217433d70" containerName="revoke-keys" Oct 30 20:23:10.369605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369507 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15ea609-33fa-42cc-9bc6-6b3217433d70" containerName="revoke-keys" Oct 30 20:23:10.369605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369575 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a60177f4-e29b-47ea-b5cd-3068808eb97f" containerName="payload-processing" Oct 30 20:23:10.369605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369586 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="26191153-8910-4ca1-ad13-54f97b744a37" containerName="payload-pre-processing" Oct 30 20:23:10.369605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369594 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerName="istio-proxy" Oct 30 20:23:10.369605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369601 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d5e6444-233c-4c4e-b82f-559b46e46a38" containerName="istio-proxy" Oct 30 20:23:10.369605 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.369609 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15ea609-33fa-42cc-9bc6-6b3217433d70" containerName="revoke-keys" Oct 30 20:23:10.374488 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.374464 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" Oct 30 20:23:10.383224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.383200 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:23:10.383411 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.383384 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m"] Oct 30 20:23:10.384853 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.384833 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:23:10.441494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.441454 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829" path="/var/lib/kubelet/pods/bd2b74c7-1f59-4e6f-ba87-eb7f9ea6b829/volumes" Oct 30 20:23:10.441960 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.441938 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4d0a6d4-3f36-4614-be89-64a2585183ca" path="/var/lib/kubelet/pods/c4d0a6d4-3f36-4614-be89-64a2585183ca/volumes" Oct 30 20:23:10.509136 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.509101 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e50c786-95a9-42b3-94b2-b894ee74e33a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-712c6641-2c48m\" (UID: \"2e50c786-95a9-42b3-94b2-b894ee74e33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" Oct 30 20:23:10.509136 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.509155 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6brzl\" (UniqueName: \"kubernetes.io/projected/2e50c786-95a9-42b3-94b2-b894ee74e33a-kube-api-access-6brzl\") pod \"maas-api-revoke-sub-712c6641-2c48m\" (UID: \"2e50c786-95a9-42b3-94b2-b894ee74e33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" Oct 30 20:23:10.509520 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.509497 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e50c786-95a9-42b3-94b2-b894ee74e33a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-712c6641-2c48m\" (UID: \"2e50c786-95a9-42b3-94b2-b894ee74e33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" Oct 30 20:23:10.611019 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.610977 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e50c786-95a9-42b3-94b2-b894ee74e33a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-712c6641-2c48m\" (UID: \"2e50c786-95a9-42b3-94b2-b894ee74e33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" Oct 30 20:23:10.611252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.611049 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e50c786-95a9-42b3-94b2-b894ee74e33a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-712c6641-2c48m\" (UID: \"2e50c786-95a9-42b3-94b2-b894ee74e33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" Oct 30 20:23:10.611252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.611075 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6brzl\" (UniqueName: \"kubernetes.io/projected/2e50c786-95a9-42b3-94b2-b894ee74e33a-kube-api-access-6brzl\") pod \"maas-api-revoke-sub-712c6641-2c48m\" (UID: \"2e50c786-95a9-42b3-94b2-b894ee74e33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" Oct 30 20:23:10.611944 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.611909 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e50c786-95a9-42b3-94b2-b894ee74e33a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-712c6641-2c48m\" (UID: \"2e50c786-95a9-42b3-94b2-b894ee74e33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" Oct 30 20:23:10.613622 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.613599 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e50c786-95a9-42b3-94b2-b894ee74e33a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-712c6641-2c48m\" (UID: \"2e50c786-95a9-42b3-94b2-b894ee74e33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" Oct 30 20:23:10.626191 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.626123 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6brzl\" (UniqueName: \"kubernetes.io/projected/2e50c786-95a9-42b3-94b2-b894ee74e33a-kube-api-access-6brzl\") pod \"maas-api-revoke-sub-712c6641-2c48m\" (UID: \"2e50c786-95a9-42b3-94b2-b894ee74e33a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" Oct 30 20:23:10.696704 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.696663 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" Oct 30 20:23:10.718921 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.718870 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:10.875968 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.875809 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:10.990318 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:10.990288 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m"] Oct 30 20:23:10.991634 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:23:10.991607 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e50c786_95a9_42b3_94b2_b894ee74e33a.slice/crio-a41534f1e91c6c227ed1431123b9dc5416716bffed4f9799e290ce1b27385e1b WatchSource:0}: Error finding container a41534f1e91c6c227ed1431123b9dc5416716bffed4f9799e290ce1b27385e1b: Status 404 returned error can't find the container with id a41534f1e91c6c227ed1431123b9dc5416716bffed4f9799e290ce1b27385e1b Oct 30 20:23:11.118405 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:11.118372 5091 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-worker-w1-7a745c: configmap "payload-processing-plugins-e2e-worker-w1-7a745c" not found Oct 30 20:23:11.118581 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:11.118442 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume podName:f8550b23-cb62-4ddb-90b9-8d2b0706dd20 nodeName:}" failed. No retries permitted until 2026-10-30 20:23:15.11842787 +0000 UTC m=+3114.813635806 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume") pod "payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" (UID: "f8550b23-cb62-4ddb-90b9-8d2b0706dd20") : configmap "payload-processing-plugins-e2e-worker-w1-7a745c" not found Oct 30 20:23:11.144732 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.144643 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm"] Oct 30 20:23:11.145381 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.144894 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" podUID="ec317404-f9f4-499c-bdbe-65109b6a18d2" containerName="istio-proxy" containerID="cri-o://4012fdf9bce4045b0905513abedc184e3277bd538e5ff2dbcb8fa38a21195e76" gracePeriod=30 Oct 30 20:23:11.163339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.163314 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" Oct 30 20:23:11.281164 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.280981 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:11.320024 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.319976 5091 generic.go:356] "Generic (PLEG): container finished" podID="18693447-e4c1-4975-be81-368f7b3d54ed" containerID="9db49a4c7a5c19f47fc89422c25e8defb60a39aa355f44caef1d5ccbc006b608" exitCode=0 Oct 30 20:23:11.320024 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.320020 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" Oct 30 20:23:11.320312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.320054 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" event={"ID":"18693447-e4c1-4975-be81-368f7b3d54ed","Type":"ContainerDied","Data":"9db49a4c7a5c19f47fc89422c25e8defb60a39aa355f44caef1d5ccbc006b608"} Oct 30 20:23:11.320312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.320111 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v" event={"ID":"18693447-e4c1-4975-be81-368f7b3d54ed","Type":"ContainerDied","Data":"96817b6efbf20be92fbf1af064c89d7c723d6518ffed68952d27634c4764eba4"} Oct 30 20:23:11.320312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.320139 5091 scope.go:122] "RemoveContainer" containerID="9db49a4c7a5c19f47fc89422c25e8defb60a39aa355f44caef1d5ccbc006b608" Oct 30 20:23:11.320312 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.320185 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18693447-e4c1-4975-be81-368f7b3d54ed-kube-api-access-m7ndb\" (UniqueName: \"kubernetes.io/projected/18693447-e4c1-4975-be81-368f7b3d54ed-kube-api-access-m7ndb\") pod \"18693447-e4c1-4975-be81-368f7b3d54ed\" (UID: \"18693447-e4c1-4975-be81-368f7b3d54ed\") " Oct 30 20:23:11.320556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.320409 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18693447-e4c1-4975-be81-368f7b3d54ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18693447-e4c1-4975-be81-368f7b3d54ed-plugins-config-volume\") pod \"18693447-e4c1-4975-be81-368f7b3d54ed\" (UID: \"18693447-e4c1-4975-be81-368f7b3d54ed\") " Oct 30 20:23:11.320823 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.320796 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18693447-e4c1-4975-be81-368f7b3d54ed-plugins-config-volume" pod "18693447-e4c1-4975-be81-368f7b3d54ed" (UID: "18693447-e4c1-4975-be81-368f7b3d54ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:11.322246 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.322214 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" event={"ID":"2e50c786-95a9-42b3-94b2-b894ee74e33a","Type":"ContainerStarted","Data":"a41534f1e91c6c227ed1431123b9dc5416716bffed4f9799e290ce1b27385e1b"} Oct 30 20:23:11.322776 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.322755 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18693447-e4c1-4975-be81-368f7b3d54ed-kube-api-access-m7ndb" pod "18693447-e4c1-4975-be81-368f7b3d54ed" (UID: "18693447-e4c1-4975-be81-368f7b3d54ed"). InnerVolumeSpecName "kube-api-access-m7ndb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:11.331334 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.331309 5091 scope.go:122] "RemoveContainer" containerID="9db49a4c7a5c19f47fc89422c25e8defb60a39aa355f44caef1d5ccbc006b608" Oct 30 20:23:11.331615 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:11.331589 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9db49a4c7a5c19f47fc89422c25e8defb60a39aa355f44caef1d5ccbc006b608\": container with ID starting with 9db49a4c7a5c19f47fc89422c25e8defb60a39aa355f44caef1d5ccbc006b608 not found: ID does not exist" containerID="9db49a4c7a5c19f47fc89422c25e8defb60a39aa355f44caef1d5ccbc006b608" Oct 30 20:23:11.331710 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.331624 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9db49a4c7a5c19f47fc89422c25e8defb60a39aa355f44caef1d5ccbc006b608"} err="failed to get container status \"9db49a4c7a5c19f47fc89422c25e8defb60a39aa355f44caef1d5ccbc006b608\": rpc error: code = NotFound desc = could not find container \"9db49a4c7a5c19f47fc89422c25e8defb60a39aa355f44caef1d5ccbc006b608\": container with ID starting with 9db49a4c7a5c19f47fc89422c25e8defb60a39aa355f44caef1d5ccbc006b608 not found: ID does not exist" Oct 30 20:23:11.422703 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.422622 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18693447-e4c1-4975-be81-368f7b3d54ed-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:11.422703 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.422653 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7ndb\" (UniqueName: \"kubernetes.io/projected/18693447-e4c1-4975-be81-368f7b3d54ed-kube-api-access-m7ndb\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:11.663178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.663133 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v"] Oct 30 20:23:11.667342 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.667313 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-981ee5-684c7cfc78-bxr9v"] Oct 30 20:23:11.701048 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.700979 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s"] Oct 30 20:23:11.701538 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.701523 5091 cpu_manager.go:414] "containerMap: removing container" podUID="18693447-e4c1-4975-be81-368f7b3d54ed" containerName="payload-processing" Oct 30 20:23:11.701587 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.701539 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18693447-e4c1-4975-be81-368f7b3d54ed" containerName="payload-processing" Oct 30 20:23:11.701645 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.701636 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="18693447-e4c1-4975-be81-368f7b3d54ed" containerName="payload-processing" Oct 30 20:23:11.704889 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.704873 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" Oct 30 20:23:11.754227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.750696 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s"] Oct 30 20:23:11.754227 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.752170 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:11.851969 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.851933 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s\" (UID: \"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" Oct 30 20:23:11.852155 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.852074 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s\" (UID: \"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" Oct 30 20:23:11.852218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.852201 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9hlk\" (UniqueName: \"kubernetes.io/projected/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-kube-api-access-b9hlk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s\" (UID: \"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" Oct 30 20:23:11.953514 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.953432 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9hlk\" (UniqueName: \"kubernetes.io/projected/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-kube-api-access-b9hlk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s\" (UID: \"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" Oct 30 20:23:11.953682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.953538 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s\" (UID: \"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" Oct 30 20:23:11.953682 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.953603 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s\" (UID: \"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" Oct 30 20:23:11.954272 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.954249 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s\" (UID: \"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" Oct 30 20:23:11.955979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.955949 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s\" (UID: \"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" Oct 30 20:23:11.968477 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:11.968451 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9hlk\" (UniqueName: \"kubernetes.io/projected/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-kube-api-access-b9hlk\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s\" (UID: \"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" Oct 30 20:23:12.053420 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:12.053383 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" Oct 30 20:23:12.331957 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:12.331913 5091 generic.go:356] "Generic (PLEG): container finished" podID="2e50c786-95a9-42b3-94b2-b894ee74e33a" containerID="3ce8308b9a8e8b234095459df0b60a42006dacbb659399847099a3bf3af49612" exitCode=0 Oct 30 20:23:12.332174 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:12.331999 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" event={"ID":"2e50c786-95a9-42b3-94b2-b894ee74e33a","Type":"ContainerDied","Data":"3ce8308b9a8e8b234095459df0b60a42006dacbb659399847099a3bf3af49612"} Oct 30 20:23:12.344056 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:12.344030 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s"] Oct 30 20:23:12.344918 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:23:12.344894 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13e1b14_87e8_41e4_99ec_3cfe6e68f7a1.slice/crio-4e5da732314bdfb7194990fff171508fe3115f437bcb8ff8704101e6d652e651 WatchSource:0}: Error finding container 4e5da732314bdfb7194990fff171508fe3115f437bcb8ff8704101e6d652e651: Status 404 returned error can't find the container with id 4e5da732314bdfb7194990fff171508fe3115f437bcb8ff8704101e6d652e651 Oct 30 20:23:12.440862 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:12.440825 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18693447-e4c1-4975-be81-368f7b3d54ed" path="/var/lib/kubelet/pods/18693447-e4c1-4975-be81-368f7b3d54ed/volumes" Oct 30 20:23:12.467747 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:12.467702 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" podUID="405da951-aef4-4187-8755-b264fff0c2a0" containerName="payload-processing" containerID="cri-o://9885d65fc47a9858488a86c9c29c6839ce1b3063d236382ae821e0318aaefc61" gracePeriod=25 Oct 30 20:23:12.675219 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:12.673078 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:13.296716 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:13.296678 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" podUID="f8550b23-cb62-4ddb-90b9-8d2b0706dd20" containerName="payload-pre-processing" containerID="cri-o://29e268e0fb5627d4bed7385eee0ab553f70cad063702b3398fbce6cda92c94ce" gracePeriod=25 Oct 30 20:23:13.341545 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:13.341510 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" event={"ID":"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1","Type":"ContainerStarted","Data":"fafeb0884f239ec8e4ea5b610f9a587c0ac12ebbb39de621f0efbc13134d922b"} Oct 30 20:23:13.341672 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:13.341565 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" event={"ID":"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1","Type":"ContainerStarted","Data":"4e5da732314bdfb7194990fff171508fe3115f437bcb8ff8704101e6d652e651"} Oct 30 20:23:13.367526 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:13.367472 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" podStartSLOduration=2.3674563969999998 podStartE2EDuration="2.367456397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 20:23:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:23:13.36555196 +0000 UTC m=+3113.060759915" watchObservedRunningTime="2026-10-30 20:23:13.367456397 +0000 UTC m=+3113.062664354" Oct 30 20:23:13.493589 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:13.485513 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:13.534585 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:13.534555 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" Oct 30 20:23:13.683497 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:13.683409 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e50c786-95a9-42b3-94b2-b894ee74e33a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e50c786-95a9-42b3-94b2-b894ee74e33a-maas-api-xks-ca\") pod \"2e50c786-95a9-42b3-94b2-b894ee74e33a\" (UID: \"2e50c786-95a9-42b3-94b2-b894ee74e33a\") " Oct 30 20:23:13.683497 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:13.683472 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e50c786-95a9-42b3-94b2-b894ee74e33a-kube-api-access-6brzl\" (UniqueName: \"kubernetes.io/projected/2e50c786-95a9-42b3-94b2-b894ee74e33a-kube-api-access-6brzl\") pod \"2e50c786-95a9-42b3-94b2-b894ee74e33a\" (UID: \"2e50c786-95a9-42b3-94b2-b894ee74e33a\") " Oct 30 20:23:13.683768 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:13.683625 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e50c786-95a9-42b3-94b2-b894ee74e33a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e50c786-95a9-42b3-94b2-b894ee74e33a-maas-api-openshift-ca\") pod \"2e50c786-95a9-42b3-94b2-b894ee74e33a\" (UID: \"2e50c786-95a9-42b3-94b2-b894ee74e33a\") " Oct 30 20:23:13.684044 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:13.684016 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e50c786-95a9-42b3-94b2-b894ee74e33a-maas-api-openshift-ca" pod "2e50c786-95a9-42b3-94b2-b894ee74e33a" (UID: "2e50c786-95a9-42b3-94b2-b894ee74e33a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:13.685730 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:13.685706 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e50c786-95a9-42b3-94b2-b894ee74e33a-kube-api-access-6brzl" pod "2e50c786-95a9-42b3-94b2-b894ee74e33a" (UID: "2e50c786-95a9-42b3-94b2-b894ee74e33a"). InnerVolumeSpecName "kube-api-access-6brzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:13.685998 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:13.685981 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e50c786-95a9-42b3-94b2-b894ee74e33a-maas-api-xks-ca" pod "2e50c786-95a9-42b3-94b2-b894ee74e33a" (UID: "2e50c786-95a9-42b3-94b2-b894ee74e33a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:23:13.785693 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:13.785655 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e50c786-95a9-42b3-94b2-b894ee74e33a-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:13.785693 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:13.785693 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6brzl\" (UniqueName: \"kubernetes.io/projected/2e50c786-95a9-42b3-94b2-b894ee74e33a-kube-api-access-6brzl\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:13.785893 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:13.785709 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e50c786-95a9-42b3-94b2-b894ee74e33a-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:14.034336 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:14.034299 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc"] Oct 30 20:23:14.037242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:14.037211 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc99947e-fsdlc"] Oct 30 20:23:14.039555 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:14.039528 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk"] Oct 30 20:23:14.043116 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:14.043094 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5e44932-59sgk"] Oct 30 20:23:14.100060 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:14.100010 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:14.278597 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:14.278555 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:14.350281 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:14.350198 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" Oct 30 20:23:14.350453 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:14.350197 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-712c6641-2c48m" event={"ID":"2e50c786-95a9-42b3-94b2-b894ee74e33a","Type":"ContainerDied","Data":"a41534f1e91c6c227ed1431123b9dc5416716bffed4f9799e290ce1b27385e1b"} Oct 30 20:23:14.350453 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:14.350326 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a41534f1e91c6c227ed1431123b9dc5416716bffed4f9799e290ce1b27385e1b" Oct 30 20:23:14.352392 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:14.352364 5091 generic.go:356] "Generic (PLEG): container finished" podID="a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1" containerID="fafeb0884f239ec8e4ea5b610f9a587c0ac12ebbb39de621f0efbc13134d922b" exitCode=0 Oct 30 20:23:14.352518 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:14.352411 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" event={"ID":"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1","Type":"ContainerDied","Data":"fafeb0884f239ec8e4ea5b610f9a587c0ac12ebbb39de621f0efbc13134d922b"} Oct 30 20:23:14.441651 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:14.441617 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47417445-63ec-4320-8296-59c85ec05fa3" path="/var/lib/kubelet/pods/47417445-63ec-4320-8296-59c85ec05fa3/volumes" Oct 30 20:23:14.442348 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:14.442325 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c187d6c7-f5d8-41ee-a906-11e33e892cd5" path="/var/lib/kubelet/pods/c187d6c7-f5d8-41ee-a906-11e33e892cd5/volumes" Oct 30 20:23:14.450757 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:14.450722 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:15.208302 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:15.208270 5091 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-worker-w1-7a745c: configmap "payload-processing-plugins-e2e-worker-w1-7a745c" not found Oct 30 20:23:15.208690 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:15.208349 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume podName:f8550b23-cb62-4ddb-90b9-8d2b0706dd20 nodeName:}" failed. No retries permitted until 2026-10-30 20:23:23.208332214 +0000 UTC m=+3122.903540180 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume") pod "payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" (UID: "f8550b23-cb62-4ddb-90b9-8d2b0706dd20") : configmap "payload-processing-plugins-e2e-worker-w1-7a745c" not found Oct 30 20:23:15.277307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:15.277268 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:15.525966 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:15.525941 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" Oct 30 20:23:15.603656 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:15.603626 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:15.613674 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:15.613642 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-kube-api-access-b9hlk\" (UniqueName: \"kubernetes.io/projected/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-kube-api-access-b9hlk\") pod \"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1\" (UID: \"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1\") " Oct 30 20:23:15.613882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:15.613864 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-maas-api-openshift-ca\") pod \"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1\" (UID: \"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1\") " Oct 30 20:23:15.613979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:15.613962 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-maas-api-xks-ca\") pod \"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1\" (UID: \"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1\") " Oct 30 20:23:15.614303 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:15.614273 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-maas-api-openshift-ca" pod "a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1" (UID: "a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:15.614481 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:15.614459 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:15.615698 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:15.615675 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-kube-api-access-b9hlk" pod "a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1" (UID: "a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1"). InnerVolumeSpecName "kube-api-access-b9hlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:15.615968 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:15.615944 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-maas-api-xks-ca" pod "a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1" (UID: "a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:23:15.715265 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:15.715228 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:15.715265 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:15.715259 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9hlk\" (UniqueName: \"kubernetes.io/projected/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1-kube-api-access-b9hlk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:15.976454 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:15.976361 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:23:16.041427 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.041382 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl"] Oct 30 20:23:16.044660 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.044638 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61a852f6-47svl"] Oct 30 20:23:16.053676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.053639 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2"] Oct 30 20:23:16.065849 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.065816 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a61e133f-h95p2"] Oct 30 20:23:16.281037 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.281000 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:16.369438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.369304 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" Oct 30 20:23:16.369438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.369350 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s" event={"ID":"a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1","Type":"ContainerDied","Data":"4e5da732314bdfb7194990fff171508fe3115f437bcb8ff8704101e6d652e651"} Oct 30 20:23:16.369438 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.369402 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e5da732314bdfb7194990fff171508fe3115f437bcb8ff8704101e6d652e651" Oct 30 20:23:16.373026 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.372997 5091 generic.go:356] "Generic (PLEG): container finished" podID="ec317404-f9f4-499c-bdbe-65109b6a18d2" containerID="4012fdf9bce4045b0905513abedc184e3277bd538e5ff2dbcb8fa38a21195e76" exitCode=0 Oct 30 20:23:16.373242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.373030 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" event={"ID":"ec317404-f9f4-499c-bdbe-65109b6a18d2","Type":"ContainerDied","Data":"4012fdf9bce4045b0905513abedc184e3277bd538e5ff2dbcb8fa38a21195e76"} Oct 30 20:23:16.373242 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.373120 5091 scope.go:122] "RemoveContainer" containerID="c6e5709981ad9b33dfea33bf266708bda762034c9a1072c405e98bd7d0a21d90" Oct 30 20:23:16.431819 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.431784 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:23:16.441045 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.441017 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97462176-f0ce-492c-bcad-1e125c73a4cd" path="/var/lib/kubelet/pods/97462176-f0ce-492c-bcad-1e125c73a4cd/volumes" Oct 30 20:23:16.441769 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.441745 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b62a6025-b883-4a28-bdea-ab4078ffc1bf" path="/var/lib/kubelet/pods/b62a6025-b883-4a28-bdea-ab4078ffc1bf/volumes" Oct 30 20:23:16.522363 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.522331 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-credential-socket\") pod \"ec317404-f9f4-499c-bdbe-65109b6a18d2\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " Oct 30 20:23:16.522509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.522387 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-podinfo\") pod \"ec317404-f9f4-499c-bdbe-65109b6a18d2\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " Oct 30 20:23:16.522509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.522424 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-envoy\") pod \"ec317404-f9f4-499c-bdbe-65109b6a18d2\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " Oct 30 20:23:16.522509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.522488 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec317404-f9f4-499c-bdbe-65109b6a18d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec317404-f9f4-499c-bdbe-65109b6a18d2-istiod-ca-cert\") pod \"ec317404-f9f4-499c-bdbe-65109b6a18d2\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " Oct 30 20:23:16.522632 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.522540 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec317404-f9f4-499c-bdbe-65109b6a18d2-kube-api-access-5c2kc\" (UniqueName: \"kubernetes.io/projected/ec317404-f9f4-499c-bdbe-65109b6a18d2-kube-api-access-5c2kc\") pod \"ec317404-f9f4-499c-bdbe-65109b6a18d2\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " Oct 30 20:23:16.522632 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.522617 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-workload-certs\") pod \"ec317404-f9f4-499c-bdbe-65109b6a18d2\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " Oct 30 20:23:16.522725 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.522645 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-credential-socket" pod "ec317404-f9f4-499c-bdbe-65109b6a18d2" (UID: "ec317404-f9f4-499c-bdbe-65109b6a18d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:16.522761 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.522719 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-data\") pod \"ec317404-f9f4-499c-bdbe-65109b6a18d2\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " Oct 30 20:23:16.522823 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.522762 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-workload-socket\") pod \"ec317404-f9f4-499c-bdbe-65109b6a18d2\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " Oct 30 20:23:16.522882 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.522860 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-workload-certs" pod "ec317404-f9f4-499c-bdbe-65109b6a18d2" (UID: "ec317404-f9f4-499c-bdbe-65109b6a18d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:16.522949 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.522887 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-token\" (UniqueName: \"kubernetes.io/projected/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-token\") pod \"ec317404-f9f4-499c-bdbe-65109b6a18d2\" (UID: \"ec317404-f9f4-499c-bdbe-65109b6a18d2\") " Oct 30 20:23:16.523099 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.523044 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec317404-f9f4-499c-bdbe-65109b6a18d2-istiod-ca-cert" pod "ec317404-f9f4-499c-bdbe-65109b6a18d2" (UID: "ec317404-f9f4-499c-bdbe-65109b6a18d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:16.523099 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.523080 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-workload-socket" pod "ec317404-f9f4-499c-bdbe-65109b6a18d2" (UID: "ec317404-f9f4-499c-bdbe-65109b6a18d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:16.523375 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.523017 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-data" pod "ec317404-f9f4-499c-bdbe-65109b6a18d2" (UID: "ec317404-f9f4-499c-bdbe-65109b6a18d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:16.523504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.523479 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:16.523504 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.523505 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:16.523701 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.523521 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec317404-f9f4-499c-bdbe-65109b6a18d2-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:16.523701 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.523532 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:16.523701 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.523540 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:16.524721 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.524697 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-envoy" pod "ec317404-f9f4-499c-bdbe-65109b6a18d2" (UID: "ec317404-f9f4-499c-bdbe-65109b6a18d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:16.524838 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.524821 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec317404-f9f4-499c-bdbe-65109b6a18d2-kube-api-access-5c2kc" pod "ec317404-f9f4-499c-bdbe-65109b6a18d2" (UID: "ec317404-f9f4-499c-bdbe-65109b6a18d2"). InnerVolumeSpecName "kube-api-access-5c2kc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:16.525108 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.525090 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-podinfo" pod "ec317404-f9f4-499c-bdbe-65109b6a18d2" (UID: "ec317404-f9f4-499c-bdbe-65109b6a18d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:23:16.525315 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.525295 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-token" pod "ec317404-f9f4-499c-bdbe-65109b6a18d2" (UID: "ec317404-f9f4-499c-bdbe-65109b6a18d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:16.624597 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.624560 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:16.624766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.624617 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:16.624766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.624637 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec317404-f9f4-499c-bdbe-65109b6a18d2-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:16.624766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.624653 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5c2kc\" (UniqueName: \"kubernetes.io/projected/ec317404-f9f4-499c-bdbe-65109b6a18d2-kube-api-access-5c2kc\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:16.883025 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.882940 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" podUID="405da951-aef4-4187-8755-b264fff0c2a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:23:16.883025 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:16.882960 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" podUID="f8550b23-cb62-4ddb-90b9-8d2b0706dd20" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:23:17.362366 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:17.362331 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:17.383230 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:17.383206 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" Oct 30 20:23:17.383394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:17.383226 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm" event={"ID":"ec317404-f9f4-499c-bdbe-65109b6a18d2","Type":"ContainerDied","Data":"b817866e9fd5bc2ce8ee3e947f25f7e3f4c29d49686ebffe9e25638748094b24"} Oct 30 20:23:17.383394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:17.383287 5091 scope.go:122] "RemoveContainer" containerID="4012fdf9bce4045b0905513abedc184e3277bd538e5ff2dbcb8fa38a21195e76" Oct 30 20:23:17.421749 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:17.421710 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm"] Oct 30 20:23:17.424852 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:17.424829 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-4d2317-openshift-default-7ccd49676f-phrkm"] Oct 30 20:23:18.278307 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:18.278262 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:18.440342 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:18.440307 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec317404-f9f4-499c-bdbe-65109b6a18d2" path="/var/lib/kubelet/pods/ec317404-f9f4-499c-bdbe-65109b6a18d2/volumes" Oct 30 20:23:19.183184 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:19.183106 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:20.078099 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:20.076853 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:20.987280 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:20.985099 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:21.650662 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:21.650584 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:21.994859 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:21.994772 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7"] Oct 30 20:23:22.076810 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:22.076205 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq"] Oct 30 20:23:22.078491 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:22.078454 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-7a745c-tgvpq"] Oct 30 20:23:22.099480 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:22.099416 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:22.260111 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:22.260070 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:22.459884 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:22.455398 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15ea609-33fa-42cc-9bc6-6b3217433d70" path="/var/lib/kubelet/pods/a15ea609-33fa-42cc-9bc6-6b3217433d70/volumes" Oct 30 20:23:22.768753 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:22.768715 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:23.214309 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:23.214226 5091 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-worker-w1-7a745c: configmap "payload-processing-plugins-e2e-worker-w1-7a745c" not found Oct 30 20:23:23.214309 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:23.214297 5091 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume podName:f8550b23-cb62-4ddb-90b9-8d2b0706dd20 nodeName:}" failed. No retries permitted until 2026-10-30 20:23:39.214282627 +0000 UTC m=+3138.909490581 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume") pod "payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" (UID: "f8550b23-cb62-4ddb-90b9-8d2b0706dd20") : configmap "payload-processing-plugins-e2e-worker-w1-7a745c" not found Oct 30 20:23:23.307088 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:23.307048 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:24.370726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:24.369916 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:25.106254 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:25.106210 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:25.299581 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:25.299547 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:25.346111 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:25.346078 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s"] Oct 30 20:23:25.350925 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:25.350895 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-75b361-l5c5s"] Oct 30 20:23:25.448100 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:25.447956 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:25.589081 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:25.589045 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:25.928965 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:25.928929 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:26.364195 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:26.364135 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:26.441424 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:26.441392 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1" path="/var/lib/kubelet/pods/a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1/volumes" Oct 30 20:23:26.883792 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:26.883751 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" podUID="405da951-aef4-4187-8755-b264fff0c2a0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:23:26.884218 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:26.883814 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" podUID="f8550b23-cb62-4ddb-90b9-8d2b0706dd20" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:23:26.995228 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:26.995188 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" podUID="432c8f27-589b-4be4-aac3-e37148e70ef9" containerName="payload-processing" containerID="cri-o://fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4" gracePeriod=25 Oct 30 20:23:27.261013 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.260981 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" Oct 30 20:23:27.369302 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.369270 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/432c8f27-589b-4be4-aac3-e37148e70ef9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432c8f27-589b-4be4-aac3-e37148e70ef9-plugins-config-volume\") pod \"432c8f27-589b-4be4-aac3-e37148e70ef9\" (UID: \"432c8f27-589b-4be4-aac3-e37148e70ef9\") " Oct 30 20:23:27.370124 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.369345 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/432c8f27-589b-4be4-aac3-e37148e70ef9-kube-api-access-5z8d9\" (UniqueName: \"kubernetes.io/projected/432c8f27-589b-4be4-aac3-e37148e70ef9-kube-api-access-5z8d9\") pod \"432c8f27-589b-4be4-aac3-e37148e70ef9\" (UID: \"432c8f27-589b-4be4-aac3-e37148e70ef9\") " Oct 30 20:23:27.372338 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.372281 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432c8f27-589b-4be4-aac3-e37148e70ef9-kube-api-access-5z8d9" pod "432c8f27-589b-4be4-aac3-e37148e70ef9" (UID: "432c8f27-589b-4be4-aac3-e37148e70ef9"). InnerVolumeSpecName "kube-api-access-5z8d9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:27.387172 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.387080 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432c8f27-589b-4be4-aac3-e37148e70ef9-plugins-config-volume" pod "432c8f27-589b-4be4-aac3-e37148e70ef9" (UID: "432c8f27-589b-4be4-aac3-e37148e70ef9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:27.388614 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.388572 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:27.447258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.447141 5091 scope.go:122] "RemoveContainer" containerID="ad27fda62ec07b14de265f7ad372e74b6acb9c367c8783c7fa899e56506d4989" Oct 30 20:23:27.458713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.458685 5091 scope.go:122] "RemoveContainer" containerID="20bc969515e87e36c85fbba75c399cb963835874925bb0eeaec1c682386803b3" Oct 30 20:23:27.468028 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.467994 5091 scope.go:122] "RemoveContainer" containerID="02c5678127773ae6d48174b611fad090ce97b6aca773ae541ef0e36de18bde9d" Oct 30 20:23:27.470694 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.470672 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/432c8f27-589b-4be4-aac3-e37148e70ef9-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:27.470779 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.470696 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z8d9\" (UniqueName: \"kubernetes.io/projected/432c8f27-589b-4be4-aac3-e37148e70ef9-kube-api-access-5z8d9\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:27.476945 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.476924 5091 scope.go:122] "RemoveContainer" containerID="04b65825c9602565f820af37c3b5986f26b28c3f4138e594b411e10889f9886a" Oct 30 20:23:27.486024 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.486003 5091 scope.go:122] "RemoveContainer" containerID="fac137b2e79462925158430ab8df78e6f5cdb8c2ba565a1287a8593b2a97be4d" Oct 30 20:23:27.495630 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.495609 5091 scope.go:122] "RemoveContainer" containerID="9096330f968af83c7bf98816c28cca87083a87f15470a377fb048578aae5969e" Oct 30 20:23:27.505009 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.504991 5091 scope.go:122] "RemoveContainer" containerID="b4af77de3804181316ac2573b0da9f04d590547ddb6bd51db98a33f0f68be308" Oct 30 20:23:27.506979 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.506954 5091 generic.go:356] "Generic (PLEG): container finished" podID="432c8f27-589b-4be4-aac3-e37148e70ef9" containerID="fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4" exitCode=0 Oct 30 20:23:27.507085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.506996 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" Oct 30 20:23:27.507085 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.507039 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" event={"ID":"432c8f27-589b-4be4-aac3-e37148e70ef9","Type":"ContainerDied","Data":"fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4"} Oct 30 20:23:27.507275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.507096 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7" event={"ID":"432c8f27-589b-4be4-aac3-e37148e70ef9","Type":"ContainerDied","Data":"dce2e30e687e4eb20b773b54ef60b03f45eb55479f3cb584ebedb584eabb6cfe"} Oct 30 20:23:27.507275 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.507125 5091 scope.go:122] "RemoveContainer" containerID="fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4" Oct 30 20:23:27.520419 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.520399 5091 scope.go:122] "RemoveContainer" containerID="1b19fac93a0417620496709cf5b39bf2270766db5089abbe9e82e37a8382a404" Oct 30 20:23:27.520806 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.520790 5091 scope.go:122] "RemoveContainer" containerID="fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4" Oct 30 20:23:27.521072 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:27.521050 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4\": container with ID starting with fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4 not found: ID does not exist" containerID="fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4" Oct 30 20:23:27.521185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.521077 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4"} err="failed to get container status \"fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4\": rpc error: code = NotFound desc = could not find container \"fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4\": container with ID starting with fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4 not found: ID does not exist" Oct 30 20:23:27.530109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.530085 5091 scope.go:122] "RemoveContainer" containerID="2f6b27a94c5903e94dfacfb80b8216c9ba171be0ea76ffe58f5101ba88956c8b" Oct 30 20:23:27.543114 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.543022 5091 scope.go:122] "RemoveContainer" containerID="a7a03f85339cec74afb7a2b34c8934d2c51683bf8bdec844f756acf684bc6f42" Oct 30 20:23:27.551267 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.551240 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7"] Oct 30 20:23:27.553054 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.553034 5091 scope.go:122] "RemoveContainer" containerID="3eba951691ad960dae81576dfcb31df52fd1b5656c665d4abeea4da221368b5f" Oct 30 20:23:27.559699 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.559675 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-75b361-5d9d794d86-dlpq7"] Oct 30 20:23:27.562481 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.562459 5091 scope.go:122] "RemoveContainer" containerID="fca466f8097ad7f34aab865f0a84b55e36713f376ac5034e5cb5d1c18ddc60ba" Oct 30 20:23:27.571812 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.571794 5091 scope.go:122] "RemoveContainer" containerID="fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4" Oct 30 20:23:27.572114 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:27.572090 5091 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4\": rpc error: code = NotFound desc = could not find container \"fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4\": container with ID starting with fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4 not found: ID does not exist" containerID="fd90939d1620341be653c997defacef510bc108c832689ff19291e4bd5e9dbc4" Oct 30 20:23:27.572224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:27.572117 5091 scope.go:122] "RemoveContainer" containerID="56747554a424103d36cedcea199914e38ed7a3fc661b958d07360438881d8ad0" Oct 30 20:23:28.386426 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:28.384849 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:28.441198 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:28.441162 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="432c8f27-589b-4be4-aac3-e37148e70ef9" path="/var/lib/kubelet/pods/432c8f27-589b-4be4-aac3-e37148e70ef9/volumes" Oct 30 20:23:29.277859 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:29.277817 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:29.601317 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:29.601243 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:29.744695 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:29.744656 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:30.050168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:30.041212 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:30.330839 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:30.330733 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:31.317321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:31.317281 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:32.288825 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:32.288783 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:32.733592 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:32.733566 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" Oct 30 20:23:32.835368 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:32.835278 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405da951-aef4-4187-8755-b264fff0c2a0-kube-api-access-tkt6f\" (UniqueName: \"kubernetes.io/projected/405da951-aef4-4187-8755-b264fff0c2a0-kube-api-access-tkt6f\") pod \"405da951-aef4-4187-8755-b264fff0c2a0\" (UID: \"405da951-aef4-4187-8755-b264fff0c2a0\") " Oct 30 20:23:32.835368 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:32.835322 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405da951-aef4-4187-8755-b264fff0c2a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405da951-aef4-4187-8755-b264fff0c2a0-plugins-config-volume\") pod \"405da951-aef4-4187-8755-b264fff0c2a0\" (UID: \"405da951-aef4-4187-8755-b264fff0c2a0\") " Oct 30 20:23:32.835749 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:32.835724 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405da951-aef4-4187-8755-b264fff0c2a0-plugins-config-volume" pod "405da951-aef4-4187-8755-b264fff0c2a0" (UID: "405da951-aef4-4187-8755-b264fff0c2a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:32.837569 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:32.837551 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405da951-aef4-4187-8755-b264fff0c2a0-kube-api-access-tkt6f" pod "405da951-aef4-4187-8755-b264fff0c2a0" (UID: "405da951-aef4-4187-8755-b264fff0c2a0"). InnerVolumeSpecName "kube-api-access-tkt6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:32.936561 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:32.936529 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkt6f\" (UniqueName: \"kubernetes.io/projected/405da951-aef4-4187-8755-b264fff0c2a0-kube-api-access-tkt6f\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:32.936739 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:32.936580 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405da951-aef4-4187-8755-b264fff0c2a0-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:32.982246 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:32.982206 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:33.128690 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.128609 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:33.404073 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.403959 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:33.556317 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.556275 5091 generic.go:356] "Generic (PLEG): container finished" podID="f8550b23-cb62-4ddb-90b9-8d2b0706dd20" containerID="29e268e0fb5627d4bed7385eee0ab553f70cad063702b3398fbce6cda92c94ce" exitCode=0 Oct 30 20:23:33.556522 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.556344 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" event={"ID":"f8550b23-cb62-4ddb-90b9-8d2b0706dd20","Type":"ContainerDied","Data":"29e268e0fb5627d4bed7385eee0ab553f70cad063702b3398fbce6cda92c94ce"} Oct 30 20:23:33.558058 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.558033 5091 generic.go:356] "Generic (PLEG): container finished" podID="405da951-aef4-4187-8755-b264fff0c2a0" containerID="9885d65fc47a9858488a86c9c29c6839ce1b3063d236382ae821e0318aaefc61" exitCode=0 Oct 30 20:23:33.558224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.558094 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" Oct 30 20:23:33.558224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.558107 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" event={"ID":"405da951-aef4-4187-8755-b264fff0c2a0","Type":"ContainerDied","Data":"9885d65fc47a9858488a86c9c29c6839ce1b3063d236382ae821e0318aaefc61"} Oct 30 20:23:33.558224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.558194 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk" event={"ID":"405da951-aef4-4187-8755-b264fff0c2a0","Type":"ContainerDied","Data":"c84e2db017a9d9e03937528d9518a9d06287f7041857921dc73db131234a960e"} Oct 30 20:23:33.558224 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.558218 5091 scope.go:122] "RemoveContainer" containerID="9885d65fc47a9858488a86c9c29c6839ce1b3063d236382ae821e0318aaefc61" Oct 30 20:23:33.570540 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.570452 5091 scope.go:122] "RemoveContainer" containerID="9885d65fc47a9858488a86c9c29c6839ce1b3063d236382ae821e0318aaefc61" Oct 30 20:23:33.570783 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:33.570762 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9885d65fc47a9858488a86c9c29c6839ce1b3063d236382ae821e0318aaefc61\": container with ID starting with 9885d65fc47a9858488a86c9c29c6839ce1b3063d236382ae821e0318aaefc61 not found: ID does not exist" containerID="9885d65fc47a9858488a86c9c29c6839ce1b3063d236382ae821e0318aaefc61" Oct 30 20:23:33.570926 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.570791 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9885d65fc47a9858488a86c9c29c6839ce1b3063d236382ae821e0318aaefc61"} err="failed to get container status \"9885d65fc47a9858488a86c9c29c6839ce1b3063d236382ae821e0318aaefc61\": rpc error: code = NotFound desc = could not find container \"9885d65fc47a9858488a86c9c29c6839ce1b3063d236382ae821e0318aaefc61\": container with ID starting with 9885d65fc47a9858488a86c9c29c6839ce1b3063d236382ae821e0318aaefc61 not found: ID does not exist" Oct 30 20:23:33.588939 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.588792 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk"] Oct 30 20:23:33.592180 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.592137 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-7a745c-c7bc6d4fd-bf7sk"] Oct 30 20:23:33.599655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.599637 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" Oct 30 20:23:33.745991 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.745897 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume\") pod \"f8550b23-cb62-4ddb-90b9-8d2b0706dd20\" (UID: \"f8550b23-cb62-4ddb-90b9-8d2b0706dd20\") " Oct 30 20:23:33.746494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.746123 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-kube-api-access-zrvnk\" (UniqueName: \"kubernetes.io/projected/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-kube-api-access-zrvnk\") pod \"f8550b23-cb62-4ddb-90b9-8d2b0706dd20\" (UID: \"f8550b23-cb62-4ddb-90b9-8d2b0706dd20\") " Oct 30 20:23:33.746494 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.746275 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume" pod "f8550b23-cb62-4ddb-90b9-8d2b0706dd20" (UID: "f8550b23-cb62-4ddb-90b9-8d2b0706dd20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:33.746632 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.746575 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:33.748411 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.748386 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-kube-api-access-zrvnk" pod "f8550b23-cb62-4ddb-90b9-8d2b0706dd20" (UID: "f8550b23-cb62-4ddb-90b9-8d2b0706dd20"). InnerVolumeSpecName "kube-api-access-zrvnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:33.834573 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.834545 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:33.848241 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.848212 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrvnk\" (UniqueName: \"kubernetes.io/projected/f8550b23-cb62-4ddb-90b9-8d2b0706dd20-kube-api-access-zrvnk\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:33.978351 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:33.978315 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:34.441457 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:34.441416 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405da951-aef4-4187-8755-b264fff0c2a0" path="/var/lib/kubelet/pods/405da951-aef4-4187-8755-b264fff0c2a0/volumes" Oct 30 20:23:34.567598 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:34.567563 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" Oct 30 20:23:34.567598 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:34.567583 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc" event={"ID":"f8550b23-cb62-4ddb-90b9-8d2b0706dd20","Type":"ContainerDied","Data":"72c76b6c359eb3d40d0197545da52e5720e765e0dffa4ec6fac14043f4e42b22"} Oct 30 20:23:34.567870 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:34.567650 5091 scope.go:122] "RemoveContainer" containerID="29e268e0fb5627d4bed7385eee0ab553f70cad063702b3398fbce6cda92c94ce" Oct 30 20:23:34.593520 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:34.593492 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc"] Oct 30 20:23:34.598904 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:34.598879 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-7a745c-69b56f599-z6ztc"] Oct 30 20:23:34.851715 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:34.851676 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:35.682070 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:35.681823 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:36.442570 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:36.442528 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8550b23-cb62-4ddb-90b9-8d2b0706dd20" path="/var/lib/kubelet/pods/f8550b23-cb62-4ddb-90b9-8d2b0706dd20/volumes" Oct 30 20:23:36.577314 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:36.577274 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:37.582519 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:37.582480 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:38.515163 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:38.515107 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:38.768105 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:38.768024 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm"] Oct 30 20:23:38.962616 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:38.962584 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:39.543483 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:39.543450 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:39.710161 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:39.710109 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:40.496917 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:40.496877 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:41.568191 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:41.565699 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:41.805478 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:41.805442 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:42.369372 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:42.365773 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:43.268394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:43.268341 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:44.076895 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.076860 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7"] Oct 30 20:23:44.105052 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.105014 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk"] Oct 30 20:23:44.202371 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.202335 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:44.370928 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.370828 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:44.886519 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.886490 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph"] Oct 30 20:23:44.886929 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.886916 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1" containerName="revoke-keys" Oct 30 20:23:44.886986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.886930 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1" containerName="revoke-keys" Oct 30 20:23:44.886986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.886954 5091 cpu_manager.go:414] "containerMap: removing container" podUID="405da951-aef4-4187-8755-b264fff0c2a0" containerName="payload-processing" Oct 30 20:23:44.886986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.886960 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405da951-aef4-4187-8755-b264fff0c2a0" containerName="payload-processing" Oct 30 20:23:44.886986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.886967 5091 cpu_manager.go:414] "containerMap: removing container" podUID="f8550b23-cb62-4ddb-90b9-8d2b0706dd20" containerName="payload-pre-processing" Oct 30 20:23:44.886986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.886972 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8550b23-cb62-4ddb-90b9-8d2b0706dd20" containerName="payload-pre-processing" Oct 30 20:23:44.886986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.886984 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ec317404-f9f4-499c-bdbe-65109b6a18d2" containerName="istio-proxy" Oct 30 20:23:44.886986 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.886989 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec317404-f9f4-499c-bdbe-65109b6a18d2" containerName="istio-proxy" Oct 30 20:23:44.887264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.887000 5091 cpu_manager.go:414] "containerMap: removing container" podUID="432c8f27-589b-4be4-aac3-e37148e70ef9" containerName="payload-processing" Oct 30 20:23:44.887264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.887005 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="432c8f27-589b-4be4-aac3-e37148e70ef9" containerName="payload-processing" Oct 30 20:23:44.887264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.887014 5091 cpu_manager.go:414] "containerMap: removing container" podUID="ec317404-f9f4-499c-bdbe-65109b6a18d2" containerName="istio-proxy" Oct 30 20:23:44.887264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.887020 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec317404-f9f4-499c-bdbe-65109b6a18d2" containerName="istio-proxy" Oct 30 20:23:44.887264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.887029 5091 cpu_manager.go:414] "containerMap: removing container" podUID="2e50c786-95a9-42b3-94b2-b894ee74e33a" containerName="revoke-keys" Oct 30 20:23:44.887264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.887034 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e50c786-95a9-42b3-94b2-b894ee74e33a" containerName="revoke-keys" Oct 30 20:23:44.887264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.887107 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="405da951-aef4-4187-8755-b264fff0c2a0" containerName="payload-processing" Oct 30 20:23:44.887264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.887115 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8550b23-cb62-4ddb-90b9-8d2b0706dd20" containerName="payload-pre-processing" Oct 30 20:23:44.887264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.887123 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec317404-f9f4-499c-bdbe-65109b6a18d2" containerName="istio-proxy" Oct 30 20:23:44.887264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.887129 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="432c8f27-589b-4be4-aac3-e37148e70ef9" containerName="payload-processing" Oct 30 20:23:44.887264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.887135 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec317404-f9f4-499c-bdbe-65109b6a18d2" containerName="istio-proxy" Oct 30 20:23:44.887264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.887141 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e50c786-95a9-42b3-94b2-b894ee74e33a" containerName="revoke-keys" Oct 30 20:23:44.887264 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.887167 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13e1b14-87e8-41e4-99ec-3cfe6e68f7a1" containerName="revoke-keys" Oct 30 20:23:44.891761 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.891740 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" Oct 30 20:23:44.901709 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.901685 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-zgv8j\"" Oct 30 20:23:44.901836 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.901760 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 20:23:44.903710 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.903689 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph"] Oct 30 20:23:44.964777 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.964746 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88fd350d-lg4ph\" (UID: \"a9e4cada-6f7a-4396-9cdf-933341e4dcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" Oct 30 20:23:44.964970 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.964878 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npvz2\" (UniqueName: \"kubernetes.io/projected/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-kube-api-access-npvz2\") pod \"maas-api-revoke-sub-88fd350d-lg4ph\" (UID: \"a9e4cada-6f7a-4396-9cdf-933341e4dcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" Oct 30 20:23:44.964970 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:44.964949 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88fd350d-lg4ph\" (UID: \"a9e4cada-6f7a-4396-9cdf-933341e4dcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" Oct 30 20:23:45.066455 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:45.066415 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88fd350d-lg4ph\" (UID: \"a9e4cada-6f7a-4396-9cdf-933341e4dcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" Oct 30 20:23:45.066647 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:45.066531 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npvz2\" (UniqueName: \"kubernetes.io/projected/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-kube-api-access-npvz2\") pod \"maas-api-revoke-sub-88fd350d-lg4ph\" (UID: \"a9e4cada-6f7a-4396-9cdf-933341e4dcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" Oct 30 20:23:45.066647 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:45.066583 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88fd350d-lg4ph\" (UID: \"a9e4cada-6f7a-4396-9cdf-933341e4dcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" Oct 30 20:23:45.067190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:45.067165 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88fd350d-lg4ph\" (UID: \"a9e4cada-6f7a-4396-9cdf-933341e4dcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" Oct 30 20:23:45.068873 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:45.068853 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88fd350d-lg4ph\" (UID: \"a9e4cada-6f7a-4396-9cdf-933341e4dcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" Oct 30 20:23:45.080185 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:45.080162 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npvz2\" (UniqueName: \"kubernetes.io/projected/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-kube-api-access-npvz2\") pod \"maas-api-revoke-sub-88fd350d-lg4ph\" (UID: \"a9e4cada-6f7a-4396-9cdf-933341e4dcbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" Oct 30 20:23:45.205877 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:45.205798 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" Oct 30 20:23:45.487377 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:45.487350 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph"] Oct 30 20:23:45.490176 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:23:45.490134 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9e4cada_6f7a_4396_9cdf_933341e4dcbe.slice/crio-73db7773579cbfd2d8bf9b5136c9f0c6bc26713c0947ab4e8cc44a3e349bbdf4 WatchSource:0}: Error finding container 73db7773579cbfd2d8bf9b5136c9f0c6bc26713c0947ab4e8cc44a3e349bbdf4: Status 404 returned error can't find the container with id 73db7773579cbfd2d8bf9b5136c9f0c6bc26713c0947ab4e8cc44a3e349bbdf4 Oct 30 20:23:45.664953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:45.664919 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" event={"ID":"a9e4cada-6f7a-4396-9cdf-933341e4dcbe","Type":"ContainerStarted","Data":"73db7773579cbfd2d8bf9b5136c9f0c6bc26713c0947ab4e8cc44a3e349bbdf4"} Oct 30 20:23:45.976337 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:45.976297 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:23:46.674550 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:46.674449 5091 generic.go:356] "Generic (PLEG): container finished" podID="a9e4cada-6f7a-4396-9cdf-933341e4dcbe" containerID="7793c7ecda0654bef5d08f47f1475f8d8cfffa4a50e35fdb178f7952e59d0000" exitCode=0 Oct 30 20:23:46.674550 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:46.674528 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" event={"ID":"a9e4cada-6f7a-4396-9cdf-933341e4dcbe","Type":"ContainerDied","Data":"7793c7ecda0654bef5d08f47f1475f8d8cfffa4a50e35fdb178f7952e59d0000"} Oct 30 20:23:47.834602 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:47.834575 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" Oct 30 20:23:47.895486 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:47.895458 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-maas-api-openshift-ca\") pod \"a9e4cada-6f7a-4396-9cdf-933341e4dcbe\" (UID: \"a9e4cada-6f7a-4396-9cdf-933341e4dcbe\") " Oct 30 20:23:47.895676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:47.895587 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-kube-api-access-npvz2\" (UniqueName: \"kubernetes.io/projected/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-kube-api-access-npvz2\") pod \"a9e4cada-6f7a-4396-9cdf-933341e4dcbe\" (UID: \"a9e4cada-6f7a-4396-9cdf-933341e4dcbe\") " Oct 30 20:23:47.895676 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:47.895623 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-maas-api-xks-ca\") pod \"a9e4cada-6f7a-4396-9cdf-933341e4dcbe\" (UID: \"a9e4cada-6f7a-4396-9cdf-933341e4dcbe\") " Oct 30 20:23:47.895867 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:47.895842 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-maas-api-openshift-ca" pod "a9e4cada-6f7a-4396-9cdf-933341e4dcbe" (UID: "a9e4cada-6f7a-4396-9cdf-933341e4dcbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:47.897767 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:47.897732 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-kube-api-access-npvz2" pod "a9e4cada-6f7a-4396-9cdf-933341e4dcbe" (UID: "a9e4cada-6f7a-4396-9cdf-933341e4dcbe"). InnerVolumeSpecName "kube-api-access-npvz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:47.897907 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:47.897786 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-maas-api-xks-ca" pod "a9e4cada-6f7a-4396-9cdf-933341e4dcbe" (UID: "a9e4cada-6f7a-4396-9cdf-933341e4dcbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:23:47.997362 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:47.997294 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-maas-api-xks-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:47.997362 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:47.997321 5091 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-maas-api-openshift-ca\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:47.997362 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:47.997334 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npvz2\" (UniqueName: \"kubernetes.io/projected/a9e4cada-6f7a-4396-9cdf-933341e4dcbe-kube-api-access-npvz2\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:48.690018 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:48.689978 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" event={"ID":"a9e4cada-6f7a-4396-9cdf-933341e4dcbe","Type":"ContainerDied","Data":"73db7773579cbfd2d8bf9b5136c9f0c6bc26713c0947ab4e8cc44a3e349bbdf4"} Oct 30 20:23:48.690018 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:48.690004 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88fd350d-lg4ph" Oct 30 20:23:48.690278 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:48.690040 5091 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73db7773579cbfd2d8bf9b5136c9f0c6bc26713c0947ab4e8cc44a3e349bbdf4" Oct 30 20:23:49.064855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:49.064824 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:49.077726 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:49.077687 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" podUID="8b5a523a-97be-4828-b79e-ec093df306ff" containerName="payload-processing" containerID="cri-o://aea8a3e36f4b039fa2826060f60cac91afa29ba326559a23f428a1ba49d7412a" gracePeriod=25 Oct 30 20:23:49.106121 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:49.106055 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" podUID="6f7ca223-be1f-4f97-aef8-cf286b756081" containerName="payload-pre-processing" containerID="cri-o://35c301b70ba0f174f9a475980824e99b36ae596fc12b55cb617a4ee3199ff7ba" gracePeriod=25 Oct 30 20:23:51.357107 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:51.357073 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:23:52.682259 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:52.682224 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck"] Oct 30 20:23:52.682754 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:52.682416 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" podUID="961fed7b-2029-4b0e-886f-f4d6cda91f81" containerName="istio-proxy" containerID="cri-o://ba8a5e01a21350edbb0dbc00c606c6bd3bc8a8e5f343d9c302902bf75f45ba95" gracePeriod=30 Oct 30 20:23:53.789340 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:53.789286 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" containerID="cri-o://259ab7b2c5a1b53a55ba0a41576190f3a2fd3bb6a3b68ab2dc1043edc55cd48f" gracePeriod=45 Oct 30 20:23:54.039561 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.039495 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 20:23:54.164666 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.164113 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-model-cache\") pod \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " Oct 30 20:23:54.164666 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.164188 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-kube-api-access-5j68n\" (UniqueName: \"kubernetes.io/projected/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-kube-api-access-5j68n\") pod \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " Oct 30 20:23:54.164666 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.164272 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-tls-certs\" (UniqueName: \"kubernetes.io/secret/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-tls-certs\") pod \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " Oct 30 20:23:54.164666 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.164291 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-dshm\") pod \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " Oct 30 20:23:54.164666 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.164321 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-tmp-dir\") pod \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " Oct 30 20:23:54.164666 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.164374 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-home\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-home\") pod \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\" (UID: \"706afd71-a9e1-4b85-a243-b6a9f5e1edb1\") " Oct 30 20:23:54.165207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.164964 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-home" pod "706afd71-a9e1-4b85-a243-b6a9f5e1edb1" (UID: "706afd71-a9e1-4b85-a243-b6a9f5e1edb1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:54.165207 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.165182 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-model-cache" pod "706afd71-a9e1-4b85-a243-b6a9f5e1edb1" (UID: "706afd71-a9e1-4b85-a243-b6a9f5e1edb1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:54.165896 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.165869 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-tmp-dir" pod "706afd71-a9e1-4b85-a243-b6a9f5e1edb1" (UID: "706afd71-a9e1-4b85-a243-b6a9f5e1edb1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:54.170319 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.170268 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-tls-certs" pod "706afd71-a9e1-4b85-a243-b6a9f5e1edb1" (UID: "706afd71-a9e1-4b85-a243-b6a9f5e1edb1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 20:23:54.170319 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.170278 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-dshm" pod "706afd71-a9e1-4b85-a243-b6a9f5e1edb1" (UID: "706afd71-a9e1-4b85-a243-b6a9f5e1edb1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:54.170491 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.170315 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-kube-api-access-5j68n" pod "706afd71-a9e1-4b85-a243-b6a9f5e1edb1" (UID: "706afd71-a9e1-4b85-a243-b6a9f5e1edb1"). InnerVolumeSpecName "kube-api-access-5j68n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:54.266446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.266408 5091 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-tmp-dir\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:54.266446 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.266445 5091 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-home\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:54.266834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.266457 5091 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-model-cache\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:54.266834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.266470 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5j68n\" (UniqueName: \"kubernetes.io/projected/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-kube-api-access-5j68n\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:54.266834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.266483 5091 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-tls-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:54.266834 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.266494 5091 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/706afd71-a9e1-4b85-a243-b6a9f5e1edb1-dshm\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:54.730270 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.730175 5091 generic.go:356] "Generic (PLEG): container finished" podID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerID="259ab7b2c5a1b53a55ba0a41576190f3a2fd3bb6a3b68ab2dc1043edc55cd48f" exitCode=0 Oct 30 20:23:54.730270 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.730225 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" Oct 30 20:23:54.730270 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.730247 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" event={"ID":"706afd71-a9e1-4b85-a243-b6a9f5e1edb1","Type":"ContainerDied","Data":"259ab7b2c5a1b53a55ba0a41576190f3a2fd3bb6a3b68ab2dc1043edc55cd48f"} Oct 30 20:23:54.730535 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.730297 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm" event={"ID":"706afd71-a9e1-4b85-a243-b6a9f5e1edb1","Type":"ContainerDied","Data":"9ba84a1ad031948560b7020dcfdddaf2ddcaee7439279f33596f0cb6c2a00b4c"} Oct 30 20:23:54.730535 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.730316 5091 scope.go:122] "RemoveContainer" containerID="259ab7b2c5a1b53a55ba0a41576190f3a2fd3bb6a3b68ab2dc1043edc55cd48f" Oct 30 20:23:54.740602 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.740582 5091 scope.go:122] "RemoveContainer" containerID="259ab7b2c5a1b53a55ba0a41576190f3a2fd3bb6a3b68ab2dc1043edc55cd48f" Oct 30 20:23:54.740865 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:54.740842 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"259ab7b2c5a1b53a55ba0a41576190f3a2fd3bb6a3b68ab2dc1043edc55cd48f\": container with ID starting with 259ab7b2c5a1b53a55ba0a41576190f3a2fd3bb6a3b68ab2dc1043edc55cd48f not found: ID does not exist" containerID="259ab7b2c5a1b53a55ba0a41576190f3a2fd3bb6a3b68ab2dc1043edc55cd48f" Oct 30 20:23:54.740958 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.740876 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"259ab7b2c5a1b53a55ba0a41576190f3a2fd3bb6a3b68ab2dc1043edc55cd48f"} err="failed to get container status \"259ab7b2c5a1b53a55ba0a41576190f3a2fd3bb6a3b68ab2dc1043edc55cd48f\": rpc error: code = NotFound desc = could not find container \"259ab7b2c5a1b53a55ba0a41576190f3a2fd3bb6a3b68ab2dc1043edc55cd48f\": container with ID starting with 259ab7b2c5a1b53a55ba0a41576190f3a2fd3bb6a3b68ab2dc1043edc55cd48f not found: ID does not exist" Oct 30 20:23:54.753987 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.753964 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm"] Oct 30 20:23:54.758864 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.758842 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-8836a8/distinct-2-w0-8836a8-kserve-7476c99d8b-kwxmm"] Oct 30 20:23:54.863291 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:54.863265 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" podUID="6f7ca223-be1f-4f97-aef8-cf286b756081" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:23:55.874915 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:55.874877 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" podUID="8b5a523a-97be-4828-b79e-ec093df306ff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:23:56.441949 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:56.441909 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" path="/var/lib/kubelet/pods/706afd71-a9e1-4b85-a243-b6a9f5e1edb1/volumes" Oct 30 20:23:57.969314 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:57.969228 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:23:58.112526 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.112003 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-token\" (UniqueName: \"kubernetes.io/projected/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-token\") pod \"961fed7b-2029-4b0e-886f-f4d6cda91f81\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " Oct 30 20:23:58.112526 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.112060 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-data\") pod \"961fed7b-2029-4b0e-886f-f4d6cda91f81\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " Oct 30 20:23:58.112526 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.112125 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961fed7b-2029-4b0e-886f-f4d6cda91f81-kube-api-access-fvfhn\" (UniqueName: \"kubernetes.io/projected/961fed7b-2029-4b0e-886f-f4d6cda91f81-kube-api-access-fvfhn\") pod \"961fed7b-2029-4b0e-886f-f4d6cda91f81\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " Oct 30 20:23:58.112872 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.112839 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-data" pod "961fed7b-2029-4b0e-886f-f4d6cda91f81" (UID: "961fed7b-2029-4b0e-886f-f4d6cda91f81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:58.112940 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.112902 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-workload-socket\") pod \"961fed7b-2029-4b0e-886f-f4d6cda91f81\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " Oct 30 20:23:58.113136 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.113051 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961fed7b-2029-4b0e-886f-f4d6cda91f81-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961fed7b-2029-4b0e-886f-f4d6cda91f81-istiod-ca-cert\") pod \"961fed7b-2029-4b0e-886f-f4d6cda91f81\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " Oct 30 20:23:58.113136 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.113134 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-workload-socket" pod "961fed7b-2029-4b0e-886f-f4d6cda91f81" (UID: "961fed7b-2029-4b0e-886f-f4d6cda91f81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:58.113376 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.113356 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-podinfo\") pod \"961fed7b-2029-4b0e-886f-f4d6cda91f81\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " Oct 30 20:23:58.113462 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.113425 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-workload-certs\") pod \"961fed7b-2029-4b0e-886f-f4d6cda91f81\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " Oct 30 20:23:58.113528 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.113467 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961fed7b-2029-4b0e-886f-f4d6cda91f81-istiod-ca-cert" pod "961fed7b-2029-4b0e-886f-f4d6cda91f81" (UID: "961fed7b-2029-4b0e-886f-f4d6cda91f81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:23:58.113596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.113529 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-envoy\") pod \"961fed7b-2029-4b0e-886f-f4d6cda91f81\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " Oct 30 20:23:58.113596 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.113570 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-credential-socket\") pod \"961fed7b-2029-4b0e-886f-f4d6cda91f81\" (UID: \"961fed7b-2029-4b0e-886f-f4d6cda91f81\") " Oct 30 20:23:58.113721 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.113666 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-workload-certs" pod "961fed7b-2029-4b0e-886f-f4d6cda91f81" (UID: "961fed7b-2029-4b0e-886f-f4d6cda91f81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:58.114222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.114107 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:58.114222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.114123 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-credential-socket" pod "961fed7b-2029-4b0e-886f-f4d6cda91f81" (UID: "961fed7b-2029-4b0e-886f-f4d6cda91f81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:58.114222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.114136 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:58.114222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.114172 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:58.114222 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.114188 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961fed7b-2029-4b0e-886f-f4d6cda91f81-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:58.115194 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.115141 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-token" pod "961fed7b-2029-4b0e-886f-f4d6cda91f81" (UID: "961fed7b-2029-4b0e-886f-f4d6cda91f81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:58.115321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.115235 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961fed7b-2029-4b0e-886f-f4d6cda91f81-kube-api-access-fvfhn" pod "961fed7b-2029-4b0e-886f-f4d6cda91f81" (UID: "961fed7b-2029-4b0e-886f-f4d6cda91f81"). InnerVolumeSpecName "kube-api-access-fvfhn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:23:58.115614 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.115593 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-envoy" pod "961fed7b-2029-4b0e-886f-f4d6cda91f81" (UID: "961fed7b-2029-4b0e-886f-f4d6cda91f81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:23:58.115908 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.115890 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-podinfo" pod "961fed7b-2029-4b0e-886f-f4d6cda91f81" (UID: "961fed7b-2029-4b0e-886f-f4d6cda91f81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:23:58.215766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.215728 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvfhn\" (UniqueName: \"kubernetes.io/projected/961fed7b-2029-4b0e-886f-f4d6cda91f81-kube-api-access-fvfhn\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:58.215766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.215762 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:58.215766 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.215772 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:58.216109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.215782 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961fed7b-2029-4b0e-886f-f4d6cda91f81-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:58.216109 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.215790 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961fed7b-2029-4b0e-886f-f4d6cda91f81-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:23:58.763851 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.763813 5091 generic.go:356] "Generic (PLEG): container finished" podID="961fed7b-2029-4b0e-886f-f4d6cda91f81" containerID="ba8a5e01a21350edbb0dbc00c606c6bd3bc8a8e5f343d9c302902bf75f45ba95" exitCode=0 Oct 30 20:23:58.764043 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.763897 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" Oct 30 20:23:58.764043 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.763914 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" event={"ID":"961fed7b-2029-4b0e-886f-f4d6cda91f81","Type":"ContainerDied","Data":"ba8a5e01a21350edbb0dbc00c606c6bd3bc8a8e5f343d9c302902bf75f45ba95"} Oct 30 20:23:58.764043 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.763974 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck" event={"ID":"961fed7b-2029-4b0e-886f-f4d6cda91f81","Type":"ContainerDied","Data":"a5ea39cf719d694b3293bef9fe5134a886b589ef9bf80044de969600e87a3bb9"} Oct 30 20:23:58.764043 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.763999 5091 scope.go:122] "RemoveContainer" containerID="ba8a5e01a21350edbb0dbc00c606c6bd3bc8a8e5f343d9c302902bf75f45ba95" Oct 30 20:23:58.778370 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.778349 5091 scope.go:122] "RemoveContainer" containerID="ba8a5e01a21350edbb0dbc00c606c6bd3bc8a8e5f343d9c302902bf75f45ba95" Oct 30 20:23:58.778683 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:23:58.778660 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8a5e01a21350edbb0dbc00c606c6bd3bc8a8e5f343d9c302902bf75f45ba95\": container with ID starting with ba8a5e01a21350edbb0dbc00c606c6bd3bc8a8e5f343d9c302902bf75f45ba95 not found: ID does not exist" containerID="ba8a5e01a21350edbb0dbc00c606c6bd3bc8a8e5f343d9c302902bf75f45ba95" Oct 30 20:23:58.778772 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.778691 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8a5e01a21350edbb0dbc00c606c6bd3bc8a8e5f343d9c302902bf75f45ba95"} err="failed to get container status \"ba8a5e01a21350edbb0dbc00c606c6bd3bc8a8e5f343d9c302902bf75f45ba95\": rpc error: code = NotFound desc = could not find container \"ba8a5e01a21350edbb0dbc00c606c6bd3bc8a8e5f343d9c302902bf75f45ba95\": container with ID starting with ba8a5e01a21350edbb0dbc00c606c6bd3bc8a8e5f343d9c302902bf75f45ba95 not found: ID does not exist" Oct 30 20:23:58.790497 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.790469 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck"] Oct 30 20:23:58.796343 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:23:58.796318 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5db77fbd9f-d5bck"] Oct 30 20:24:00.037984 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:00.037941 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:00.235122 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:00.235081 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:00.383170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:00.383041 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:00.447810 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:00.447773 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961fed7b-2029-4b0e-886f-f4d6cda91f81" path="/var/lib/kubelet/pods/961fed7b-2029-4b0e-886f-f4d6cda91f81/volumes" Oct 30 20:24:00.807119 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:00.807053 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:04.863584 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:04.863548 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" podUID="6f7ca223-be1f-4f97-aef8-cf286b756081" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:24:05.874501 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:05.874462 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" podUID="8b5a523a-97be-4828-b79e-ec093df306ff" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 20:24:07.478415 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:07.478292 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:07.632251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:07.632206 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:07.766679 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:07.766643 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc"] Oct 30 20:24:07.766920 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:07.766896 5091 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" podUID="c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" containerName="istio-proxy" containerID="cri-o://dae5d1a96ac8c54f041dfb2054e9f3a5d289d810c2c58f870e196ee9dafd753b" gracePeriod=30 Oct 30 20:24:07.836132 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:07.836099 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:09.094442 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.094407 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gbld7_dc0d07b1-78e3-4608-9c78-a33564585b2f/manager/0.log" Oct 30 20:24:09.360881 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.360857 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" Oct 30 20:24:09.413176 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.413136 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" Oct 30 20:24:09.549899 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.549868 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f7ca223-be1f-4f97-aef8-cf286b756081-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7ca223-be1f-4f97-aef8-cf286b756081-plugins-config-volume\") pod \"6f7ca223-be1f-4f97-aef8-cf286b756081\" (UID: \"6f7ca223-be1f-4f97-aef8-cf286b756081\") " Oct 30 20:24:09.550106 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.549938 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b5a523a-97be-4828-b79e-ec093df306ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5a523a-97be-4828-b79e-ec093df306ff-plugins-config-volume\") pod \"8b5a523a-97be-4828-b79e-ec093df306ff\" (UID: \"8b5a523a-97be-4828-b79e-ec093df306ff\") " Oct 30 20:24:09.550106 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.550002 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b5a523a-97be-4828-b79e-ec093df306ff-kube-api-access-m4p57\" (UniqueName: \"kubernetes.io/projected/8b5a523a-97be-4828-b79e-ec093df306ff-kube-api-access-m4p57\") pod \"8b5a523a-97be-4828-b79e-ec093df306ff\" (UID: \"8b5a523a-97be-4828-b79e-ec093df306ff\") " Oct 30 20:24:09.550106 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.550065 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f7ca223-be1f-4f97-aef8-cf286b756081-kube-api-access-xmvwg\" (UniqueName: \"kubernetes.io/projected/6f7ca223-be1f-4f97-aef8-cf286b756081-kube-api-access-xmvwg\") pod \"6f7ca223-be1f-4f97-aef8-cf286b756081\" (UID: \"6f7ca223-be1f-4f97-aef8-cf286b756081\") " Oct 30 20:24:09.550339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.550311 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f7ca223-be1f-4f97-aef8-cf286b756081-plugins-config-volume" pod "6f7ca223-be1f-4f97-aef8-cf286b756081" (UID: "6f7ca223-be1f-4f97-aef8-cf286b756081"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:09.550411 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.550338 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b5a523a-97be-4828-b79e-ec093df306ff-plugins-config-volume" pod "8b5a523a-97be-4828-b79e-ec093df306ff" (UID: "8b5a523a-97be-4828-b79e-ec093df306ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:09.552205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.552176 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f7ca223-be1f-4f97-aef8-cf286b756081-kube-api-access-xmvwg" pod "6f7ca223-be1f-4f97-aef8-cf286b756081" (UID: "6f7ca223-be1f-4f97-aef8-cf286b756081"). InnerVolumeSpecName "kube-api-access-xmvwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:09.552205 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.552198 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b5a523a-97be-4828-b79e-ec093df306ff-kube-api-access-m4p57" pod "8b5a523a-97be-4828-b79e-ec093df306ff" (UID: "8b5a523a-97be-4828-b79e-ec093df306ff"). InnerVolumeSpecName "kube-api-access-m4p57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:09.651046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.650991 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b5a523a-97be-4828-b79e-ec093df306ff-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:09.651046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.651017 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4p57\" (UniqueName: \"kubernetes.io/projected/8b5a523a-97be-4828-b79e-ec093df306ff-kube-api-access-m4p57\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:09.651046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.651028 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmvwg\" (UniqueName: \"kubernetes.io/projected/6f7ca223-be1f-4f97-aef8-cf286b756081-kube-api-access-xmvwg\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:09.651046 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.651037 5091 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f7ca223-be1f-4f97-aef8-cf286b756081-plugins-config-volume\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:09.854642 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.854597 5091 generic.go:356] "Generic (PLEG): container finished" podID="8b5a523a-97be-4828-b79e-ec093df306ff" containerID="aea8a3e36f4b039fa2826060f60cac91afa29ba326559a23f428a1ba49d7412a" exitCode=0 Oct 30 20:24:09.855041 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.855014 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" event={"ID":"8b5a523a-97be-4828-b79e-ec093df306ff","Type":"ContainerDied","Data":"aea8a3e36f4b039fa2826060f60cac91afa29ba326559a23f428a1ba49d7412a"} Oct 30 20:24:09.855220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.855101 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" event={"ID":"8b5a523a-97be-4828-b79e-ec093df306ff","Type":"ContainerDied","Data":"7235132911b5418d13f6a3eb2bad614626b92008a98053456a7aa8bb839886c3"} Oct 30 20:24:09.855220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.855108 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7" Oct 30 20:24:09.855220 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.855125 5091 scope.go:122] "RemoveContainer" containerID="aea8a3e36f4b039fa2826060f60cac91afa29ba326559a23f428a1ba49d7412a" Oct 30 20:24:09.863065 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.863037 5091 generic.go:356] "Generic (PLEG): container finished" podID="6f7ca223-be1f-4f97-aef8-cf286b756081" containerID="35c301b70ba0f174f9a475980824e99b36ae596fc12b55cb617a4ee3199ff7ba" exitCode=0 Oct 30 20:24:09.863225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.863131 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" Oct 30 20:24:09.863225 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.863172 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" event={"ID":"6f7ca223-be1f-4f97-aef8-cf286b756081","Type":"ContainerDied","Data":"35c301b70ba0f174f9a475980824e99b36ae596fc12b55cb617a4ee3199ff7ba"} Oct 30 20:24:09.863377 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.863247 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk" event={"ID":"6f7ca223-be1f-4f97-aef8-cf286b756081","Type":"ContainerDied","Data":"3c8404bdd648daa3bd40a43025c14b86e9217c056b2e16747427bd45ee6c66f7"} Oct 30 20:24:09.875037 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.875012 5091 scope.go:122] "RemoveContainer" containerID="aea8a3e36f4b039fa2826060f60cac91afa29ba326559a23f428a1ba49d7412a" Oct 30 20:24:09.875498 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:24:09.875469 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aea8a3e36f4b039fa2826060f60cac91afa29ba326559a23f428a1ba49d7412a\": container with ID starting with aea8a3e36f4b039fa2826060f60cac91afa29ba326559a23f428a1ba49d7412a not found: ID does not exist" containerID="aea8a3e36f4b039fa2826060f60cac91afa29ba326559a23f428a1ba49d7412a" Oct 30 20:24:09.875638 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.875538 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aea8a3e36f4b039fa2826060f60cac91afa29ba326559a23f428a1ba49d7412a"} err="failed to get container status \"aea8a3e36f4b039fa2826060f60cac91afa29ba326559a23f428a1ba49d7412a\": rpc error: code = NotFound desc = could not find container \"aea8a3e36f4b039fa2826060f60cac91afa29ba326559a23f428a1ba49d7412a\": container with ID starting with aea8a3e36f4b039fa2826060f60cac91afa29ba326559a23f428a1ba49d7412a not found: ID does not exist" Oct 30 20:24:09.875638 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.875566 5091 scope.go:122] "RemoveContainer" containerID="35c301b70ba0f174f9a475980824e99b36ae596fc12b55cb617a4ee3199ff7ba" Oct 30 20:24:09.889928 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.889579 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7"] Oct 30 20:24:09.890313 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.890291 5091 scope.go:122] "RemoveContainer" containerID="35c301b70ba0f174f9a475980824e99b36ae596fc12b55cb617a4ee3199ff7ba" Oct 30 20:24:09.890733 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:24:09.890704 5091 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35c301b70ba0f174f9a475980824e99b36ae596fc12b55cb617a4ee3199ff7ba\": container with ID starting with 35c301b70ba0f174f9a475980824e99b36ae596fc12b55cb617a4ee3199ff7ba not found: ID does not exist" containerID="35c301b70ba0f174f9a475980824e99b36ae596fc12b55cb617a4ee3199ff7ba" Oct 30 20:24:09.890843 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.890738 5091 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35c301b70ba0f174f9a475980824e99b36ae596fc12b55cb617a4ee3199ff7ba"} err="failed to get container status \"35c301b70ba0f174f9a475980824e99b36ae596fc12b55cb617a4ee3199ff7ba\": rpc error: code = NotFound desc = could not find container \"35c301b70ba0f174f9a475980824e99b36ae596fc12b55cb617a4ee3199ff7ba\": container with ID starting with 35c301b70ba0f174f9a475980824e99b36ae596fc12b55cb617a4ee3199ff7ba not found: ID does not exist" Oct 30 20:24:09.893727 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.893699 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-8836a8-7f7565d946-zl9c7"] Oct 30 20:24:09.912399 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.912338 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk"] Oct 30 20:24:09.918648 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:09.918622 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-8836a8-5847dc85d7-vc4fk"] Oct 30 20:24:10.155193 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:10.155138 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-4ttjv_9632993c-8352-4aee-ada2-c5e5fdc74cd0/manager/0.log" Oct 30 20:24:10.441557 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:10.441511 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f7ca223-be1f-4f97-aef8-cf286b756081" path="/var/lib/kubelet/pods/6f7ca223-be1f-4f97-aef8-cf286b756081/volumes" Oct 30 20:24:10.442011 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:10.441987 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b5a523a-97be-4828-b79e-ec093df306ff" path="/var/lib/kubelet/pods/8b5a523a-97be-4828-b79e-ec093df306ff/volumes" Oct 30 20:24:11.736449 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:11.736416 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wjk89_62578a06-c01e-4e9b-8580-2c82df02f25e/manager/0.log" Oct 30 20:24:12.487767 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:12.487730 5091 patch_prober.go:29] interesting pod/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 20:24:12.487962 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:12.487792 5091 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" podUID="c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 20:24:12.894496 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:12.894468 5091 generic.go:356] "Generic (PLEG): container finished" podID="c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" containerID="dae5d1a96ac8c54f041dfb2054e9f3a5d289d810c2c58f870e196ee9dafd753b" exitCode=0 Oct 30 20:24:12.894860 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:12.894545 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" event={"ID":"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a","Type":"ContainerDied","Data":"dae5d1a96ac8c54f041dfb2054e9f3a5d289d810c2c58f870e196ee9dafd753b"} Oct 30 20:24:12.894860 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:12.894610 5091 scope.go:122] "RemoveContainer" containerID="53c68f7af2201706beb91d56fafb0e1723121d0329ed60ead9a128cc3c7d41dd" Oct 30 20:24:13.055012 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.054981 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 20:24:13.083888 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.083851 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-token\" (UniqueName: \"kubernetes.io/projected/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-token\") pod \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " Oct 30 20:24:13.084063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.083949 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-credential-socket\") pod \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " Oct 30 20:24:13.084063 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.084037 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-envoy\") pod \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " Oct 30 20:24:13.084221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.084087 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-kube-api-access-sjg48\" (UniqueName: \"kubernetes.io/projected/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-kube-api-access-sjg48\") pod \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " Oct 30 20:24:13.084221 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.084168 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-data\") pod \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " Oct 30 20:24:13.084343 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.084228 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-workload-certs\") pod \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " Oct 30 20:24:13.084343 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.084255 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-workload-socket\") pod \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " Oct 30 20:24:13.084343 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.084294 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istiod-ca-cert\") pod \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " Oct 30 20:24:13.084343 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.084327 5091 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-podinfo\") pod \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\" (UID: \"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a\") " Oct 30 20:24:13.087011 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.086567 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-workload-certs" pod "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" (UID: "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:13.087011 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.086668 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-data" pod "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" (UID: "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:13.087655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.087382 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-workload-socket" pod "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" (UID: "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:13.087655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.087475 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istiod-ca-cert" pod "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" (UID: "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 20:24:13.087655 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.087626 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-podinfo" pod "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" (UID: "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 20:24:13.088082 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.088056 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-token" pod "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" (UID: "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:13.089937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.089893 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-credential-socket" pod "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" (UID: "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:13.089937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.089912 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-envoy" pod "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" (UID: "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 20:24:13.089937 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.089919 5091 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-kube-api-access-sjg48" pod "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" (UID: "c2fb8632-e65d-4559-a9ad-1a6ecd4e978a"). InnerVolumeSpecName "kube-api-access-sjg48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 20:24:13.185226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.185193 5091 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-envoy\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:13.185226 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.185223 5091 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjg48\" (UniqueName: \"kubernetes.io/projected/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-kube-api-access-sjg48\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:13.185414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.185239 5091 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-data\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:13.185414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.185253 5091 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-workload-certs\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:13.185414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.185265 5091 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-workload-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:13.185414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.185277 5091 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istiod-ca-cert\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:13.185414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.185291 5091 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-podinfo\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:13.185414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.185302 5091 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-istio-token\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:13.185414 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.185315 5091 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a-credential-socket\") on node \"ip-10-0-132-214.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 20:24:13.902858 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.902828 5091 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" Oct 30 20:24:13.903319 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.902866 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc" event={"ID":"c2fb8632-e65d-4559-a9ad-1a6ecd4e978a","Type":"ContainerDied","Data":"0c5c3ffedcd73d0d5a402bf28ddb89cb0f97d0f35f59f5c312afaea0cdd26df2"} Oct 30 20:24:13.903319 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.902925 5091 scope.go:122] "RemoveContainer" containerID="dae5d1a96ac8c54f041dfb2054e9f3a5d289d810c2c58f870e196ee9dafd753b" Oct 30 20:24:13.930829 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.930806 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc"] Oct 30 20:24:13.932828 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:13.932810 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-8836a8-openshift-default-84f65665d5-p7hsc"] Oct 30 20:24:14.439626 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:14.439596 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" path="/var/lib/kubelet/pods/c2fb8632-e65d-4559-a9ad-1a6ecd4e978a/volumes" Oct 30 20:24:15.369914 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:15.369879 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:15.976771 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:24:15.976729 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:24:16.492662 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:16.492627 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:16.708702 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:16.708665 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:18.014394 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:18.014358 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:24.149863 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:24.149827 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:25.117326 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:25.117289 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wjk89_62578a06-c01e-4e9b-8580-2c82df02f25e/manager/0.log" Oct 30 20:24:27.703375 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:27.703343 5091 scope.go:122] "RemoveContainer" containerID="2d439bf3480137cf2e074914cb35272876fc70124dde01a359029617cdd14bfe" Oct 30 20:24:27.713170 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:27.713130 5091 scope.go:122] "RemoveContainer" containerID="2d8f71f98dfedac8f0fc007bf784fa01416e8f2e5d0973d333ba1f26ecdc5a89" Oct 30 20:24:27.721751 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:27.721732 5091 scope.go:122] "RemoveContainer" containerID="04f582944081d056e3030050db4075515f3234af2aaf13a9bb11a45ec67c7990" Oct 30 20:24:27.730178 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:27.730155 5091 scope.go:122] "RemoveContainer" containerID="d1e173bad783867e3714f7f0d26f940e7b9a186246d5c75a47bb641888913445" Oct 30 20:24:27.738362 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:27.738343 5091 scope.go:122] "RemoveContainer" containerID="ce4be51ca712a33e348cd4945f66e24c83927f389a021e8756f7b22ecdf6e6bd" Oct 30 20:24:27.740842 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:27.740822 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b105644-dbmr6_0855b356-7f3a-41aa-a78a-ab8366b566b3/revoke-keys/0.log" Oct 30 20:24:27.746536 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:27.746519 5091 scope.go:122] "RemoveContainer" containerID="7e64a415ab07cd8b9b9ee59426d369d8ddc318594220d0cfeddd2083a35299c5" Oct 30 20:24:27.758045 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:27.758028 5091 scope.go:122] "RemoveContainer" containerID="6c8c09df7713428c2e52a25c7d2d396b66b141c585a1754c412bafb54cd22128" Oct 30 20:24:27.766177 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:27.766135 5091 scope.go:122] "RemoveContainer" containerID="552b6a834aae880c4927eb3a4ad5f5ac76db57f7b85f7d17695e85a43233d80d" Oct 30 20:24:27.774189 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:27.774165 5091 scope.go:122] "RemoveContainer" containerID="c6ac0eb923175b8937d85b657204ca3b938bb7deae5f5b2fdcc7061efbb41deb" Oct 30 20:24:28.088309 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:28.088278 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-712c6641-2c48m_2e50c786-95a9-42b3-94b2-b894ee74e33a/revoke-keys/0.log" Oct 30 20:24:28.271247 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:28.271212 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-7d004ccf-nlptn_f372eb8c-e213-4899-9819-fb9ecb84ed89/revoke-keys/0.log" Oct 30 20:24:28.447265 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:28.447185 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-88fd350d-lg4ph_a9e4cada-6f7a-4396-9cdf-933341e4dcbe/revoke-keys/0.log" Oct 30 20:24:28.626593 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:28.626562 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-99d640d0-t9pmd_19fc6e23-7fad-4948-8650-7835db35d161/revoke-keys/0.log" Oct 30 20:24:28.810693 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:28.810663 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-cf4ff9e7-nmmrj_e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b/revoke-keys/0.log" Oct 30 20:24:28.983432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:28.983403 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-2fgzx_8b14f008-1553-4840-9071-b5899529835d/postgres/0.log" Oct 30 20:24:30.744331 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:30.744300 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gbld7_dc0d07b1-78e3-4608-9c78-a33564585b2f/manager/0.log" Oct 30 20:24:31.783287 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:31.783247 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-4ttjv_9632993c-8352-4aee-ada2-c5e5fdc74cd0/manager/0.log" Oct 30 20:24:32.900011 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:32.899976 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b57kmt86_617067b0-e672-454d-a74c-f80936e85ac3/istio-proxy/0.log" Oct 30 20:24:33.088779 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:33.088745 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-h486p_e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628/discovery/0.log" Oct 30 20:24:33.626617 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:33.626581 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-84zhb_d0dfebf4-1e1b-46a5-86aa-cb488de34ca2/istio-proxy/0.log" Oct 30 20:24:34.571677 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:34.571644 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-zzkkt_c63bc1ab-ca04-4273-ae53-2d0923e4900b/istio-proxy/0.log" Oct 30 20:24:34.753579 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:34.753545 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s_ba979687-111f-44c6-a659-ad087a5fba4b/istio-proxy/0.log" Oct 30 20:24:35.322326 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:35.322290 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-787bddbb47-hfcvw_a8e91049-4a10-4472-922d-7b7875e7de3a/router/0.log" Oct 30 20:24:36.241738 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:36.241707 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-kkzgw_540a152c-2be5-4253-b272-586b34835043/main/0.log" Oct 30 20:24:36.969886 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:36.969849 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-zp9tr_634bf037-b673-4967-b0d9-9d6ef06eb1ca/main/0.log" Oct 30 20:24:37.163231 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:37.163198 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-cs6bc_f1fc93c8-2259-4138-9691-4c9f4e758a01/main/0.log" Oct 30 20:24:37.351201 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:37.351175 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-msxt7_542be5bf-ced0-491a-aade-a84c72a92e5c/main/0.log" Oct 30 20:24:45.976988 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:24:45.976948 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:24:47.343569 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:47.343535 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:47.537953 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:47.537921 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:24:51.978828 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:51.978797 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mm6ld_5bfb73f4-9290-4440-938a-feb9fafa3287/global-pull-secret-syncer/0.log" Oct 30 20:24:52.045713 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:52.045685 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-fkjv2_ad103ddc-523c-4881-8526-00458255bb6c/connection-checker/0.log" Oct 30 20:24:52.083179 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:52.083130 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-pggqx_9458eb02-2921-4f34-9578-60056e3505a4/connection-checker/0.log" Oct 30 20:24:52.121954 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:52.121880 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-rtf4t_06bbc899-e828-4a45-bd9a-3e703c4416e7/connection-checker/0.log" Oct 30 20:24:52.210581 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:52.210551 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" Oct 30 20:24:52.367829 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:52.367797 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-214.us-east-2.compute.internal_84cfec117dc1db7ebfa66a1a705b525e/haproxy/0.log" Oct 30 20:24:59.877555 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:24:59.877501 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:00.065031 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:00.064994 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:00.311542 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:00.311509 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-gbld7_dc0d07b1-78e3-4608-9c78-a33564585b2f/manager/0.log" Oct 30 20:25:00.956741 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:00.956696 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-4ttjv_9632993c-8352-4aee-ada2-c5e5fdc74cd0/manager/0.log" Oct 30 20:25:02.900233 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:02.900195 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-fcjbl_02832de1-9584-4e12-9b94-8b6b52a71941/cluster-monitoring-operator/0.log" Oct 30 20:25:03.244320 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244241 5091 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p"] Oct 30 20:25:03.244715 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244687 5091 cpu_manager.go:414] "containerMap: removing container" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" Oct 30 20:25:03.244715 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244710 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" Oct 30 20:25:03.244876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244735 5091 cpu_manager.go:414] "containerMap: removing container" podUID="6f7ca223-be1f-4f97-aef8-cf286b756081" containerName="payload-pre-processing" Oct 30 20:25:03.244876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244742 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f7ca223-be1f-4f97-aef8-cf286b756081" containerName="payload-pre-processing" Oct 30 20:25:03.244876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244783 5091 cpu_manager.go:414] "containerMap: removing container" podUID="8b5a523a-97be-4828-b79e-ec093df306ff" containerName="payload-processing" Oct 30 20:25:03.244876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244790 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b5a523a-97be-4828-b79e-ec093df306ff" containerName="payload-processing" Oct 30 20:25:03.244876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244801 5091 cpu_manager.go:414] "containerMap: removing container" podUID="a9e4cada-6f7a-4396-9cdf-933341e4dcbe" containerName="revoke-keys" Oct 30 20:25:03.244876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244808 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9e4cada-6f7a-4396-9cdf-933341e4dcbe" containerName="revoke-keys" Oct 30 20:25:03.244876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244820 5091 cpu_manager.go:414] "containerMap: removing container" podUID="961fed7b-2029-4b0e-886f-f4d6cda91f81" containerName="istio-proxy" Oct 30 20:25:03.244876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244826 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961fed7b-2029-4b0e-886f-f4d6cda91f81" containerName="istio-proxy" Oct 30 20:25:03.244876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244838 5091 cpu_manager.go:414] "containerMap: removing container" podUID="c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" containerName="istio-proxy" Oct 30 20:25:03.244876 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244845 5091 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" containerName="istio-proxy" Oct 30 20:25:03.245258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244970 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b5a523a-97be-4828-b79e-ec093df306ff" containerName="payload-processing" Oct 30 20:25:03.245258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244982 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9e4cada-6f7a-4396-9cdf-933341e4dcbe" containerName="revoke-keys" Oct 30 20:25:03.245258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244992 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="961fed7b-2029-4b0e-886f-f4d6cda91f81" containerName="istio-proxy" Oct 30 20:25:03.245258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.244999 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" containerName="istio-proxy" Oct 30 20:25:03.245258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.245012 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2fb8632-e65d-4559-a9ad-1a6ecd4e978a" containerName="istio-proxy" Oct 30 20:25:03.245258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.245022 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="706afd71-a9e1-4b85-a243-b6a9f5e1edb1" containerName="main" Oct 30 20:25:03.245258 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.245032 5091 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f7ca223-be1f-4f97-aef8-cf286b756081" containerName="payload-pre-processing" Oct 30 20:25:03.248901 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.248883 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.258556 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.258534 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p"] Oct 30 20:25:03.333240 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.333192 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p6rpl_f72b3621-e387-4252-a171-17982ec3af80/node-exporter/0.log" Oct 30 20:25:03.339509 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.339487 5091 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4jlph\"/\"default-dockercfg-q7rn9\"" Oct 30 20:25:03.342326 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.342308 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4jlph\"/\"openshift-service-ca.crt\"" Oct 30 20:25:03.342326 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.342312 5091 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4jlph\"/\"kube-root-ca.crt\"" Oct 30 20:25:03.382855 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.382834 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p6rpl_f72b3621-e387-4252-a171-17982ec3af80/kube-rbac-proxy/0.log" Oct 30 20:25:03.397933 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.397911 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e9abc01-d73b-4dca-a57a-ef47730ce87f-lib-modules\") pod \"perf-node-gather-daemonset-ppl4p\" (UID: \"3e9abc01-d73b-4dca-a57a-ef47730ce87f\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.398053 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.397974 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e9abc01-d73b-4dca-a57a-ef47730ce87f-podres\") pod \"perf-node-gather-daemonset-ppl4p\" (UID: \"3e9abc01-d73b-4dca-a57a-ef47730ce87f\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.398102 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.398059 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9bmp\" (UniqueName: \"kubernetes.io/projected/3e9abc01-d73b-4dca-a57a-ef47730ce87f-kube-api-access-r9bmp\") pod \"perf-node-gather-daemonset-ppl4p\" (UID: \"3e9abc01-d73b-4dca-a57a-ef47730ce87f\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.398191 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.398171 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e9abc01-d73b-4dca-a57a-ef47730ce87f-sys\") pod \"perf-node-gather-daemonset-ppl4p\" (UID: \"3e9abc01-d73b-4dca-a57a-ef47730ce87f\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.398249 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.398220 5091 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e9abc01-d73b-4dca-a57a-ef47730ce87f-proc\") pod \"perf-node-gather-daemonset-ppl4p\" (UID: \"3e9abc01-d73b-4dca-a57a-ef47730ce87f\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.419253 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.419237 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p6rpl_f72b3621-e387-4252-a171-17982ec3af80/init-textfile/0.log" Oct 30 20:25:03.499102 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.499074 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e9abc01-d73b-4dca-a57a-ef47730ce87f-lib-modules\") pod \"perf-node-gather-daemonset-ppl4p\" (UID: \"3e9abc01-d73b-4dca-a57a-ef47730ce87f\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.499252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.499134 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e9abc01-d73b-4dca-a57a-ef47730ce87f-podres\") pod \"perf-node-gather-daemonset-ppl4p\" (UID: \"3e9abc01-d73b-4dca-a57a-ef47730ce87f\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.499252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.499217 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9bmp\" (UniqueName: \"kubernetes.io/projected/3e9abc01-d73b-4dca-a57a-ef47730ce87f-kube-api-access-r9bmp\") pod \"perf-node-gather-daemonset-ppl4p\" (UID: \"3e9abc01-d73b-4dca-a57a-ef47730ce87f\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.499252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.499227 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e9abc01-d73b-4dca-a57a-ef47730ce87f-podres\") pod \"perf-node-gather-daemonset-ppl4p\" (UID: \"3e9abc01-d73b-4dca-a57a-ef47730ce87f\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.499443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.499227 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e9abc01-d73b-4dca-a57a-ef47730ce87f-lib-modules\") pod \"perf-node-gather-daemonset-ppl4p\" (UID: \"3e9abc01-d73b-4dca-a57a-ef47730ce87f\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.499443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.499305 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e9abc01-d73b-4dca-a57a-ef47730ce87f-sys\") pod \"perf-node-gather-daemonset-ppl4p\" (UID: \"3e9abc01-d73b-4dca-a57a-ef47730ce87f\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.499443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.499341 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e9abc01-d73b-4dca-a57a-ef47730ce87f-sys\") pod \"perf-node-gather-daemonset-ppl4p\" (UID: \"3e9abc01-d73b-4dca-a57a-ef47730ce87f\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.499443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.499345 5091 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e9abc01-d73b-4dca-a57a-ef47730ce87f-proc\") pod \"perf-node-gather-daemonset-ppl4p\" (UID: \"3e9abc01-d73b-4dca-a57a-ef47730ce87f\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.499443 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.499372 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e9abc01-d73b-4dca-a57a-ef47730ce87f-proc\") pod \"perf-node-gather-daemonset-ppl4p\" (UID: \"3e9abc01-d73b-4dca-a57a-ef47730ce87f\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.512907 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.512879 5091 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9bmp\" (UniqueName: \"kubernetes.io/projected/3e9abc01-d73b-4dca-a57a-ef47730ce87f-kube-api-access-r9bmp\") pod \"perf-node-gather-daemonset-ppl4p\" (UID: \"3e9abc01-d73b-4dca-a57a-ef47730ce87f\") " pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.565296 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.565268 5091 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:03.844246 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:03.844210 5091 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p"] Oct 30 20:25:03.844633 ip-10-0-132-214 kubenswrapper[5091]: W1030 20:25:03.844601 5091 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3e9abc01_d73b_4dca_a57a_ef47730ce87f.slice/crio-2c63cd44447591ca93c484801a59f8c9a7b9b4c677d7453e93ca6fe67cd221a4 WatchSource:0}: Error finding container 2c63cd44447591ca93c484801a59f8c9a7b9b4c677d7453e93ca6fe67cd221a4: Status 404 returned error can't find the container with id 2c63cd44447591ca93c484801a59f8c9a7b9b4c677d7453e93ca6fe67cd221a4 Oct 30 20:25:04.038484 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:04.038454 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-smbbf_216ca825-8a83-4100-ba3b-7d6294c3dadf/prometheus-operator-admission-webhook/0.log" Oct 30 20:25:04.204519 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:04.204430 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" event={"ID":"3e9abc01-d73b-4dca-a57a-ef47730ce87f","Type":"ContainerStarted","Data":"9fcd4ccfdf46e3a979d94d6f055ef9b4f750e2c7b2c90b8171af168fb0c6671d"} Oct 30 20:25:04.204519 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:04.204493 5091 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" event={"ID":"3e9abc01-d73b-4dca-a57a-ef47730ce87f","Type":"ContainerStarted","Data":"2c63cd44447591ca93c484801a59f8c9a7b9b4c677d7453e93ca6fe67cd221a4"} Oct 30 20:25:04.204779 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:04.204630 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:04.226229 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:04.226183 5091 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" podStartSLOduration=1.226171509 podStartE2EDuration="1.226171509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 20:25:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 20:25:04.225844164 +0000 UTC m=+3223.921052123" watchObservedRunningTime="2026-10-30 20:25:04.226171509 +0000 UTC m=+3223.921379457" Oct 30 20:25:04.273379 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:04.273354 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:05.040190 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:05.040136 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn"] Oct 30 20:25:05.042405 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:05.042380 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d004ccf-nlptn"] Oct 30 20:25:05.889567 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:05.889534 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-86c478bb88-twdf7_6e13582a-7d64-443a-a62f-c14ecc655f18/networking-console-plugin/0.log" Oct 30 20:25:06.439839 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:06.439807 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f372eb8c-e213-4899-9819-fb9ecb84ed89" path="/var/lib/kubelet/pods/f372eb8c-e213-4899-9819-fb9ecb84ed89/volumes" Oct 30 20:25:06.764977 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:06.764942 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/1.log" Oct 30 20:25:06.773339 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:06.773311 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-4kpqr_067a4436-690d-4b2c-aaf2-1d84830b7287/console-operator/2.log" Oct 30 20:25:07.813516 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:07.813471 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wjk89_62578a06-c01e-4e9b-8580-2c82df02f25e/manager/0.log" Oct 30 20:25:07.827530 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:07.827501 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-xxslt_d3052bdd-cd3b-4d52-998c-538138c6d060/download-server/0.log" Oct 30 20:25:08.021419 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:08.021386 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wjk89_62578a06-c01e-4e9b-8580-2c82df02f25e/manager/0.log" Oct 30 20:25:08.901683 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:08.901654 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-58c9f5f46d-vblv9_6693a492-433f-4614-a8e3-71ee75c177c2/volume-data-source-validator/0.log" Oct 30 20:25:10.221018 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:10.220991 5091 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4jlph/perf-node-gather-daemonset-ppl4p" Oct 30 20:25:11.107117 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:11.107088 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j2h9z_01188e4b-c110-477a-ba88-d9f48741c0a2/dns/0.log" Oct 30 20:25:11.139833 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:11.139797 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-j2h9z_01188e4b-c110-477a-ba88-d9f48741c0a2/kube-rbac-proxy/0.log" Oct 30 20:25:11.291033 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:11.290996 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-9g689_7fa163f8-129c-4f64-ad4b-ad3cc08bed13/dns-node-resolver/0.log" Oct 30 20:25:12.245609 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:12.245575 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wjk89_62578a06-c01e-4e9b-8580-2c82df02f25e/manager/0.log" Oct 30 20:25:12.484690 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:12.484661 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-79fdc76dbd-5q48r_6fa2a92c-d7cc-45c9-8424-347bf91f45d9/registry/0.log" Oct 30 20:25:12.576129 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:12.576098 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wjk89_62578a06-c01e-4e9b-8580-2c82df02f25e/manager/0.log" Oct 30 20:25:12.631955 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:12.631931 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-m77bc_fb5d13ef-8197-4960-b142-d8a1bdd025e2/node-ca/0.log" Oct 30 20:25:15.044321 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:15.044281 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b57kmt86_617067b0-e672-454d-a74c-f80936e85ac3/istio-proxy/0.log" Oct 30 20:25:15.443313 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:15.443231 5091 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9hgsn_6f0f5eb3-92c0-4822-867c-cf802c9e8490/konnectivity-agent/0.log" limit=1024 Oct 30 20:25:15.982975 ip-10-0-132-214 kubenswrapper[5091]: E1030 20:25:15.982934 5091 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-fndt2" containerName="extractor" Oct 30 20:25:16.030972 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:16.030937 5091 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj"] Oct 30 20:25:16.039461 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:16.039379 5091 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf4ff9e7-nmmrj"] Oct 30 20:25:16.445182 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:16.445108 5091 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b" path="/var/lib/kubelet/pods/e468b395-2a21-4cc9-a82f-0e6d1e9ecc2b/volumes" Oct 30 20:25:16.937600 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:16.937558 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wjk89_62578a06-c01e-4e9b-8580-2c82df02f25e/manager/0.log" Oct 30 20:25:17.084787 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:17.084759 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-wjk89_62578a06-c01e-4e9b-8580-2c82df02f25e/manager/0.log" Oct 30 20:25:17.289432 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:17.289395 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-h486p_e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628/discovery/0.log" Oct 30 20:25:17.711301 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:17.711222 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b57kmt86_617067b0-e672-454d-a74c-f80936e85ac3/istio-proxy/0.log" Oct 30 20:25:18.185442 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:18.185407 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-84zhb_d0dfebf4-1e1b-46a5-86aa-cb488de34ca2/istio-proxy/0.log" Oct 30 20:25:18.406800 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:18.406762 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-zzkkt_c63bc1ab-ca04-4273-ae53-2d0923e4900b/istio-proxy/0.log" Oct 30 20:25:18.443252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:18.443178 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s_ba979687-111f-44c6-a659-ad087a5fba4b/istio-proxy/0.log" Oct 30 20:25:18.579392 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:18.579344 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-787bddbb47-hfcvw_a8e91049-4a10-4472-922d-7b7875e7de3a/router/0.log" Oct 30 20:25:19.978251 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:19.978221 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-h486p_e6760b67-0f0c-4bcc-b36f-f3d3f1bbe628/discovery/0.log" Oct 30 20:25:20.055203 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:20.055170 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-84zhb_d0dfebf4-1e1b-46a5-86aa-cb488de34ca2/istio-proxy/0.log" Oct 30 20:25:20.198267 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:20.198234 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5db77fbd9f-zzkkt_c63bc1ab-ca04-4273-ae53-2d0923e4900b/istio-proxy/0.log" Oct 30 20:25:20.217965 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:20.217935 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-7454465567-q9n4s_ba979687-111f-44c6-a659-ad087a5fba4b/istio-proxy/0.log" Oct 30 20:25:20.296801 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:20.296767 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-787bddbb47-hfcvw_a8e91049-4a10-4472-922d-7b7875e7de3a/router/0.log" Oct 30 20:25:21.542068 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:21.542030 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b57kmt86_617067b0-e672-454d-a74c-f80936e85ac3/istio-proxy/0.log" Oct 30 20:25:21.637715 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:21.637630 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-77ms2_534aacd9-7102-4a50-8696-c50362ce87e4/serve-healthcheck-canary/0.log" Oct 30 20:25:21.714356 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:21.714314 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6c786f9b57kmt86_617067b0-e672-454d-a74c-f80936e85ac3/istio-proxy/0.log" Oct 30 20:25:22.642006 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:22.641967 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-7jhj7_24ac5496-c680-4b2a-8362-f12b662df997/insights-operator/1.log" Oct 30 20:25:22.646825 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:22.646784 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-7c4986d449-7jhj7_24ac5496-c680-4b2a-8362-f12b662df997/insights-operator/2.log" Oct 30 20:25:22.726252 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:22.726218 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fndt2_b6e429ff-3fb2-48da-a77b-7806b90139e5/kube-rbac-proxy/0.log" Oct 30 20:25:22.782168 ip-10-0-132-214 kubenswrapper[5091]: I1030 20:25:22.782111 5091 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-fndt2_b6e429ff-3fb2-48da-a77b-7806b90139e5/exporter/0.log"